Fetch using Gitlab API

Ammar Najjar · Posted on February 19, 2019

pythongitgitlab

Fetch using Gitlab API

Motivation

Between gitlab repos that need daily tracking and frequent machine changes, the manual cloning and sync process became exhausting. An automation script was the obvious next step to offload the boring tasks.

Process

The first version was a simple bash script with a for-loop, but repo paths were hard-coded.

#!/bin/bash

current_dir=$(pwd)
REPOS=(
	# list of all repos names
)

function doit() {
	for repo in ${REPOS[@]}; do
		if [ ! -d "$repo" ]; then
			echo " - cloning $repo ..."
				# git clone $repo
		else
			cd $current_dir/$repo
			echo " - fetching $repo ..."
			git fetch
			cd $current_dir
		fi
	done
  echo "Done"
}

doit

This was fast and worked for a while, but when multiple projects entered the picture and co-workers added/removed/renamed repos, the script required constant manual updates. Being lazy by nature, a more automated approach was necessary, which led to the Gitlab API-Documentation, where the fun started.

Solution Idea

A python script to talk to the gitlab API, and use the /projects end point to access the repos.

Requirements:

• Gtilab Token: create a private token first to use it accessing the API as explained in the docs.
• Virtual access to the repos (e.g.: gitlab.com)

Optional:

• Pipenv to install the dev-dependencies:

brew install pipenv

First Try: argparse

The first try was to pass the options as arguments to the script:

#!/usr/local/bin/python3.7
"""
Clone/fetch projects from Gitlab using the private token
"""

import argparse
from os import path
import json
import subprocess
import shlex
from urllib.request import urlopen

IGNORE_LIST = [
    'test',
    'example',
    'tour'
]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--token', '-t', help='Gitlab private Token')
    parser.add_argument('--url', '-u', help='Gitlab URL')
    args = parser.parse_args()

    if not args.token or not args.url:
        parser.print_help()
        exit(1)

    gitlab_token = args.token
    gitlab_url = args.url

    projects = urlopen(
        f'https://{gitlab_url}/api/v4/projects?membership=1&order_by=path&per_page=1000&private_token={gitlab_token}')
    all_projects = json.loads(projects.read().decode())

    for project in all_projects:
        try:
            url = project.get('ssh_url_to_repo')
            if any([x in url for x in IGNORE_LIST]):
                continue
            name = project.get('name').replace(' ', '-').replace('.', '-')
            repo_path = path.join(path.dirname(path.realpath(__file__)), name)
            if path.isdir(repo_path):
                print(f'Fetching {name}')
                command = shlex.split(f'git -C {repo_path} fetch')
                process = subprocess.Popen(command)
                process.communicate(input=None)
            else:
                print(f'Cloning {name}')
                command = shlex.split(f'git clone {url} {name}')
                process = subprocess.Popen(command)
                process.communicate(input=None)
        except Exception as unexpected_exception:
            print(f"Error on {url}: {str(unexpected_exception)}")
    print('Done')


if __name__ == '__main__':
    main()

Usage:

usage: main.py [-h] [--token TOKEN] [--url URL]

optional arguments:
  -h, --help            show this help message and exit
  --token TOKEN, -t TOKEN
                        Gitlab private Token
  --url URL, -u URL     Gitlab URL<Paste>

Second Try: JSON

Configuration flexibility demanded a separate config file, so a JSON-based approach replaced the argument parser.

Full project repository can be found here

#!/usr/local/bin/python3.7
"""
Clone/fetch projects from Gitlab using the private token
NOTE:
The gitlab url, token should be provided in a config.json
file, which should exist in the same direcotry as this script.
"""

import json
import shlex
import subprocess
import sys
from os import path
from urllib.request import urlopen

ROOT = path.dirname(path.abspath(__file__))


def read_configs():
    configs = {}
    try:
        with open(f'{ROOT}/config.json', 'r') as config_file:
            configs = json.loads(config_file.read())
    except FileNotFoundError:
        print('config.json cannot be found')
    except json.decoder.JSONDecodeError:
        print('Please provide gitlab configs in json format')
    return configs


def shell(command):
    process = subprocess.Popen(command)
    process.communicate(input=None)


def main():
    configs = read_configs()
    gitlab_url = configs.get('gitlab_url')
    gitlab_token = configs.get('gitlab_token')
    if not(gitlab_url and gitlab_token):
        print('Please provide gitlab configs in your config.json')
        sys.exit(1)
    ignore_list = configs.get('ignore_list')

    projects = urlopen(
        f'https://{gitlab_url}/api/v4/projects?membership=1&order_by=path&per_page=1000&private_token={gitlab_token}')
    all_projects = json.loads(projects.read().decode())

    for project in all_projects:
        try:
            url = project.get('ssh_url_to_repo')
            if any([x in url for x in ignore_list]):
                continue
            name = project.get('name').replace(' ', '-').replace('.', '-')
            repo_path = path.join(path.dirname(path.realpath(__file__)), name)
            if path.isdir(repo_path):
                print(f'Fetching {name}')
                command = shlex.split(f'git -C {repo_path} fetch')
                shell(command)
            else:
                print(f'Cloning {name}')
                command = shlex.split(f'git clone {url} {name}')
                shell(command)
        except Exception as unexpected_exception:
            print(f"Error on {url}: {str(unexpected_exception)}")
    print('Done')
    sys.exit(0)


if __name__ == '__main__':
    main()

Usage

python main.py

Go to Top