A Gitlab clone/pull utility for backing up or cloning Gitlab groups
Project description
.. image:: https://github.com/ezbz/gitlabber/actions/workflows/python-app.yml/badge.svg?branch=master :target: https://github.com/ezbz/gitlabber/actions/workflows/python-app.yml
.. image:: https://codecov.io/gh/ezbz/gitlabber/branch/master/graph/badge.svg :target: https://codecov.io/gh/ezbz/gitlabber
.. image:: https://badge.fury.io/py/gitlabber.svg :target: https://badge.fury.io/py/gitlabber
.. image:: https://img.shields.io/pypi/l/gitlabber.svg :target: https://pypi.python.org/pypi/gitlabber/
.. image:: https://img.shields.io/pypi/pyversions/gitlabber :target: https://pypi.python.org/pypi/gitlabber/
.. image:: https://readthedocs.org/projects/gitlabber/badge/?version=latest&style=plastic :target: https://gitlabber.readthedocs.io/en/latest/README.html
Gitlabber
- A utility to clone and pull GitLab groups, subgroups, projects based on path selection
Purpose
Gitlabber clones or pulls all projects under a subset of groups / subgroups by building a tree from the GitLab API and allowing you to specify which subset of the tree you want to clone using glob patterns and/or regex expressions.
Installation
- You can install Gitlabber from
PyPi <https://pypi.org/project/gitlabber>
_:
.. code-block:: bash
pip install gitlabber
- You'll need to create an
access token <https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html>
_ from GitLab with API scopesread_repository
andread_api
(orapi
, for GitLab versions <12.0)
Usage
-
Arguments can be provided via the CLI arguments directly or via environment variables:: +---------------+---------------+---------------------------+ | Argument | Flag | Environment Variable | +===============+===============+===========================+ | token | -t |
GITLAB_TOKEN
| +---------------+---------------+---------------------------+ | url | -u |GITLAB_URL
| +---------------+---------------+---------------------------+ | method | -m |GITLABBER_CLONE_METHOD
| +---------------+---------------+---------------------------+ | naming | -n |GITLABBER_FOLDER_NAMING
| +---------------+---------------+---------------------------+ | include | -i |GITLABBER_INCLUDE
| +---------------+---------------+---------------------------+ | exclude | -x |GITLABBER_EXCLUDE
| +---------------+---------------+---------------------------+ -
To view the tree run the command with your includes/excludes and the
-p
flag. It will print your tree like so:
.. code-block:: bash
root [http://gitlab.my.com]
├── group1 [/group1]
│ └── subgroup1 [/group1/subgroup1]
│ └── project1 [/group1/subgroup1/project1]
└── group2 [/group2]
├── subgroup1 [/group2/subgroup1]
│ └── project2 [/group2/subgroup1/project2]
├── subgroup2 [/group2/subgroup2]
└── subgroup3 [/group2/subgroup3]
-
To see how to use glob patterns and regex to filter tree nodes, see the
globre project page <https://pypi.org/project/globre/#details>
_. -
Cloning vs Pulling: when running Gitlabber consecutively with the same parameters, it will scan the local tree structure; if the project directory exists and is a valid git repository (has .git folder in it) Gitlabber will perform a git pull in the directory, otherwise the project directory will be created and the GitLab project will be cloned into it.
-
Cloning submodules: use the
-r
flag to recurse git submodules, uses the--recursive
for cloning and utilizesGitPython's smart update method <https://github.com/gitpython-developers/GitPython/blob/20f4a9d49b466a18f1af1fdfb480bc4520a4cdc2/git/objects/submodule/root.py#L67>
_ for updating cloned repositories -
Printed Usage:
.. code-block:: bash
usage: gitlabber [-h] [-t token] [-u url] [--verbose] [-p] [-d]
[--print-format {json,yaml,tree}] [-m {ssh,https}] [-i csv]
[-x csv] [--version] [-g {id,full_path,full_name}]
[dest]
Gitlabber - clones or pulls entire groups/projects tree from GitLab
positional arguments:
dest destination path for the cloned tree (created if doesn't exist)
optional arguments:
-h, --help show this help message and exit
-t token, --token token
gitlab personal access token https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html
-u url, --url url base gitlab url (e.g.: 'http://gitlab.mycompany.com')
--verbose print more verbose output
-p, --print print the tree without cloning
--print-format {json,yaml,tree}
print format (default: 'tree')
-n {name,path}, --naming {name,path}
the folder naming strategy for projects (default: "name")
-m {ssh,http}, --method {ssh,http}
the git transport method to use for cloning (default: "ssh")
-a {include,exclude,only}, --archived {include,exclude,only}
include archived projects and groups in the results (default: "include")
-i csv, --include csv
comma delimited list of glob patterns of paths to projects or groups to clone/pull
-x csv, --exclude csv
comma delimited list of glob patterns of paths to projects or groups to exclude from clone/pull
-r, --recursive clone/pull git submodules recursively
-T, --hide-token hide the token from the gitlab request URL (for method: http)
-f, --use-fetch clone/fetch git repository (mirrored repositories)
-s, --include-shared include shared projects in the results
--version print the version
examples:
clone an entire gitlab tree using a base url and a token:
gitlabber -t <personal access token> -u <gitlab url> .
# the following examples assume you provided token/url in environment variables so these arguments are omitted
only print the gitlab tree:
gitlabber -p .
clone only projects under subgroup 'MySubGroup' to location '~/GitlabRoot':
gitlabber -i '/MyGroup/MySubGroup**' ~/GitlabRoot
clone only projects under group 'MyGroup' excluding any projects under subgroup 'MySubGroup':
gitlabber -i '/MyGroup**' -x '/MyGroup/MySubGroup**' .
clone an entire gitlab tree except projects under groups named 'ArchiveGroup':
gitlabber -x '/ArchiveGroup**' .
clone projects that start with a case insensitive 'w' using a regular expression:
gitlabber -i '/{[w].*}' .
Debugging
- You can use the
--verbose
flag to print Gitlabber debug messages - For more verbose GitLab messages, you can get the
GitPython <https://gitpython.readthedocs.io/en/stable>
_ module to print more debug messages by setting the environment variable:
.. code-block:: bash
export GIT_PYTHON_TRACE='full'
Troubleshooting
GitlabHttpError: 503
: make sure you provide the base URL to your GitLab installation (e.g.,https://gitlab.my.com
and nothttps://gitlab.my.com/some/nested/path
)
Known Limitations
- Renaming, moving and deleting projects: Gitlabber doesn't maintain local tree state (projects and groups). For that reason is does not rename move or delete local projects when they are modified on the server. When projects are moved or renamed, Gitlabber will clone them again under their new name or location. When deleted, Gitlabber will not delete the local project.
- Folder naming strategy: Consecutively running Gitlabber with different values for the
-n
parameter will produce undesirable results. Use the same value as previous runs, or simply don't change it from the default (project name). - If you're going to clone a large number of projects, observe rate limits
for gitlab.com <https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits/>
, andfor on-premise installations <https://docs.gitlab.com/ee/security/rate_limits.html>
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for gitlabber-1.2.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd59da9ddf2dd9d1a9c61b790c5cb5422d5f53deefdf77281f36f8a9d23adc58 |
|
MD5 | ea84aaf1cb23c00d83eb952fc8c65922 |
|
BLAKE2b-256 | 130e50ed2a98fc65d18cdc0c906f9a7c4d8000d5be0a37f1272debccceeaa61d |