ghcloneall
It’s a script to clone/update all repos for a user/organization from GitHub.
Target audience: maintainers of large collections of projects (for example, ZopeFoundation members).
Usage examples
First pip install ghcloneall.
Clone all mgedmin’s vim plugins:
mkdir ~/src/vim-plugins cd ~/src/vim-plugins ghcloneall --init --user mgedmin --pattern '*.vim' ghcloneall
Clone all ZopeFoundation repositories:
mkdir ~/src/zf cd ~/src/zf ghcloneall --init --org ZopeFoundation ghcloneall
Here’s a screencast of the above (running a slightly older version so the script name differs):
Details
What it does:
clones repositories you don’t have locally
pulls changes for repositories you already have locally
warns you about local changes and other unexpected situations:
unknown files in the tree (in –verbose mode only)
staged but not committed changes
uncommitted (and unstaged changes)
non-master branch checked out
committed changes that haven’t been pushed to master
remote URL pointing to an unexpected location (in –verbose mode only)
You can ask it to not change any files on disk and just look for pending changes by running ghcloneall --dry-run. This will also make the check faster!
Synopsis
Other command-line options:
$ ghcloneall --help
usage: ghcloneall [-h] [--version] [-c CONCURRENCY] [-n] [-v]
[--start-from REPO] [--organization ORGANIZATION]
[--user USER] [--pattern PATTERN] [--init]
[--http-cache DBNAME] [--no-http-cache]
Clone/update all user/org repositories from GitHub.
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit
-c CONCURRENCY, --concurrency CONCURRENCY
set concurrency level
-n, --dry-run don't pull/clone, just print what would be done
-v, --verbose perform additional checks
--start-from REPO skip all repositories that come before REPO
alphabetically
--organization ORGANIZATION
specify the GitHub organization
--user USER specify the GitHub user
--pattern PATTERN specify repository name pattern to filter
--init create a .ghcloneallrc from command-line arguments
--http-cache DBNAME cache HTTP requests on disk in an sqlite database
(default: .httpcache)
--no-http-cache disable HTTP disk caching
Configuration file
The script looks for .ghcloneallrc in the current working directory, which should look like this:
[ghcloneall] # Provide either github_user or github_org, but not both # github_org = ZopeFoundation github_user = mgedmin pattern = *.vim
You can create one with ghcloneall --init --{user,org} X [--pattern Y].
Tips
For best results configure SSH persistence to speed up git pulls – in your ~/.ssh/config:
Host github.com ControlMaster auto ControlPersist yes ControlPath ~/.ssh/control-%r@%h-%p
It takes about 1 minute to run git pull on all 339 ZopeFoundation repos on my laptop with this kind of setup.
Changelog
1.6.1 (2018-10-19)
Fix TypeError: get() got an unexpected keyword argument ‘fallback’ on Python 2.
1.6 (2016-12-29)
Comprehensive rebranding:
Rename the GitHub repository to https://github.com/mgedmin/ghcloneall
Rename cloneall.py to ghcloneall.py
Rename the config file to .ghcloneallrc, and rename the config section to [ghcloneall].
Don’t print tracebacks on ^C (this regressed in 1.5).
1.5 (2016-12-29)
Released to PyPI as ghcloneall
Added Python 2.7 support
1.4 (2016-12-28)
Command line args: –user, –pattern, –init
Load (some) options from a .cloneallrc
Stop using --organization=ZopeFoundation by default, require an explicit option (or config file)
Rename clone_all_zf_repos.py to cloneall.py
1.3 (2016-12-28)
Command line args: -c
Show progress while fetching the list of repositories from GitHub
Update repositories concurrently by default
Highlight items in progress
Highlight failed items in red
Tweak progress bar style from [=== ] to [###..]
Clear the progress bar on ^C
Handle git errors nicely
Bugfix: -vv could fail with NameError if unknown files were present in a working tree
Bugfix: correctly show progress when using –start-from
Bugfix: script would hang (for 10 minutes) if you didn’t already have an SSH control master process running
Bugfix: –dry-run didn’t show which repos were new
1.2 (2016-11-09)
Command line args: –dry-run, –verbose
Cache HTTP responses on disk for 10 minutes to avoid GitHub API rate limits
Report about forgotten uncommitted and staged changes
Warn about local (unpushed) commits too
Warn about other branches being checked out
Default to SSH URLs again (faster when using SSH’s ControlPersist)
1.1 (2015-11-07)
Command line args: –version, –start-from, –organization
Output formatting: shorter repository names, totals at the end
Use ANSI colors to indicate changes
Don’t print tracebacks on ^C
Default to HTTPS URLs
1.0 (2015-11-07)
Moved from a gist to a proper GitHub repository.
Metadata
Release files for ghcloneall 1.6.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ghcloneall-1.6.1.tar.gz | 12.4 kB | Without provenance | Details for ghcloneall-1.6.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ghcloneall-1.6.1-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details for ghcloneall-1.6.1-py2.py3-none-any.whl |
Total release size: 23.5 kB
Release files / ghcloneall-1.6.1.tar.gz
| Download URL | ghcloneall-1.6.1.tar.gz |
|---|---|
| Size | 12.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cdce59aa2693979de489303d22ac55717c0dc939476e4e4db71eea313ac38970
|
|
BLAKE2b-256 checksum How to use checksums |
aa7f25d74c5a830b007a0b639e245337d91d4ebf6579081a1876c49282228748
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.18.4 setuptools/40.4.3 requests-toolbelt/0.8.0 tqdm/4.27.0 CPython/2.7.15+
|
Release files / ghcloneall-1.6.1-py2.py3-none-any.whl
| Download URL | ghcloneall-1.6.1-py2.py3-none-any.whl |
|---|---|
| Size | 11.1 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
06406d1ae1e85e79b99a0fe7c053cc2b32b9258f5fe658677acd3dffa8f260b5
|
|
BLAKE2b-256 checksum How to use checksums |
2c96fa263f8b63f452e5a4d4b6688a62a274fc02a2fdfa7b81a14e64a31106a2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.18.4 setuptools/40.4.3 requests-toolbelt/0.8.0 tqdm/4.27.0 CPython/2.7.15+
|
Security for ghcloneall 1.6.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.