muld 0.0.11

git; Mirror from Upstream to Local to Downstream

If you use git submodules, then you might have faced the issue that a user of your project does not have access to the remotes that your submodules point to.

You can then provide a full source-tarball to them instead, however, that is not always ideal. Sometimes, other infrastructure such as GitLAB, or BitBucket can provide automatic mirroring your third-party dependencies but if they cannot access that either, well then that is not really a solution either.

So, one might need to manually maintain some downstream mirrors, based on the actual upstream repositories. This is an annoyingly manual tasks, however, this command muld takes most of the heavy lifting out of it, such that you only have to fix issues when mirroring for some reason fails, but it will otherwise run happily in a cron-job.

NOTE, what muld does, could probably be done with 10 lines of Bash. But, using Python for the task is 1000 times more fun. Also, muld does plain git clone, does not use --bare, --mirror or --recursive.

Install

The common mode of usage is to install muld via the Python Package Index:

pip3 install --user muld

Once you have done so, you should be able to run it, try:

muld --help

If that does not work, then make sure that your shell looks for executables/binaries in the Python --user home, you can see where that is by running:

python3 -m site --user-base

And if you use the Bash shell, then you can add it to PATH like so:

echo "export PATH=\"$PATH:$(python3 -m site --user-base)/bin\"" >> ~/.bash_aliases

Usage

Provide a list of repository upstream/downstream pairs such as:

repos:
  - name: "spdk"
    upstream: "https://github.com/spdk/spdk"
    downstream: "ssh://[e-mail]/somewhere/else/spdk.git"
  - name: "dpdk"
    upstream: "https://github.com/spdk/dpdk.git"
    downstream: "ssh://[e-mail]/somewhere/else/dpdk.git"

In a file and name it as e.g. muld.yml. Then invoke muld:

muld --mirrors $HOME/mirrors --conf muld.yml

Behaviour

It will initially clone the upstream repository, when it exists then just fetch and prune. Regardless, after it will then push tags and branches to downstream.

If --conf is not given, then it assumes that a file named muld.yml exists inside the directory pointed to by --mirrors.

If --mirrors is not given, then it uses the current-working-directory.

Thus, in case you have prepared a home for mirrors with a config inside it, for example like so:

# Setup
mkdir $HOME/mirrors
echo "repos: [{upstream: 'https://github.com/spdk/spdk'}]" > muld.yml
cd $HOME/mirrors

Then you can just go to the directory and run muld:

# Run it...
cd $HOME/mirrors
muld

Regarding the config-file, only upstream is required. When name is left out, then muld will produce a named based on the given upstream URL.

If downstream is left out, then it simply won’t push anywhere, it will just clone, prune and fetch.