grokmirror 2.0.11

DESCRIPTION

Grokmirror was written to make replicating large git repository collections more efficient. Grokmirror uses the manifest file published by the origin server in order to figure out which repositories to clone, and to track which repositories require updating. The process is lightweight and efficient both for the primary and for the replicas.

CONCEPTS

The origin server publishes a json-formatted manifest file containing information about all git repositories that it carries. The format of the manifest file is as follows:

{
  "/path/to/bare/repository.git": {
    "description": "Repository description",
    "head":        "ref: refs/heads/branchname",
    "reference":   "/path/to/reference/repository.git",
    "forkgroup":   "forkgroup-guid",
    "modified":    timestamp,
    "fingerprint": sha1sum(git show-ref),
    "symlinks": [
        "/location/to/symlink",
        ...
    ],
   }
   ...
}

The manifest file is usually gzip-compressed to preserve bandwidth.

Each time a commit is made to one of the git repositories, it automatically updates the manifest file using an appropriate git hook, so the manifest.js file should always contain the most up-to-date information about the state of all repositories.

The mirroring clients will poll the manifest.js file and download the updated manifest if it is newer than the locally stored copy (using Last-Modified and If-Modified-Since http headers). After downloading the updated manifest.js file, the mirrors will parse it to find out which repositories have been updated and which new repositories have been added.

ORIGIN SETUP

Install grokmirror on the origin server using your preferred way.

IMPORTANT: Only bare git repositories are supported.

You will need to add a hook to each one of your repositories that would update the manifest upon repository modification. This can either be a post-receive hook, or a post-update hook. The hook must call the following command:

/usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \
    -t /var/lib/gitolite3/repositories -n `pwd`

The -m flag is the path to the manifest.js file. The git process must be able to write to it and to the directory the file is in (it creates a manifest.js.randomstring file first, and then moves it in place of the old one for atomicity).

The -t flag is to help grokmirror trim the irrelevant toplevel disk path, so it is trimmed from the top.

The -n flag tells grokmirror to use the current timestamp instead of the exact timestamp of the commit (much faster this way).

Before enabling the hook, you will need to generate the manifest.js of all your git repositories. In order to do that, run the same command, but omit the -n and the `pwd` argument. E.g.:

/usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \
    -t /var/lib/gitolite3/repositories

The last component you need to set up is to automatically purge deleted repositories from the manifest. As this can’t be added to a git hook, you can either run the --purge command from cron:

/usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \
    -t /var/lib/gitolite3/repositories -p

Or add it to your gitolite’s D command using the --remove flag:

/usr/bin/grok-manifest -m /var/www/html/manifest.js.gz \
    -t /var/lib/gitolite3/repositories -x $repo.git

If you would like grok-manifest to honor the git-daemon-export-ok magic file and only add to the manifest those repositories specifically marked as exportable, pass the --check-export-ok flag. See git-daemon(1) for more info on git-daemon-export-ok file.

You will need to have some kind of httpd server to serve the manifest file.

REPLICA SETUP

Install grokmirror on the replica using your preferred way.

Locate grokmirror.conf and modify it to reflect your needs. The default configuration file is heavily commented to explain what each option does.

Make sure the user “mirror” (or whichever user you specified) is able to write to the toplevel and log locations specified in grokmirror.conf.

You can either run grok-pull manually, from cron, or as a systemd-managed daemon (see contrib). If you do it more frequently than once every few hours, you should definitely run it as a daemon in order to improve performance.

GROK-FSCK

Git repositories should be routinely repacked and checked for corruption. This utility will perform the necessary optimizations and report any problems to the email defined via fsck.report_to (‘root’ by default). It should run weekly from cron or from the systemd timer (see contrib).

Please examine the example grokmirror.conf file for various things you can tweak.

FAQ