clowder-repo 4.0b6

Utility for managing multiple git repositories

clowder - A group of cats

herding cats - An idiom that refers to a frustrating attempt to control or organize a class of entities which are uncontrollable or chaotic

Why clowder

There are many ways to organize projects with git. Monorepos, submodules, subtrees, or some other tool. clowder is one of the other tools. Its approach is heavily influeced by the repo tool Google uses to manage the Android Open Source Project.

Projects are listed in a clowder.yml file that can be checked into its own repo, allowing it to be shared across teams. clowder essentially makes this file executable, allowing commands to be run across projects. clowder can update submodules, lfs files, and custom git config entries. Projects can track branches, or be tied to specific tags or commits. Forks can be configured along with their upstream source, wherever they may live. Snapshots of project states can be saved for later restoration. And more...

Daily development still takes place in individual repos, with normal git commands. But clowder is there if you need to synchronize or run commands on multiple repos.

Installation

Requirements:

• git
• Python 3

To install or upgrade clowder from PyPI:

sudo pip3 install clowder-repo --upgrade

To install the latest pre-release version:

sudo pip3 install clowder-repo --force-reinstall --pre

The clowder.yml file

For the full specification, see the clowder.yml syntax reference

An example clowder.yml for some well-known projects:

name: cool-projects

defaults:
  branch: master
  remote: origin
  source: github
  protocol: ssh

sources:
  github:
    url: github.com
  google:
    url: gerrit.googlesource.com
    protocol: https

projects:
  - name: llvm/llvm-project
  - name: apple/swift
  - name: tensorflow/tensorflow
  - name: git-repo
    path: repo
    source: google

The name is simply a descriptive label. The defaults section contains the git branch and remote, the source to clone from, and the protocol to use for cloning repositories. clowder assumes the following defaults:

• branch: master
• remote: origin
• source: github
• protocol: ssh

The sources section contains all the git hosting providers. The following sources are built in to clowder:

• github: github.com
• gitlab: gitlab.com
• bitbucket: bitbucket.org

So the previous clowder.yml can be simplified to:

name: cool-projects

sources:
  google:
    url: gerrit.googlesource.com
    protocol: https

projects:
  - name: llvm/llvm-project
  - name: apple/swift
  - name: tensorflow/tensorflow
  - name: git-repo
    path: repo
    source: google

A project requires a name, the path component of the git clone url. This is combined with defaults.protocol or sources.protocol to form the full git clone url, taking the form of git@${sources.url}:${projects.name}.git or https://${sources.url}/${projects.name}.git. If path is not specified, the last component of the name is used for the local directory.

In order to be able to run commands for only certain sets of projects, there are groups:

name: cool-projects

sources:
  google:
    url: gerrit.googlesource.com
    protocol: https

projects:
  - name: llvm/llvm-project
    groups: [notdefault, clattner]
  - name: apple/swift
    groups: [clattner]
  - name: tensorflow/tensorflow
    groups: [google, clattner]
  - name: git-repo
    path: repo
    source: google
    groups: [google]

Projects are automatically added to the all group, a group of their name, and a group of their path. If notdefault is specified, the project will not be included in commands unless another group argument is given that it belongs to.

For some more custom examples, see:

• Cats clowder.yml example
• Forks clowder.yml example

Command Usage

For the full command reference, see the commands doc

The following examples use an existing repo containing a clowder.yml file.

First, create a directory where all the projects will be cloned:

mkdir cats
cd cats

clowder init

clowder init [e-mail]:JrGoodle/clowder-examples.git

The clowder init command does the following:

• Clones the examples clowder repo in the .clowder directory
• Creates a symlink in the cats directory: clowder.yml -> .clowder/clowder.yml

clowder herd

The clowder herd command updates the state of the projects. When the command is run, the following happens:

• If any projects don't have a clean git status then clowder exits
• Projects are cloned if they don't currently exist
• Each project fetches the latest changes
• If the current git ref checked out doesn't match the clowder.yml configuration, the correct ref will be checked out
• The latest changes are pulled for branches. For commits and tags, the commits are checked out into a detached HEAD state

clowder status

The clowder status command prints the current state of all projects.

clowder forall

clowder forall -c 'git status' # Run command in all project directories

git commands

For more information, see the commands doc

clowder branch # Print all local branches
clowder checkout 'my_branch' # Checkout 'my_branch' in projects if it exists
clowder clean # Discard any changes in projects
clowder diff # Print git diff for all projects
clowder start 'my_feature' # Create new branch 'my_feature' for all projects
clowder stash # Stash changes in all projects
clowder prune 'stale_branch' # Prune branch 'stale_branch' for all projects

clowder repo commands

For more information, see the commands doc

clowder link 'v0.1' # Set clowder.yml symlink to a previously saved version
clowder repo run 'git status' # Run command in .clowder directory
clowder save 'v0.1' # Save a version of clowder.yml with current commit sha's

config commands

NOTE: EXPERIMENTAL

For more information, see the commands doc

clowder config get # Get config values
clowder config set projects 'my_group' # Set config values
clowder config clear projects # Clear config values

Development

See CONTRIBUTING.md for information on setting up your environment for development and contribution guidelines