dflock 0.1.0

Add your description here

Dflock

Dflock is a git-complementing tool that automates the tedious parts of diff-stacking on platforms that use branch-based change-requests such as Gitlab and Github (but see snags for Github).

It is designed specifically for a workflow in which all work happens on a single branch and commits are periodically and selectively submitted to change requests.

What does dflock offer?

Dflock offers the following:

• Support for a workflow in which you never create branches
• Smoothly creating multiple change requests at once
• Flexibly stacked change requests
• Interaction via your favourite text editor

Dflock uses git under the hood and adds a thin layer of abstraction. It aims to complement git, not replace it. You can use it together with git and collaboration platforms like Github and Gitlab, although is has only been tested extensively it with Gitlab.

What does it do?

Dflock's main specialty is translating a text-based plan specifying dependencies between commits into a set of (optionally stacked) change requests.

The plan specifies how to divide these commits over one or more change requests and in what order these change requests can be integrated. It looks a bit like the interface of git interactive rebase. Here's an example of such a plan.

d0 0001b3a Update README with overview of commands
d1 1dc86f8 Add functionality to update change requests to App class
d2@d1 9ee1334 Add --update-change-request flag to push command

It contains three commits and instructs dflock to create three branches and try to cherry pick commits into them: d0 will contain commit 0001b3a, d1, will contain commits 1dc86f8, and d2 will contain commits 1dc86f8 and 9ee1334. This will be successful if the changes in d1 are independent from the changes in d0. These branches can be used to create three change requests: change requests d0 and d1 have the upstream branch as they target branch, while d2 has the d1 as its target branch. Note that d0 and d1 can be integrated in any order, but d2 requires d1 to be integrated first (or to be integrated in d1).

Dflock remembers the plan, even after more commits are added or some change requests have been integrated. Its footprint is light: the only information dflock needs is the names of the branches it created and the commits they contain.

How do I install and use it?

Install dflock by cloning the repository or using pip:

pip install dflock

Go to the repository of the project in which you want to use dflock.

To configure a custom local branch, upstream branch (learn more about the workflow to understand these concepts), and text editor for interaction, type

dfl init

Now you can start committing to your local branch. To create change requests, use

dfl plan

to bring up the integration plan.

If the branches have been created successfully, you can push them with

dfl push

On the collaboration platform you use you can now create change requests manually. To show an overview of branches and target branches created by dflock, use

dfl status --show-targets

For some platforms dflock can create change requests automatically. See this section for more information.

To learn more about how to make effective use of dflock, see the recommended workflow.

What workflow was dflock built for?

Dflock was built for a workflow in which almost all work is done on a single branch, called the local branch. This branch tracks an upstream branch, into which your work aspires to be integrated via change requests. Your local branch is usually a few commits ahead of the upstream. These commits are your local commits. Your local commits usually consist of in-progress work and work awaiting review and approval.

Development of changes

Changes are committed incrementally to the local branch. There is no need to create branches for new features. When one feature is finished, you simply continue committing to the same branch.

Submitting change requests

To create change requests, you periodically use dfl plan bring up the integration plan and selectively assign sets of commits (deltas) to change requests and specify their inter-dependencies. Based on the plan, dflock will create one ephemeral branch per change request and cherry-pick commits into it.

Importantly, ephemeral branches only exist to serve change requests. You generally do not commit to or manipulate ephemeral branches directly because dflock may delete them when they are no longer needed. This happens for example after the change request has been integrated. All changes originate from the local branch, flowing via ephemeral branches into the remote upstream.

When the change requests are ready for submission, use dfl push to push them to the remote. Dflock offers some support for automaticaly creating change requests on Gitlab or Github as described below.

Amending previous changes

When making changes to published change requests, for example to address reviewer comments, you are free to use whatever method you prefer to re-order, amend or drop local commits. After amending commits used in change requests, you can use dfl write to update the ephemeral branches. If you instead want to add updates to a change request as extra commits, you can use dfl plan to do so.

Incorporating upstream changes

Once change requests have been integrated into the upstream branch, you can update the local branch using dfl pull. This performs a git pull --rebase <remote> <upstream> and prunes ephemeral branches. The same method can be used to pull in changes integrated into the upstream by others.

The integration plan

Integration plans are a central concept in dflock. Typing dfl plan brings up a text editor showing the current integration plan. Each line of the plan consists corresponds to a commit and consists of a command, a truncated commit checksum, and the first line of the commit message.

The command starts either with d to add the commit to a delta or s to skip it. To distinguish different deltas, an optional numeric label can be given. E.g., d0 and d1 create different deltas. To specify a dependency, the @ symbol followed by a delta label is used. E.g., d1@d0 creates a delta that depends on d0.

Example plans

In the plans below, commit checksums have been replaced by the numbers 0, 1, and 3, and commit messages which would be shown by dfl plan are omitted.

Selectively added commits

This plan creates a single change request containing commit 1.

s 0 ...
d0 1 ...
s 2 ...

Multiple commits in one change-request

This plan creates a single change request containing commit 1 and 2.

s 0 ...
d0 1 ...
d0 2 ...

Non-contiguous commits in one change-request

Commits in a plan don't need to follow each other sequentially.

This plan creates a single change request containing commit 0 and 2.

d0 0 ...
s 1 ...
d0 2 ...

Fully independent change requests

This plan creates three fully independent change requests that can be integrated into the upstream branch in any order.

d0 0 ...
d1 1 ...
d2 2 ...

Fully stacked change requests

This plans creates three change requests that are "fully stacked". That is, each change request requires the previous one to be integrated first.

d0 0 ...
d1@0 1 ...
d2@1 2 ...

Syntactical rules

Since plans are edited in a text editor their syntax is designed to require a minimal number of edits, while maintaining unambiguousness.

Delta-labels can be ommitted

An omitted label also distinguishes deltas.

d 0 ...
d0 1 ...

is the same as

d0 0 ...
d1 1 ...

and

d 0 ...
d1@ 1 ...

is the same as

d0 0 ...
d1@d0 1 ...

The d prefix can be omitted when referring to deltas after @

The commands d1@d0 and d1@0 are equivalent.

Skipping a commit is the same as deleting the entire line.

s 0 ...
d1 1 ...

is the same as

d1 1 ...

The target-branch needs to be specified only once in change requests consisting of multiple commits.

d0 0 ...
d1@d0 1 ...
d1@d0 2 ...

is the same as

d0 0 ...
d1 1 ...
d1@d0 2 ...

Limitations

Currently, dflock imposes some limitations on what plans can be specified. These constraints are not inherent to the workflow and might be lifted in future versions of dflock.

A change-request cannot depend on multiple change requests.

The only way to make d2 depend on both d0 and d1 is to fully stack them:

d0 0 ...
d1@d0 1 ...
d2@d1 2 ...

A change request occurring after another change request cannot have a dependency that occurring before the dependency of the last change-request.

This is not possible:

d0 0 ...
d1@d0 1 ...
d1 2 ...

Note that a change-request with no specified dependency has an implicit dependency on the upstream.

It these situations it is always possible to use dfl remix to re-order the commits such that a valid plan can be constructed.

For example, in the plan above, if commit 2 is truly independent from the others as implied by the plan, it can be swapped with commit 1 to construct the following valid plan:

d0 0 ...
d1 2 ...
d1@d0 1 ...

How should I set up Gitlab or Github?

I found the following settings to work well with dflock's workflow:

For both Gitlab and Github, configure:

1. auto-delete change request branch ("source branch" in Gitlab, "head branch" in Github) after merge
2. only allow fast-forward merges into upstream

Setting 1) ensures that when you have B -> A -> main and you merge A, the target/base branch of B is updated to the target/base branch of A, namely main.

Setting 2) ensures that running git pull --rebase origin main will auto-detect already applied commits.

Does dflock need configuring?

Dflock comes with a default configuration. In this configuration, both the local and upstream branch are assumed to be the main branch.

You can provide custom configuration in a file called .dflock placed either in your home directory (for cross-project configuration) or the root folder of a git repository (for per-project configuration). In the latter case, it's best to add .dflock to .gitignore since it contains personal configuration.

You can generate a project-local configuration interactively by running dfl init.

Can dflock automatically create change requests?

Yes, dflock has rudimentary for automatic change-requests creation. It also provides a mechanism for extending this with custom commands.

Gitlab merge requests

To create Gitlab you have two options: By default, dfl push supports the --merge-request flag which will use git push options to trigger merge-request creation when pushing to gitlab.

Alternatively, you can add a custom integration, for example using the glab CLI. To do so, add the following to your .dflock configuration file.

[integrations.gitlab]
change-request-template=glab mr create --source-branch {source} --target-branch {target}

To use this template in dfl push, invoke it with --change-request github.

Github pull requests

For github, you can create a custom change-request integration that uses the gh CLI tool. To do so, add the following to your .dflock configuration file.

[integrations.github]
change-request-template=gh pr create --head {source} --base {target}

To use this template in dfl push, invoke it with --change-request github.

What commands does dflock support?

Plan

dfl plan

Detect and edit the current integration plan. This is operation automates the most complex parts of the workflow. It detects the integration plan from existing local branches and creates or updates ephemeral branches for change requests. Most other commands are essentially convenient shortcuts for common git commands.

Write

dfl write

Detect the current integration plan and create or update the ephemeral branches accordingly. Normally dfl plan performs this operation, but dfl write is useful when the plan has not changed but you've amended or re-ordered commits in your local branch.

Push

dfl push

Push ephemeral branches to the remote and optionally create change requests.

Pull

dfl pull

Update your local branch. Shortcut for git pull --rebase <remote> <upstream>, but performs additional branch-pruning afterwards.

Remix

dfl remix

Edit the local commit history with git interactive rebase. Shorthand for git rebase --interactive <upstream>, but performs additional branch-pruning afterwards.

Log

dfl log

Show a git log containing only local commits. Shorthand for git log <local> ^<upstream>.

Status

dfl status

Show current ephemeral branches and whether they are up to date with the remote.

Checkout

dfl checkout

Perform a git checkout on the local branch or an ephemeral branch. As a shortcut, the ephemeral-branch indices shown by dfl status can be used here, or a substring of the branch name if it resolves to a unique branch.

Reset

dfl reset

Reset the integration plan by deleting all ephemeral branches associated with the current local commits.

Are there any snags?

Integrating stacked change requests causes conflicts in Github

When creating pull requests based on the plan below, two branches are created: one containing commit 0, and one containing commits 0 and 1.

d0 0 ...
d1@d0 1 ...

After merging the first pull-request (containing commit 0), Github unfortunately does not recognize that commit 0, now added to the upstream, does not conflict with commit 0 in the second pull request and marks it as a conflict.

This is seriously hampers the usefulness of this workflow in Gitlab.

Gitlab does not have this issue.

Github does not allow pull request dependencies

A Github pull request offers no way to block merging it before another pull request has been merged. Gitlab does support this, although dflock's default automatic merge request creation mechanism does not currently populate this field.

Not pushing all deltas leads to incorrect diffs and conflicts

Consider the following plan:

d0 0 ...
d1@d0 1 ...

In our change requests we expect the diff of the first change request to show the changes in commit 0 and the diff of the second change request to show the changes in commit 1.

However if we update and push only the second change request, the commit checksum of commit 0 in d1's ephemeral branch will change. As a result the diff shown in the change request might include the changes from commit 0 because the commit in the target branch is no longer recognized as commit 0.

This happens because any update to the integration plan causes dflock to re-write the ephemeral branches. Since that process relies on cherry-picking, the commit checksums change upon each write.

Visually, upon pushing d0 and d1, the situation on the remote is as follows:

0 --> 1
|     |
d0    d1

After update commit one (creating commit 1') and cherry-picking commits 0 and 1' the following situation is created on the remote:

0
|
d0

0' --> 1'
       |
       d1

Frequent updating of change requests

Trying to avoid the issue of incorrect diffs when not pushing all deltas causes a change request to be updated each time a downstream change request is updated - even though nothing changed.

This might be undesirable for various reasons including spurious notifications sent to change request reviewers or unnecessary triggering of slow or expensive CI/CD pipelines.

Limitations of automatic change request creation

Gitlab

As already mentioned, dflock only configures the target branches and does add dependencies on other merge requests to the created merge request.

Gitlab and Github

If dependencies are changed by an updated integration plan and merge requests have already been created, the target branches of the merge requests are not updated automatically.

Why is it called dflock?

The name dflock derives from delta flock because the tool allows you to herd a flock of deltas.

Glossary

Local

The branch used for development. This could be a personal branch called <your-name>-WIP (which is safe to force-push to the remote) or it could be your local copy of main (which should obviously not be pushed to the remote).

Upstream

The remote branch into which commits on your local branch aspire to eventually be integrated. Often the main branch of the repository.

Local commits

Commits you are working on that are only on your local branch and not in the upstream. More precisely: commits reachable from the local branch, but not from upstream branch.

Integration plan

A text-based representation of a tree-like structure that instructs dflock which deltas to construct and how they depend on each other. See this section for more discussion of integration plans.

Delta

A set of changes represented by one or more commits.

Change request

A request to integrate a set of commits (a delta) into the upstream. Gitlab and github call this merge requests and pull requests respectively.

Ephemeral branch

A branch containing a delta created by dflock for a change request. It's called ephemeral because it only serves to create a change request and dflock may delete the branch when it's no longer needed.