Skip to main content
License: AGPL-3 https://github.com/acsone/git-aggregator/actions/workflows/ci.yml/badge.svg pre-commit.ci status https://img.shields.io/pypi/pyversions/git-aggregator

git-aggregator

Manage the aggregation of git branches from different remotes to build a consolidated one.

Configuration file

Create a repos.yaml or repos.yml file:

./product_attribute:
    remotes:
        oca: https://github.com/OCA/product-attribute.git
        acsone: git+ssh://git@github.com/acsone/product-attribute.git
    merges:
        - oca 8.0
        - oca refs/pull/105/head
        - oca refs/pull/106/head

./connector-interfaces:
    remotes:
        oca:  https://github.com/OCA/connector-interfaces.git
        acsone:  https://github.com/acsone/connector-interfaces.git
    merges:
        - oca 6054de2c4e669f85cec380da90d746061967dc83
        - acsone 8.0-connector_flow
        - acsone 80_connector_flow_ir_cron_able-lmi
        - acsone 8.0_connector_flow_improve_eval_config
    target: acsone aggregated_branch_name
    fetch_all:
        - oca

./gitlab-repo-example:
    remotes:
        gitlab_1: https://giltab.com/sample-org/sample-repo.git
        gitlab_2: https://giltab.com/another-sample-org/sample-repo-fork.git
    merges:
        - gitlab_1 main
        - gitlab_2 merge-requests/123/head
target: gitlab_1

# FYI: Bitbucket doesn't support fetching PR's
./bitbucket-repo-example:
    remotes:
        bitbucket_1: https://bitbucket.org/sample-org/sample-repo.git
        bitbucket_2: https://bitbucker.org/another-sample-org/sample-repo-fork.git
    merges:
        - bitbucket_1 main
        - bitbucket_2 dev
target: bitbucket_1

Environment variables inside of this file will be expanded if the proper option is selected.

All the merges are combined into a single branch. By default this branch is called _git_aggregated but another name may be given in the target section.

Fetching only required branches

If any of your merges refer to a specific commit, you will probably need to fetch all remotes from the corresponding remote or use any other strategy to get that fetch working, but we recommend to simply add this like in the example above:

fetch_all:
    - oca
    - other-remote

You can specify that you want to fetch all references from all remotes you have defined with:

fetch_all: true

Shallow repositories

To save big amounts of bandwidth and disk space, you can use shallow clones. These download only a restricted amount of commits depending on some criteria. Available options are depth, shallow-since and shallow-exclude.

You can use those in the defaults sections to apply them everywhere, or specifying them in the corresponding merges section, for which you must use the dict alternate construction. If you need to disable a default in merges, set it to false:

./odoo:
    defaults:
        depth: 20
    remotes:
        odoo: https://github.com/odoo/odoo.git
        ocb: https://github.com/OCA/OCB.git
        acsone: https://github.com/acsone/odoo.git
    merges:
        -
            remote: ocb
            ref: "9.0"
            depth: 1000
        -
            remote: odoo
            ref: refs/pull/14859/head
    target: acsone 9.0

./gitlab-repo-example:
    defaults:
        depth: 20
    remotes:
        gitlab_1: https://giltab.com/sample-org/sample-repo.git
        gitlab_2: https://giltab.com/another-sample-org/sample-repo-fork.git
    merges:
        -
            remote: gitlab_1
            ref: main
            depth: 20
        -
            remote: gitlab_2
            ref: merge-requests/123/head
    target: gitlab_1

# FYI: Bitbucket doesn't support fetching PR's
./bitbucket-repo-example:
    remotes:
        bitbucket_1: https://bitbucket.org/sample-org/sample-repo.git
        bitbucket_2: https://bitbucker.org/another-sample-org/sample-repo-fork.git
    merges:
        -
            remote: bitbucket_1
            ref: main
            depth: 3
        -
            remote: bitbucket_2
            ref: dev
    target: bitbucket_1

Remember that you need to fetch at least the common ancestor of all merges for it to succeed.

Triggers

It’s also possible to specify a command or a list of shell commands to execute after the aggregation (and before the push). The commands are executed into the aggregated directory.

./product_attribute:
    remotes:
        oca: https://github.com/OCA/product-attribute.git
        acsone: git+ssh://git@github.com/acsone/product-attribute.git
    merges:
        - oca 8.0
    target: acsone aggregated_branch_name
    shell_command_after: echo 'my command'

./connector-interfaces:
    remotes:
        oca:  https://github.com/OCA/connector-interfaces.git
        acsone:  https://github.com/acsone/connector-interfaces.git
    merges:
        - oca 9.0
    target: acsone aggregated_branch_name
    shell_command_after:
        - echo 'a first command'
        - echo 'a second command'

A real life example: applying a patch

./odoo:
    remotes:
        oca: https://github.com/OCA/OCB.git
        acsone: git@github.com/acsone/OCB.git
    merges:
        - oca 9.0
    target: acsone aggregated_branch_name
    shell_command_after:
        - git am "$(git format-patch -1 XXXXXX -o ../patches)"

Command line Usage

Following the example repos.yaml file from above, aggregate your repositories at any time:

$ gitaggregate -c repos.yaml

Expand environment variables inside of the configuration file when loading:

$ gitaggregate -c repos.yaml --expand-env

The variables in the configuration file can be specified in one of the following ways:

  • $VARIABLE

  • ${VARIABLE}

For more information, see the Python’s string.Template documentation.

Use additional variables from file while expanding:

$ gitaggregate -c repos.yaml --expand-env --env-file .env

The env file should contain VAR=value lines. Lines starting with # are ignored.

You can also aggregate and automatically push the result to the target, if the target option is configured:

$ gitaggregate -c repos.yaml -p

Only aggregate a specific repository using fnmatch:

$ gitaggregate -c repos.yaml -p -d connector-interfaces

Show github pull requests

gitaggregate has a mechanism to identify merges that correpond to merged or closed Github pull requests.

Such merges are of the form refs/pull/NNN/head where NNN is the pull request number, with a https://github.com or git@github.com remote.

To work around API limitation, you must first generate a Github API token.

$ export GITHUB_TOKEN=...
$ gitaggregate -c repos.yaml show-all-prs
$ gitaggregate -c repos.yaml show-closed-prs

Changes

4.1 (2025-03-25)

  • Update README for other forges than GitHub (#88)

  • Internal API change (#91)

  • Drop support for unsupported Python versions (#92)

4.0.2 (2024-10-23)

  • target_dir may be an empty directory (#83)

4.0.1 (2024-06-04)

  • fix: git remote: change url instead of rm / add (#81 <https://github.com/acsone/git-aggregator/pull/81>`_)

4.0 (2023-07-22)

  • [BREAKING] drop support for other configuration file formats than yaml

  • Ensure git pull is always done in fast-forward mode

  • Drop support for python 3.6, test with python 3.11, stop testing with pypy

3.0.1 (2022-09-21)

  • Fix git clone issue with git < 2.17

3.0.0 (2022-09-20)

  • When updating remotes the log message now states Updating remote instead of Remote remote

  • Add --no-color option to disable colored output

  • Use git clone –filter=blob:none + fetch strategy to improve performance and benefit from git-autoshare if installed

2.1 (August 26, 2021)

2.0 (August 17, 2021)

  • Drop support for python < 3.6

  • Do not exit with success on KeyboardInterrupt

  • Make target optional.

1.8.1 (August 28, 2020)

  • Support environment variables in the configuration file.

1.7.1 (September 30, 2019)

  • If an error happens, log in which repo it happens. Helpful when running in parallel.

1.7.0 (August 14, 2019)

  • Fix a bug in --show-all-prs, which was printing a wrong PR URL.

  • Display PR labels too in --show-all-prs.

1.6.0 (March 04, 2019)

  • Add –show-all-prs command to list all GitHub pull requests used in merge sections.

1.5.0 (December 07, 2018)

  • Add –force. If set, dirty repositories will fail to aggregate.

1.4.0 (September 13, 2018)

  • Add –jobs option for multi-process operation.

1.3.0 (August 21, 2018)

  • Improve configuration file parsing by mimicing Kaptan’s behavior of resolving handler by extension (#22)

1.2.1 (July, 12, 2018)

  • show-closed-prs now displays merge status

  • some documentation improvements

1.2.0 (May, 17, 2017)

  • support .yml config file extension

  • add a show-closed-prs command to display github pull requests that are not open anymore; github pull requests must be referenced as refs/pull/NNN/head in the merges section

1.1.0 (Feb, 01, 2017)

1.0.0 (Jan, 19, 2016)

  • First release

Credits

Author

Contributors

Maintainer

ACSONE SA/NV

This project is maintained by ACSONE SA/NV.

Release files for git-aggregator 4.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for git-aggregator 4.1
File Size Uploaded
git_aggregator-4.1.tar.gz 38.8 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for git-aggregator 4.1
File Interpreter ABI Platform
git_aggregator-4.1-py3-none-any.whl Python 3 none any Details

Total release size: 69.6 kB

Release files / git_aggregator-4.1.tar.gz

Download URL git_aggregator-4.1.tar.gz
Size 38.8 kB
Tags Source
SHA-256 checksum
How to use checksums
ebf54afce59d5f4bd74686e3f14849cada3292b9358cdfee85ffec38689302f0
BLAKE2b-256 checksum
How to use checksums
237845b21909788559942eb5b7df6edd0cbd1769da2d9cf27338edbe1295982c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Mar 25, 2025.

Transparency log

Release files / git_aggregator-4.1-py3-none-any.whl

Download URL git_aggregator-4.1-py3-none-any.whl
Size 30.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
60aa52f4f6649e0cab6cb88e0be779d1ba64a97bcc821324bf7459f8b906bb21
BLAKE2b-256 checksum
How to use checksums
fe7eee622162b4790372066a79471456b863a17c27c821bda53f8326f0bf30b1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Mar 25, 2025.

Transparency log

Release history Release notifications | RSS feed

This release

4.1 This release

2 release files

4.0.2

2 release files

4.0.1

2 release files

4.0

2 release files

3.0.1

2 release files

3.0.0

2 release files

2.1.0

2 release files

2.0.0

2 release files

1.8.1

2 release files

1.8.0

2 release files

1.7.2

2 release files

1.7.1

2 release files

1.7.0

2 release files

1.6.0

2 release files

1.5.0

2 release files

1.4.0

2 release files

1.3.0

2 release files

1.2.1

2 release files

1.2.0

2 release files

1.1.0

2 release files

1.0.0

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page