gitlabform 2.11.1rc2

Specialized "configuration as a code" tool for GitLab projects, groups and more using hierarchical configuration written in YAML

GitLabForm is a specialized "configuration as a code" tool for GitLab projects, groups and more using hierarchical configuration written in YAML.

Used by

(Feel free to create PR to add your company logo/name here!)

Features

GitLabForm enables you to manage:

• Group:

  • Badges,
  • Members (users) {add/remove user, change access level, optional enforce},
  • Members (groups) {share/unshare with group, change access level, optional enforce},
  • Members using LDAP Group Links (GitLab Premium (paid) only),
  • CI/CD variables,
  • Settings,

• Project:

  • Archive/unarchive,
  • Badges,
  • Protected branches:
    • access levels (roles) allowed to push/merge/unprotect, allow force push flag,
    • users/groups allowed to push/merge/unprotect, code owner approval required flag (GitLab Premium (paid) only),
  • Deploy keys,
  • Files {add, edit or delete}, with templating based on Jinja2 (now supports custom variables!),
  • Hooks,
  • Members (users) {add/remove user, change access level, optional enforce},
  • Members (groups) {share/unshare with group, change access level, optional enforce},
  • Merge Requests approvals settings and approvers (GitLab Premium (paid) only),
  • Pipeline schedules,
  • Push Rules (GitLab Premium (paid) only),
  • CI/CD variables,
  • Integrations,
  • Settings,
  • Tags {protect/unprotect},

...for:

• all projects in your GitLab instance/that you have access to,
• a group/subgroup of projects,
• a single project,

...and a combination of them.

GitLabForm uses hierarchical configuration with inheritance, merging/overwriting and addivity. GitLabForm is also using passing the parameters as-is to GitLab APIs with PUT/POST requests.

Limitations

Some app features are limited because of the GitLab API issues. Here is the list of them. Please check the links to the GitLab issue(s) in their comments and upvote them if they are affecting you. Note that these issues/bugs affect all the apps using GitLab API, not just GitLabForm.

Comparison to similar apps

GitLabForm has roughly the same purpose as GitLab provider for Terraform (which is a tool that we love and which has inspired us to write this app), but it has a different set of features and uses a different configuration format.

Please read more about GitLab provider for Terraform vs GitLabForm. This article includes a link to the feature matrix / comparison sheet between these two tools.

To configure your GitLab instance itself (appearance, application settings, features, license) please check out the GitLab Configuration as Code (GCasC) project!

Requirements

• Docker or Python 3.6-3.10 with pipx (recommended) or pip,
• GitLab 11+
• GitLab Premium (paid) for some features

Installation

Run the latest stable version with:

• Docker: docker run -it -v $(pwd):/config ghcr.io/gdubicki/gitlabform:latest gitlabform.
• pipx: pipx run gitlabform

Run the latest 2.* version with:

• Docker: docker run -it -v $(pwd):/config ghcr.io/gdubicki/gitlabform:2 gitlabform.
• pipx: pipx run --spec gitlabform~=2.2.0 gitlabform

See this for all available Docker tags.

Installing with:

• pipx: pipx install gitlabform (recommended)
• plain pip: pip3 install gitlabform

Quick start

Let's assume that you want to add a deployment key to all projects in a group "My Group" (with path "my-group"). If so then:

1. Create example config.yml:

config_version: 2

gitlab:
  # You can also set in your environment GITLAB_URL
  url: https://gitlab.yourcompany.com
  # You can also set in your environment GITLAB_TOKEN
  token: "<private token of an admin user>"

projects_and_groups:
  my-group/*:
    deploy_keys:
      a_friendly_deploy_key_name:  # this name is only used in GitLabForm config
        key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3WiHAsm2UTz2dU1vKFYUGfHI1p5fIv84BbtV/9jAKvZhVHDqMa07PgVtkttjvDC8bA1kezhOBKcO0KNzVoDp0ENq7WLxFyLFMQ9USf8LmOY70uV/l8Gpcn1ZT7zRBdEzUUgF/PjZukqVtuHqf9TCO8Ekvjag9XRfVNadKs25rbL60oqpIpEUqAbmQ4j6GFcfBBBPuVlKfidI6O039dAnDUsmeafwCOhEvQmF+N5Diauw3Mk+9TMKNlOWM+pO2DKxX9LLLWGVA9Dqr6dWY0eHjWKUmk2B1h1HYW+aUyoWX2TGsVX9DlNY7CKiQGsL5MRH9IXKMQ8cfMweKoEcwSSXJ
        title: ssh_key_name_that_is_shown_in_gitlab
        can_push: false

2. Run gitlabform my-group

3. Watch GitLabForm add this deploy key to all projects in "My Group" group in your GitLab!

Full configuration syntax

See config.yml in this repo as a well documented example of all the features, including configuring all projects in all groups, projects in "my-group" group and specifically project "my-group/my-project1".

More cli usage examples

To apply settings for a single project, run:

gitlabform my-group/my-project1

To apply settings for a group of projects, run:

gitlabform my-group

To apply settings for all groups of projects and projects explicitly defined in the config, run:

gitlabform ALL_DEFINED

To apply settings for all projects, run:

gitlabform ALL

Run:

gitlabform -h

...to see the current set of supported command line parameters.

Running in an automated pipeline

You can use GitLabForm as a part of your CCA (Continuous Configuration Automation) pipeline.

You can run it with a schedule on ALL_DEFINED or ALL projects to unify your GitLab configuration, after it may have drifted from the configuration. For example you may allow the users to reconfigure projects during their working hours but automate cleaning up the drift each night.

An example for running GitLabForm using GitLab CI is provided in the .gitlab-ci.example.yml file.

Note that as a standard best practice you should not put your GitLab access token in your config.yml (unless it is encrypted) for security reasons - please set it in the GITLAB_TOKEN environment variable instead.

For GitLab CI a secure place to set it would be a Secret/Protected Variable in the project configuration.

Running automatically for new projects

We have documented some methods of automating running GitLabForm for newly created GitLab projects here.

Contributing

Please see the contribution guide for info about all kinds of contributions, like:

• questions, feature requests,
• documentation and code contributions,
• other.

Contribution guide is also the place to look for info how to develop the app locally, build it, run the tests and learn about the code guidelines.

For detailed info about how the app code has been organized, where is what and where and how to fix bugs and/or add new features, please see the implementation design article.

Origins

This tool has been originally created at Egnyte as a workaround for missing GitLab features such as assigning deploy keys per project groups but as of now we prefer to use it ever if there are appropriate web UI features, such as secret variables per project groups (released in GitLab 9.4) to keep the configuration as code.

Later on we added features that allowed us to use GitLabForm to improve a group containing around 100 similar projects to move to a unified development flow (by managing branches protection and the Pull Requests configuration), basic tests and deployment process (by managing secret variables, deployment keys and files, such as .gitlab-ci.yml), integrations (such as JIRA or Slack) and more.

Legal

The app code is licensed under the MIT license. A few scripts in dev/ directory are licensed under the MPL 2.0 license.

GitLab is a registered trademark of GitLab, Inc. This application is not endorsed by GitLab and is not affiliated with GitLab in any way.

The GitLabForm logo is based on the GitLab logos available here, and like the original art is licensed under the Creative Commons Attribution Non-Commercial ShareAlike 4.0 International License.

All the logos shown in the "Used by" section below to their respective owners.

Changelog

2.11.1rc2

• Fix another case of GitLab's Schedules API change/bug. Fixes #364.

2.11.0

• Allow breaking configuration inheritance. Implements #326. PR #339.

Imagine you have a configuration like this:

projects_and_groups:
  my-group/*:
    members:
      enforce: true
      groups:
        regular-developers:
          group_access: developer
 
  my-group/special-private-project:
    members:
      inherit: false # <--- the new keyword
      enforce: true
      groups:
        special-developers:
          group_access: developer
      users:
        john:
          access_level: maintainer

With the new inherit: false entry used here, the effective members for my-group/special-private-project project are ONLY the special-developers grup and john user.

• Always expect pagination for GETs. Fixes #354. PR #358.
• Workaround for the GitLab's Schedules API change/bug. Fixes #361.

Big thanks to the Elastic Path team for their contribution of the above feature, especially @ep-linden for the whole implementation and @amimas for the initial proposal and cooperation on design!

(There were 4 pre-releases of this version, 2.11.0b1-b4. b1 contained "Allow breaking configuration inheritance", b2 also #358, b3 also a fix of a bug that caused breaking config inheritance to not work in some cases, b4 - #361).

2.10.1

• Fix a problem causing some changes to not be applied. Fixes #334. PR #350.

2.10.0

• Deleting deploy keys is now possible. This partially implements #193. Also completely replace the implementation of deploy keys, secret variables and group secret variables with a new universal one. This possibly fixes #19.
• Up to 40% faster thanks to making less requests to GitLab. (For almost 1000 repositories the apply time has dropped from ~18 minutes to ~11 minutes.)
• With debug enabled a lot less duplication and a more readable output of dicts (shown as JSONs).
• For Contributors Introduce SingleEntityProcessor that generalizes editing things that are single per project, f.e. settings or push rules set. It does not edit entities if there are not changes to be applied. Using it implementing new features can be superfast! (See also MultipleEntitiesProcessor added in v2.2.0).
• For Contributors Faster tests and improved usage of fixtures.

(A pre-release of 2.9.2 RC1 contained some of the above changes. 2.10.0rc1 pre-release was the same as 2.10.0 final but with a slightly different changelog - the speed gains turned out to be higher than expected.)

2.9.1

• Fix version 2.9.0 not even starting. 🤦‍♂️ Add test to prevent this from happening again.

2.9.0

• Access level names (not only their numbers) are now accepted in the configuration.
• More strictness in parsing configuration YAML and better error messages thanks to a switch to different libraries. F.e. hash keys overwriting is not accepted anymore.
• Moved the project from egnyte/gitlabform to gdubicki/gitlabform and switch to GitHub Registry as the new main Docker registry.

2.8.1

• Don't show "Warning: Using group_shared_with: is deprecated" although the user is not really using this config syntax.

2.8.0

• Complete support for managing groups and projects members:
  • Add enforcing (direct) project members - groups and users, including being able to remove all direct members and keep only the members inherited from the group. Implements #100.
  • Unify the configuration syntax for group and project level membership.

✨ New ✨ config syntax example:

projects_and_groups:
  foo/*:
    # below key now includes what used to be under 
    # `group_shared_with` and `enforce_group_members` keys
    group_members:
      # there are only up to 3 direct keys below
      groups:
        another-group:
          # below key's name been changed to the name used in projects `members`
          # for groups (and the same as in the API to share group with group)
          group_access: 30
      users:
        my-user:
          access_level: 50 # owner
      # this will enforce group-level users to be ONLY as defined above
      enforce: true

    # this will make the projects in `foo` group not contain any **direct** users or groups other
    # (so it will make it contain only the ones inherited from the group `foo`)
    members:
      enforce: true

The 🏚 old and deprecated 🏚 syntax for a similar* config would be:

projects_and_groups:
  foo/*:
    group_shared_with:
      groups:
        another-group:
          group_access_level: 30
    group_members:
      my-user:
        access_level: 50 # owner
    enforce_group_members: true

    # !!! * - there was no enforce project members support before v2.8.0 !!!

The old syntax works but will generate warnings. Support for it will be removed in one the future major GitLabForm versions.

2.7.1

• Speed up running for ALL_DEFINED, when the defined groups and projects for just a small part of all the GitLab instance's groups and projects. Additionally always show the number of omitted groups and projects for any reasons (no config, archived, skipped). Fixes #285.

2.7.0

• Allow processing only requested configuration sections using a new cli argument -os / --only-sections.
• Minimize the number of unnecessary audit branch unprotect/protect events. Up to now every apply of the files section for protected branch resulted in unprotect and then (re)protect event for each protected branches and each file. Now this will only happen when the user running GitLabForm actually needs to do that, which should not happen often if you are using an admin account. Completely fixes #178.

2.6.0

• Complete support for Protected branches - access levels / users / groups allowed to push/merge/unprotect (GitLab Premium (paid) only). PR #289.
• Add option to allow push force in protected branches. Implements #227.
• Fix a bug causing the app to get HTTP 502 from GitLab when protecting branches in some cases.
• Fix getting members list to include usernames of all direct members not just the first page. PR #284.

Big thanks to the contributors of this release: @trissanen

2.5.0

• Make commit messages for file operations configurable. Implements #278.

Thanks to @aleung for his contribution!

2.4.0

• Add wildcard support for skip_groups and skip_projects. Implements #275 and #276.

Thanks to @chris-workingmouse for his contribution!

2.3.0

• Add Protected branches - users allowed to push/merge (GitLab Premium (paid) only), PR #273.
• For ALL_DEFINED also skip archived projects even if they are explicitly defined in the config, unless -a flag is added - for consistency.
• For Contributors Add docs for running the test themselves in a Docker container and for running GitLab in Docker using a license file, for testing paid-only features.

Thanks to @florentio, @barryib and @Pigueiras for their contribution!

2.2.0

• Add LDAP Group Links support (GitLab Premium (paid) only). Implements #140.

• Add project and group badges support. Implements #59.

• Allow 0 (no access) in Protected Tags. Fixes #172.

• Exit on configuration missing projects_and_groups key. This will provide a helpful error message for typos made in this key. Fixes #242.

• Make error messages more friendly when there is no network connection or when configuration is invalid (f.e. YAML parsing errors).

• Make the output of some processors a bit more consistent.

• Fix detecting an "empty effective config" and improve the UI related to processing groups and projects with such. Fixes #251.

• Big refactoring that should make adding new features easier and faster. The main change is introducing a new way to implement "processors" - thanks to a generalized MultipleEntitiesProcessor class adding a new feature like Project Badges should is now as easy as implementing a class like BadgesProcessor and writing an acceptance test like TestBadges. Note that this new design may change in the near future and we are open to discussions and PRs to make it even better! We also plan to create a similar generalized SingleEntityProcessor class soon.

• Change the User Agent that the app uses when making requests to GitLab to a custom GitLabForm/<gitlabform_version> (python-requests/<requests_version>).

2.1.2

• Managing project members is not incredibly slow anymore. Fixes #240

Thanks to @andrewjw (Ocado Technology) for his contribution!

2.1.1

• Fixed sharing group with a subgroup. Fixes #236
• Improved re-protecting branches after updating files in them. Fail fast if the config is invalid.
• Better Docker images:
  • Updated Alpine from 3.12 to 3.14,
  • Started to build images in ARM64 architecture (apart from x86-64),
  • Started to add tags <major_version>, <major_version>.<minor_version>. Note that Alpine-based image is the main one which gets these tags. For Debian-based images add "-buster" suffix. Implements #173

Thanks to @andrewjw (Ocado Technology) for his contribution!

2.1.0

• Added a feature to share groups with other groups, with optional enforcing. Implements #150

Thanks to @andrewjw (Ocado Technology) for this contribution!

2.0.6

• Fixed incorrect subgroups list when requesting to process ALL_DEFINED. Completes the fix for #221

2.0.5

• Really fixed issue with unprotect_branch_new_api. Fixes #219
• Fixed call to a Merge Requests Approvers API endpoint removed in GitLab 13.11.0. Fixes #220
• Fixed potential security issue by enabling autoescaping when loading Jinja templates. (Bandit security tool issue B701)

Thanks to @Pigueiras for his contribution!

2.0.4

• Fixed issue with Push Rules when the project name contains a dot. Fixes #224
• Fixed calling to process a single subgroup (like: gitlabform 'group/subgroup'). Fixes #221

2.0.3

• Fixed issue with dry-run for Project Push Rules when the current config is empty (None). Fixes #223

2.0.2

• Fixed issue with unprotect_branch_new_api. Fixes #219 (update: later it turned out that it was not really fixed in 2.0.2 but in 2.0.5 instead)

2.0.1

• Fixed issues with Jinja loader.
• Fixed calls to GitLab API that do not contain 'x-total-pages' header (gradually rolled out since GitLab MR #23931).
• Start showing deprecation warning when using the old branch protection API config syntax.

Thanks to @mkjmdski for his contribution!

(2.0.0post1-3 release is technically the same as 2.0.1 but was incorrectly versioned.)

2.0.0

(For a detailed info about changes in each RC of v2 please see the previous version of this changelog.)

• Make deep merging of configuration actually work (breaking change). Fixes #197 (RC5)

• Introduce config versioning (breaking change). ...or rather a change to avoid breakage. New major releases of GitLabForm starting with v2 will look for config_version key in the config file. If it doesn't exist, or the version does not match expected then the app will exit to avoid applying unexpected configuration and allowing the user to update the configuration. (RC1)

• New config syntax (breaking change). All 3 levels under a common key projects_and_groups. It should contain a dict, where common config is under a special "*" key, group configs under keys like group/* and project configs under keys like group/project. This will allow introducing pattern matching in these keys and introducing support for multiple config files in the future releases. Partially implements #138. (RC1)

• Exit with code != 0 when any group/project processing was failed (breaking change). This will allow you to notice problems when running the app from CI. Note that you can restore the old behaviour by running the app with (...) || true. Also standardized exit codes. Exit with 1 in case of input error (f.e. config file does not exist), with 2 in case of processing error (f.e. GitLab returns HTTP 500).Fixes #153. (RC1)

• Allow any case in groups and projects names (breaking change). GitLab groups and projects names are case sensitive but you cannot create such entities that differ only in the case. There is also a distinction between a "name" and a "path" and they may differ in case... To make work with this easier GitLabForm now accepts any case for them in both config files as well as when provided as command line arguments. We also disallow such entities that differ only in case (f.e. group/* and GROUP/*) to avoid ambiguity. Fixes #160. (RC2)

• Ignore archived projects by default (breaking change). This makes processing faster and output shorter. You can restore the previous behavior by adding --include-archived-projects/-a command line switch. Note that you have to do it if you want to unarchive archived projects! Fixes #157 in (arguably) a more convenient way. (RC2)

• Color output! Implements #141. (RC1)

• Add diffing feature for secret variables. (with values shown as hashes to protect the secrets from leaking). (RC6)

• Added checking for invalid syntax in "members" section. Defining groups or users directly under this key instead of under sub-keys "users" and "groups" will now trigger an immediate error. (RC5)

• Add support for Python 3.9 (RC8)

• Added Windows support. Fixes #206 (RC5)

• Start processing at any group using the new command line switch - --start-from-group/-sfg. Similar to --start-from/-sf switch that can be used for projects. (RC1)

• Start releasing pre-releases as Docker images. They have tags with specific versions, but not "latest" tag as it is reserved for new final releases. Implements #201 (RC5)

• Prevent multiple email notifications from being sent when adding members to project. Fixes #101 (RC6)

• Prevent project's Audit Events being filled in with "Added protected branch". Fixes #178 (RC6)

• Fixed using "expires_at" for users. Fixes #207 (RC6)

• Remove the need to add the gitlab.api_version configuration key. (RC1)

• For Contributors Make writing tests easier and the tests more robust. Deduplicate a lot of the boilerplate code, allow putting configs into the test methods and use pytest fixtures for easier setup and cleanup. This should fix issues with tests reported in #190. Also stop storing any dockerized GitLab data permanently to avoid problems like #196 and probably other related to failed dockerized GitLab upgrades. (RC3)

• For Contributors Rename "integration tests" to "acceptance tests". Because they ARE in fact acceptance tests. (RC3)

Thanks to @amimas, @weakcamel, @kowpatryk, @ss7548, @houres, @Pigueiras and @YuraBeznos for their contributions!

before 2.0.0

Please see GitHub pre-2.0 releases' descriptions.