otterdog 0.4.0

Tool to manage GitHub organizations and their repositories.

Otterdog

Introduction

Otterdog is a tool to manage GitHub organizations at scale using a configuration as code approach. It is actively developed by the Eclipse Foundation and used to manage its numerous projects hosted on GitHub.

Quickstart

Otterdog Presentation @ Open Source Summit 2023

Default Configuration used @ Eclipse Foundation

Documentation

The documentation is available at otterdog.readthedocs.io.

Build instructions

System requirements:

• python3.10 (mandatory): install using apt install python3
• poetry (mandatory): install using curl -sSL https://install.python-poetry.org | python3 - or pipx install poetry
• go (mandatory for installing jsonnet-bundler): install using apt install golang
• jsonnet-bundler (mandatory): install using go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@v0.5.1
• bitwarden cli tool (optional): install using snap install bw
• pass cli tool (optional): install using apt install pass

Building Steps

• Create a virtual python environment and install necessary python dependencies using poetry:

$ make init

• Testing build

$ ./otterdog.sh -h

Setup

The general configuration for supported organizations and their corresponding credentials in order to access their GitHub settings has to be placed in a json file (default: otterdog.json, can be changed with the -c flag):

{
  ...
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "<bitwarden | pass>",
        ... // provider specific data
      }
    }
  ]
  ...
}

Credentials

Otterdog needs certain credentials to access information from an organization and its repositories on GitHub:

• username / password / 2FA seed
• API token

The login / username / 2FA seed are required to access the web interface of GitHub in order to retrieve certain settings that are not accessible via its rest / graphql API.

The GitHub api token needs to have the following scopes enabled:

• repo
• workflow
• admin:org
• admin:org_hook
• delete_repo

The credentials can be stored in different providers (bitwarden, pass).

Bitwarden

When using bitwarden to store the credentials, you need to enter a valid item id as additional credential data:

{
  ...
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "bitwarden",
        "item_id" : "<bitwarden item id>"
      }
    }
  ]
  ...
}

The item stored in bitwarden needs to contain the following information (a sample json output of such an item):

{
  "object": "item",
  "id": "<bitwarden item id>",
  "name": "<item name>",
  "fields": [
    {
      "name": "api_token_admin",
      "value": "<github API token>"
    }
  ],
  "login": {
    "username": "<github username>",
    "password": "<github password>",
    "totp": "<github TOTP text code>"
  }
}

Mandatory items:

• Field with name "api_token_admin" and as value the GitHub token to access the organization
• login.username of a user that can access the organization with enabled 2FA
• login.password the password of that user
• login.totp the TOTP text code

Pass

When using pass to store the credentials, you need to enter fully qualified pass names to access the various required credential data:

{
  ...
  "organizations": [
    {
      "name": "<org name>",
      "github_id": "<github org id>",
      "credentials": {
        "provider": "pass",
        "api_token": "<path/to/api_token>",
        "username": "<path/to/username>",
        "password": "<path/to/password>",
        "2fa_seed": "<path/to/2fa_seed>"
      }
    }
  ]
  ...
}

In case your password storage dir is not located at the default location, you can configurate that in the defaults:

{
  "defaults": {
    "pass": {
      "password_store_dir": "path/to/storage/dir"
    }
  }
}

Usage

Run the import operation to retrieve the current live configuration for an organization:

$ otterdog.sh import <organization>

The created configuration file for the organization can be found at <data-directory>/orgs/<organization>.jsonnet

Run the plan operation to highlight differences between the live configuration and the written configuration:

$ otterdog.sh plan <organization>

Run apply operation to reflect the written configuration on github itself:

$ otterdog.sh apply <organization>

Container Runtime (Linux/MacOS)

Requirements

• An otterdog.json already in your current directory
• (Recommended) a directory orgs

Building Container Image

• Creating a container local image

make container_build

Building a Development Container Image

• Creating a container development local image

make container_build_dev

Running otterdog in a container

Using Bitwarden client

• Firstly you need to login and unlock your Bitwarden session by executing the command below

bw login
bw unlock

• As result, you will get a token session. Please follow example below to make available in your terminal (Linux/MacOS)

export BW_SESSION="ffdsajklloremipsumfxs000f000r0loremipsum"

Using pass client

• Pass needs to be already installed as well as configured with all data needed in otterdog.json by executing pass in your profile

Activating Otterdog Container Runtime

• Activate otterdog environment will create an alias otterdog

source scripts/bin/active-otterdog

• Checking otterdog alias

otterdog -h

• Deactivating otterdog environment

deactivate-otterdog

otterdog container Arguments

• An table arguments

Argument	Description
-g	.gnupg directory PATH by default $HOME/.gnupg if it is not provided
-o	Output ORGS directory path by default $PWD/orgs if it is not provided
-c	otterdog json file path by default $PWD/otterdog.json if it is not provided
-p	.password-store directory PATH by default $HOME/.password-store if it is not provided
-h	Help about container arguments and otterdog

• Please find below an example to show configuration

otterdog -c $PWD/otterdog.json -g $HOME/.gnupg -o $PWD/orgs apply -f

Usage Otterdog Container Runtime

• Please follow the section Usage
• Please bear in mind that all command need to drop .sh

Activating Development Otterdog Container Runtime

• Activating development otterdog environment will run eclipse/otterdog:dev. Thus it can be used container shell. Please find below an example

export OTTERDOG_DEV=1; source scripts/bin/active-otterdog
otterdog /bin/bash

• To activate development otterdog environment with argument

export OTTERDOG_DEV=1; source scripts/bin/active-otterdog
otterdog -c $PWD/otterdog.json -g $HOME/.gnupg -o $PWD/orgs /bin/bash

• Checking otterdog environment

/app/otterdog.sh -h

• Deactivating development otterdog runtime type exit then deactivate-otterdog

Usage Development Otterdog Container Runtime

• Please follow the section Usage

Cleaning container environment

• Please use the macro below

make container_clean

Known issues