gh-orchestrator 0.1.0

GitHub Runner Orchestrator for automated runner management

"# GitHub Orchestrator

A CLI tool for orchestrating GitHub self-hosted runners and secret management.

🚀 Getting Started

Prerequisites

• Python 3.8+

System Dependencies

Linux (Debian/Ubuntu):

sudo apt-get install libsecret-1-dev dbus-x11 gnome-keyring python3-venv

WSL2 / Linux: Requires libsecret-1-dev and gnome-keyring (handled automatically by init_env.sh on Debian/Ubuntu). Note: For WSL2, we recommend using Podman/Docker for development to ensure secure keyring storage. Direct installation on WSL2 may not provide secure token storage.

macOS / Windows: Native vault support, no additional system dependencies required.

Setup

We use an interactive setup script to manage the environment and system dependencies.

Initialize the project:

chmod +x scripts/init_env.sh
./scripts/init_env.sh

The script will:

• Install system dependencies on Linux (if needed)
• Create or update the virtual environment
• Install dependencies from lock files (reproducible builds) or fallback to requirements.txt
• Prompt to install development dependencies

Select Incremental Update for daily use or Clean Install to rebuild the environment.

Dependency Management

This project uses pip-tools for reproducible dependency management:

• requirements.in - Runtime dependencies (source)
• requirements.lock - Pinned runtime dependencies (generated)
• requirements-dev.in - Development dependencies (source)
• requirements-dev.lock - Pinned development dependencies (generated)

To generate/update lock files:

pip install pip-tools
pip-compile requirements.in -o requirements.lock
pip-compile requirements-dev.in -o requirements-dev.lock

Grant execution rights to the runner:

chmod +x scripts/start.sh

🛠️ Usage

The "One-Command" Entry

You no longer need to manually export paths or handle D-Bus sessions. Simply run:

./scripts/start.sh

Manual Execution (Development)

If you prefer manual control for debugging:

source .venv/bin/activate
export PYTHONPATH=$PYTHONPATH:$(pwd)/src
python3 src/main.py

Podman Development (Recommended for WSL2)

For development on WSL2 or for containerized development, we recommend using Podman.

Install Podman:

# Ubuntu/Debian
sudo apt-get install podman podman-compose

# macOS
brew install podman podman-compose

Build and run with Podman:

# Build the image
podman build -t gh-orchestrator -f Containerfile .

# Run with compose
podman-compose up

# Run interactively
podman run -it --rm \
  -v ./tmp:/app/tmp \
  -v ./config:/app/config:ro \
  -v gh-orchestrator-keyring:/root/.local/share/python_keyring \
  -e GH_ORCHESTRATOR_LOG_LEVEL=INFO \
  gh-orchestrator

Benefits of Podman:

• Secure keyring storage (GNOME Keyring inside container)
• Consistent development environment
• No WSL2 filesystem issues
• Rootless containers (better security)
• No Docker Desktop required

Volume Management:

• keyring_data: Stores encrypted tokens securely
• db_data: Persists database
• runner_data: Persists runner binaries and agents

⚙️ Configuration

Environment Variables

GH_ORCHESTRATOR_LOG_LEVEL Controls the verbosity of log output. Available levels:

• DEBUG - Detailed diagnostic information
• INFO - General informational messages (config.yaml default)
• WARNING - Warning messages
• ERROR - Error messages only (start.sh default)

Set via environment variable:

export GH_ORCHESTRATOR_LOG_LEVEL="ERROR"
./scripts/start.sh

Or set in config.yaml:

app:
  log_level: "ERROR"

Configuration File

The application uses config.yaml for configuration. See the file for all available options including:

• GitHub API settings
• Database configuration
• Runner base directory
• Cache settings
• Encryption settings

🧪 Testing and Development

GitHub Actions Workflow Testing

When testing self-hosted runners with GitHub Actions workflows, diagnostic logs are automatically created in the _diag directory. If a workflow is rerun, these logs can cause file conflicts.

Solution: Add a cleanup step to your workflow before running tests:

jobs:
  test:
    runs-on: self-hosted
    steps:
      - name: Clean Diagnostic Logs
        run: rm -rf /home/vin/github-orchestrator-agents/*/runner-*/_diag/
      
      # ... rest of your workflow steps

This clears old diagnostic logs before each test run, preventing file conflicts when jobs are rerun.

Install Test Dependencies

pip install -r requirements-dev.lock

Run Tests

# Run all tests with coverage
pytest

# Run tests with verbose output
pytest -v

# Run specific test file
pytest tests/test_helpers.py

# Run tests with HTML coverage report
pytest --cov=src --cov-report=html:tmp/htmlcov
# Open tmp/htmlcov/index.html in your browser

Static Analysis

# Type checking
mypy src/

# Code quality check
pylint src/

# Code formatting check
black --check src/

# Format code
black src/

# Security scanning
bandit -r src/

Pre-Commit Hooks

# Install pre-commit hooks
pre-commit install

# Run hooks on all files
pre-commit run --all-files

# Run hooks on staged files
pre-commit run

Run All Checks

pytest && mypy src/ && pylint src/ && black --check src/ && bandit -r src/

� Professional Audit Workflow

This project includes a comprehensive security, performance, and code quality audit workflow to ensure the CLI tool is production-ready.

Automated Audit Script

Run the complete audit suite:

chmod +x scripts/audit.sh
./scripts/audit.sh

The audit script checks:

• Security: Bandit (Python security), pip-audit (dependency vulnerabilities), Gitleaks (secret detection)
• Code Quality: Ruff (fast linter), Black (formatter), MyPy (type checker)
• Performance: Pyinstrument (profiling - optional)

Manual Security Checks

Bandit - Python security linter:

bandit -r src/gh_orchestrator -c pyproject.toml

pip-audit - Dependency vulnerability scanner:

pip-audit

Gitleaks - Secret detection (requires binary installation):

# Linux
wget https://github.com/gitleaks/gitleaks/releases/download/v8.18.0/gitleaks_8.18.0_linux_x64.tar.gz
tar -xzf gitleaks_8.18.0_linux_x64.tar.gz
sudo mv gitleaks /usr/local/bin/

# macOS (via Homebrew)
brew install gitleaks

# Windows (via Chocolatey)
choco install gitleaks

# Run scan
gitleaks detect --source . -v

Code Quality Tools

Ruff - Fast Python linter (replaces Flake8/Isort):

ruff check .           # Check for issues
ruff check . --fix     # Auto-fix issues

Black - Code formatter:

black --check src/ tests/   # Check formatting
black src/ tests/           # Format code

MyPy - Static type checker:

mypy src/gh_orchestrator

Performance Profiling

Pyinstrument - Statistical profiler:

# Profile the main entry point
pyinstrument -m gh_orchestrator.main

# Profile specific command
pyinstrument python src/main.py <command>

Pre-Commit Hooks

The project uses pre-commit hooks for automated quality checks:

pre-commit install
pre-commit run --all-files

Pre-commit hooks include: MyPy, Black, Ruff, Pylint, Bandit, Gitleaks, and basic file checks.

Tool Summary

Category	Tool	Purpose
Security	Bandit	Python security issues (subprocess, eval, etc.)
Security	pip-audit	Dependency CVEs
Security	Gitleaks	Secret detection (PATs, tokens)
Linting	Ruff	Fast linter (E, W, F, I, B, C4, S, UP rules)
Formatting	Black	Code formatting
Type Checking	MyPy	Static type safety
Profiling	Pyinstrument	Performance bottlenecks

�� Project Structure

• src/auth/: Manages secure token storage via system keyring.
• src/utils/: Handles GitHub API communication and scope validation.
• scripts/init_env.sh: Interactive environment and dependency manager.
• scripts/start.sh: Cross-platform execution wrapper (handles WSL headless sessions).

🔒 Security Note

This tool uses the system keyring (Keychain on macOS, Credential Manager on Windows, and Secret Service on Linux). Your GitHub Personal Access Token is never stored in plaintext within the project directory.