botmaro-secrets-manager 0.5.6

A standalone secret management tool for multi-environment deployments with Google Secret Manager

Botmaro Secrets Manager

A standalone, environment-aware secret management tool built on Google Secret Manager (GSM). Designed for multi-environment deployments with support for both GitHub Actions workflows and local development.

Features

• 🔐 Multi-environment support - Manage secrets across staging, prod, dev, and custom environments
• 🎯 Project scoping - Organize secrets by project within environments
• 🔄 Version control - Leverage GSM's built-in versioning
• 🚀 CI/CD ready - Bootstrap secrets in GitHub Actions or any CI/CD pipeline
• 📤 Multiple export formats - Export secrets in dotenv, JSON, YAML, GitHub Actions, and shell formats
• 📥 Bulk import - Import secrets from .env, JSON, or YAML files with automatic placeholder detection
• 🔗 GitHub Actions integration - Native composite action for seamless workflow integration
• 🛠️ CRUD operations - Full create, read, update, delete support via CLI
• ✅ Validation & checks - Validate secrets before deployment, detect placeholders
• 📦 Pip installable - Install as a standalone package
• 🎨 Rich CLI - Beautiful, user-friendly command-line interface
• 🔒 IAM integration - Automatic service account access management

Installation

From source (development)

# Clone the repository
git clone https://github.com/pawpeer/botmaro-gcp-secret-manager.git
cd botmaro-gcp-secret-manager

# Install in development mode
pip install -e .

From PyPI (when published)

pip install botmaro-secrets-manager

Quick Start

1. Create a configuration file

Create a secrets.yml file defining your environments and secrets:

version: "1.0"

globals:
  pawpeer:
    gcp_project: your-gcp-project-global
    prefix: pawpeer
    global_secrets:
      - name: API_KEY
        required: true

environments:
  staging:
    name: staging
    gcp_project: your-gcp-project-staging
    global_secrets:
      - name: DATABASE_URL
        required: true

See secrets.example.yml for a complete example.

2. Bootstrap your environment

# Load all secrets for staging environment
secrets-manager bootstrap staging

# Bootstrap with project scope
secrets-manager bootstrap staging --project myapp

# Export to a .env file
secrets-manager bootstrap staging --output .env.staging

3. Manage secrets

# Set a secret
secrets-manager set staging.API_KEY --value "sk-123456"

# Set a project-scoped secret
secrets-manager set staging.myapp.DATABASE_URL --value "postgres://..."

# Get a secret
secrets-manager get staging.API_KEY --reveal

# List all secrets
secrets-manager list staging

# Delete a secret
secrets-manager delete staging.OLD_KEY --force

Usage

Command-Line Interface

Bootstrap Command

Load all secrets for an environment:

secrets-manager bootstrap <environment> [OPTIONS]

Options:
  --project, -p TEXT      Project name to scope secrets
  --config, -c TEXT       Path to secrets config file
  --export/--no-export    Export secrets to environment variables [default: True]
  --runtime-sa TEXT       Runtime service account to grant access
  --deployer-sa TEXT      Deployer service account to grant access
  --output, -o TEXT       Output file for .env format
  --verbose, -v           Verbose output

Examples:

# Bootstrap staging
secrets-manager bootstrap staging

# Bootstrap with service account access grants
secrets-manager bootstrap staging \
  --runtime-sa [e-mail] \
  --deployer-sa [e-mail]

# Save to .env file
secrets-manager bootstrap prod --output .env.production

Set Command

Create or update a secret:

secrets-manager set <target> [OPTIONS]

Target format:
  - env.SECRET_NAME              (environment-scoped)
  - env.project.SECRET_NAME      (project-scoped)
  - globals.NAMESPACE.SECRET_NAME (schema-free global namespace)
  - globals.SECRET_NAME          (when config has one global namespace)

Options:
  --value, -v TEXT     Secret value (or read from stdin)
  --config, -c TEXT    Path to secrets config file
  --grant, -g TEXT     Service account to grant access (can be repeated)

Examples:

# Set an environment-scoped secret
secrets-manager set staging.API_KEY --value "sk-123456"

# Set a project-scoped secret
secrets-manager set staging.orchestrator.DATABASE_URL --value "postgres://..."

# Set a global secret
secrets-manager set globals.pawpeer.VONAGE_API_KEY --value "abc123"

# Read from stdin
echo "secret-value" | secrets-manager set staging.MY_SECRET

# Grant access to service accounts
secrets-manager set staging.API_KEY --value "sk-123" \
  --grant [e-mail] \
  --grant [e-mail]

Get Command

Retrieve a secret value:

secrets-manager get <target> [OPTIONS]

Options:
  --version TEXT       Secret version to retrieve [default: latest]
  --config, -c TEXT    Path to secrets config file
  --gcp-project TEXT   GCP project for schema-free global namespace lookups
  --reveal             Show the full secret value

Examples:

# Get latest version (masked by default)
secrets-manager get staging.API_KEY

# Reveal full value
secrets-manager get staging.API_KEY --reveal

# Get specific version
secrets-manager get staging.API_KEY --version 2 --reveal

# Get a namespaced global secret without loading secrets.yml
secrets-manager get globals.pawpeer.VONAGE_API_KEY \
  --gcp-project your-gcp-project-global \
  --reveal

List Command

List all secrets in an environment:

secrets-manager list <environment> [OPTIONS]

Options:
  --project, -p TEXT   Project name to filter by
  --config, -c TEXT    Path to secrets config file
  --gcp-project TEXT   GCP project for schema-free global namespace lookups
  --include global     Include global secrets in the listing
  --global             Alias for --include global
  --globals            Alias for --include globals
  --reveal             Show secret values

Examples:

# List all staging secrets
secrets-manager list staging

# List project-specific secrets
secrets-manager list staging --project orchestrator

# Show values (masked by default)
secrets-manager list staging --reveal

# Include global secrets as a separate section
secrets-manager list staging --include global

# Same as above, with aliases
secrets-manager list staging --global
secrets-manager list staging --globals

# List only a named global namespace
secrets-manager list globals.pawpeer --gcp-project your-gcp-project-global

Delete Command

Delete a secret:

secrets-manager delete <target> [OPTIONS]

Options:
  --config, -c TEXT    Path to secrets config file
  --force, -f          Skip confirmation

Examples:

# Delete with confirmation prompt
secrets-manager delete staging.OLD_API_KEY

# Force delete
secrets-manager delete staging.OLD_API_KEY --force

# Delete a namespaced global secret
secrets-manager delete globals.pawpeer.OLD_API_KEY --force

Export Command

Export secrets in various formats for CI/CD integration:

secrets-manager export <environment> [OPTIONS]

Options:
  --project, -p TEXT       Project name to scope secrets
  --config, -c TEXT        Path to secrets config file
  --format, -f TEXT        Export format: dotenv, github-env, github-output, json, yaml, shell [default: dotenv]
  --output, -o TEXT        Output file (default: stdout)
  --mask/--no-mask         Mask secrets in logs (for GitHub Actions formats) [default: True]
  --github-env             Write directly to $GITHUB_ENV (GitHub Actions only)
  --github-output          Write directly to $GITHUB_OUTPUT (GitHub Actions only)
  --verbose, -v            Verbose output

Supported Formats:

• dotenv / env - Standard .env file format (KEY=value)
• github-env - GitHub Actions environment file format with multiline support
• github-output - GitHub Actions job outputs format
• json - JSON object
• yaml / yml - YAML format
• shell / sh - Shell export script

Examples:

# Export as .env file
secrets-manager export staging --format dotenv --output .env.staging

# Export as JSON
secrets-manager export prod --format json --output secrets.json

# Export to stdout (for piping)
secrets-manager export staging --format yaml

# Export with project scope
secrets-manager export staging --project myapp --format dotenv --output .env.myapp

# Export for GitHub Actions (in workflow)
secrets-manager export production --github-env

# Export as shell script
secrets-manager export staging --format shell --output load-secrets.sh
chmod +x load-secrets.sh
source load-secrets.sh

Use Cases:

• Export secrets for local development (.env files)
• Generate configuration files in various formats
• Load secrets into GitHub Actions workflows
• Create shell scripts for environment setup
• Generate JSON/YAML for application configuration

Import Command

Import secrets from files into Google Secret Manager:

secrets-manager import <environment> [OPTIONS]

Options:
  --file, -f TEXT              Path to import file (.env, .json, .yml) [required]
  --config, -c TEXT            Path to secrets config file [default: ./secrets.yml]
  --project, -p TEXT           Project name to scope imported secrets
  --dry-run                    Preview what would be imported without making changes
  --skip-placeholders/--no-skip-placeholders   Skip secrets with placeholder values [default: True]
  --force                      Skip confirmation prompt
  --grant, -g TEXT             Service account to grant access (can be repeated)

Supported File Formats:

• .env - Standard environment file format (KEY=value)
• .json - JSON object with key-value pairs
• .yml / .yaml - YAML file with key-value mappings

Examples:

# Import from .env file (assumes ./secrets.yml exists)
secrets-manager import staging --file .env.staging

# Import from JSON file
secrets-manager import staging --file secrets.json

# Import from YAML file
secrets-manager import prod --file env.yml

# Dry run to preview changes
secrets-manager import staging --file .env.staging --dry-run

# Import with project scope
secrets-manager import staging --file myapp.env --project myapp

# Import and grant access to service accounts
secrets-manager import staging --file .env \
  --grant [e-mail]

# Use custom config file
secrets-manager import staging --file .env --config /path/to/secrets.yml

# Import including placeholders (not recommended)
secrets-manager import staging --file .env --no-skip-placeholders

Features:

• Automatically detects and skips placeholder values (PLACEHOLDER, TODO, CHANGEME, etc.)
• Supports quoted values in .env files
• Preview secrets before importing with masked values
• Detailed progress reporting with success/failure counts
• Validates file format and provides helpful error messages

Use Cases:

• Initial setup: Import existing secrets from .env files
• Migration: Move secrets from another system
• Bulk updates: Update multiple secrets at once
• Project onboarding: Quickly set up secrets for new team members

Grant Access Command

Grant access to all secrets in an environment or project:

secrets-manager grant-access <target> [OPTIONS]

Target format:
  - env                  (all environment-level secrets)
  - env.project          (all project-scoped secrets)

Options:
  --sa TEXT            Service account email to grant access (can be repeated)
  --config, -c TEXT    Path to secrets config file
  --force, -f          Skip confirmation prompt

Examples:

# Grant access to all staging secrets
secrets-manager grant-access staging \
  --sa [e-mail]

# Grant to multiple service accounts
secrets-manager grant-access staging \
  --sa [e-mail] \
  --sa [e-mail]

# Grant access to all project-specific secrets
secrets-manager grant-access staging.myapp \
  --sa [e-mail]

# Skip confirmation
secrets-manager grant-access staging --sa [e-mail] --force

Note: This command grants the secretmanager.secretAccessor role, allowing the service account to read secret values.

Check Command

Validate secrets configuration and state:

secrets-manager check <environment> [OPTIONS]

Options:
  --project, -p TEXT      Project name to scope secrets
  --config, -c TEXT       Path to secrets config file
  --workflows, -w TEXT    Path to workflow file or .github/workflows directory
  --verbose, -v           Show detailed findings

This command performs comprehensive validation:

• ✅ All required secrets exist in GSM
• ✅ No placeholder values in secrets (e.g., "PLACEHOLDER_", "TODO", "changeme")
• ✅ No placeholder service accounts
• ✅ Service accounts have proper access to secrets
• ✅ Workflow secrets are defined in config (when --workflows provided)

Examples:

# Check all staging secrets
secrets-manager check staging

# Check with project scope
secrets-manager check staging --project myapp

# Validate against workflow files
secrets-manager check staging --workflows .github/workflows

# Check specific workflow file
secrets-manager check prod --workflows .github/workflows/deploy.yml --verbose

# Use in CI/CD to prevent deployment with placeholders
secrets-manager check prod && echo "Secrets validated, deploying..."

Exit codes:

• 0 - All checks passed
• 1 - Validation failed (missing secrets, placeholders, or access issues)

Example output:

Validation Summary:
⚠️  2 placeholder secrets
❌ 1 missing secrets
❌ 1 service account access issues

⚠️  Placeholder Secrets (2):
  • OPENAI_API_KEY: PLACEHOLDER_OPENAI_...
  • DATABASE_URL: TODO-replace-me

❌ Missing Secrets (1):
  • STRIPE_SECRET_KEY

❌ Missing Service Account Access (1):
  • API_KEY → [e-mail]

❌ Validation failed with errors

Use cases:

• Pre-deployment validation in CI/CD pipelines
• Verify secrets before running bootstrap
• Audit service account access
• Ensure workflow secrets are properly configured
• Catch configuration issues early

GitHub Actions Integration

Botmaro Secrets Manager provides native GitHub Actions integration through a composite action that automatically loads secrets from GCP Secret Manager into your workflows.

Using the Composite Action (Recommended)

The easiest way to use secrets in GitHub Actions is with our composite action:

name: Deploy to Production

on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write  # Required for Workload Identity Federation

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production

    steps:
      - uses: actions/checkout@v4

      - name: Load Secrets from GCP
        uses: ./.github/actions/setup-secrets
        with:
          environment: production
          gcp-project-id: my-gcp-project
          workload-identity-provider: projects/123456789/locations/global/workloadIdentityPools/github/providers/github-provider
          service-account: [e-mail]

      - name: Deploy Application
        run: |
          # All secrets are now available as environment variables
          ./deploy.sh

Key Features:

• ✅ Automatic authentication with Workload Identity Federation
• ✅ Built-in secrets validation
• ✅ Automatic secret masking in logs
• ✅ Zero secret duplication

📖 Full Documentation: See GITHUB_ACTIONS.md for complete setup instructions, including Workload Identity Federation configuration.

Manual CLI Integration

You can also use the CLI directly in your workflows:

name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install secrets-manager
        run: pip install botmaro-gcp-secret-manager

      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: projects/.../providers/github-provider
          service_account: [e-mail]

      - name: Export secrets to GitHub environment
        run: |
          secrets-manager export production --github-env

      - name: Deploy application
        run: |
          # Secrets are now available
          echo "Deploying..."
          ./deploy.sh

Setting up initial secrets

Before running in GitHub Actions, you need to populate secrets using this tool's CLI.

Note: This tool is your interface to Google Secret Manager. You only need gcloud for authentication, not for managing secrets.

# 1. Authenticate to GCP (required for API access)
gcloud auth application-default login

# 2. Use secrets-manager CLI to create and manage secrets
secrets-manager set production.API_KEY --value "sk-123456"
secrets-manager set production.DATABASE_URL --value "postgres://..."

# 3. Verify secrets were created
secrets-manager list production --reveal

The secrets-manager CLI automatically creates secrets in Google Secret Manager with proper naming conventions and IAM permissions.

Configuration Reference

Environment Configuration

environments:
  <env-name>:
    name: string              # Environment name
    gcp_project: string       # GCP project ID
    prefix: string            # Optional: secret name prefix (default: botmaro-{env})

    # Optional: Service accounts that need access to all environment-level secrets
    # These will be automatically granted secretAccessor role during bootstrap
    service_accounts:
      - [e-mail]
      - [e-mail]

    global_secrets:           # Environment-level secrets
      - name: string          # Secret name
        description: string   # Optional description
        required: boolean     # Whether secret is required (default: true)
        default: string       # Optional default value

    projects:                 # Project-specific secrets
      <project-name>:
        project_id: string

        # Optional: Project-specific service accounts
        # If not specified, environment-level service accounts will be used
        service_accounts:
          - [e-mail]

        secrets:
          - name: string
            description: string
            required: boolean
            default: string

Automatic Service Account Access Grants

The tool automatically grants secretAccessor role to service accounts configured in secrets.yml during bootstrap. This ensures that your runtime and deployment service accounts always have access to the secrets they need.

Configuration Example:

environments:
  staging:
    name: staging
    gcp_project: my-project-staging

    # These service accounts will automatically get access to all staging secrets
    service_accounts:
      - [e-mail]
      - [e-mail]

    global_secrets:
      - name: API_KEY
        required: true

    projects:
      myapp:
        project_id: myapp

        # Additional service accounts for this project's secrets
        service_accounts:
          - [e-mail]

        secrets:
          - name: DATABASE_URL
            required: true

How it works:

1. During secrets-manager bootstrap, the tool checks if configured service accounts have access to each secret
2. If access is missing, it automatically grants the secretAccessor role
3. If access already exists, it skips (idempotent operation)
4. For project-scoped secrets, both environment-level and project-level service accounts get access

Manual override:

You can also specify additional service accounts at runtime:

secrets-manager bootstrap staging \
  --runtime-sa [e-mail] \
  --deployer-sa [e-mail]

These will be granted access in addition to the service accounts configured in secrets.yml.

Secret Naming Convention

Secrets are stored in GSM using a double-hyphen (--) convention for hierarchical separation, while single hyphens (-) or underscores (_) are used within component names:

• Environment-scoped: {prefix}--{SECRET_NAME}

  • Example: botmaro-staging--API_KEY
  • Example: my-longer-prefix--SUPABASE_URL

• Project-scoped: {prefix}--{project}--{SECRET_NAME}

  • Example: botmaro-staging--orchestrator--DATABASE_URL
  • Example: my-company-prod--my-service--API_KEY

Where {prefix} defaults to botmaro-{environment} but can be customized.

Why double-hyphen?

• Provides clear, unambiguous hierarchical separation
• Easy to parse: secret_id.split('--')
• GSM-compliant (only allows letters, numbers, -, and _)
• Visually distinct from naming hyphens

Local Development

Setup

# Install dependencies
pip install -e ".[dev]"

# Set up secrets config
cp secrets.example.yml secrets.yml
# Edit secrets.yml with your configuration

# Authenticate to GCP
gcloud auth application-default login

Running locally

# Bootstrap local environment
secrets-manager bootstrap dev

# Or use a custom config
secrets-manager bootstrap dev --config ./my-secrets.yml

Python API

You can also use the library programmatically:

from secrets_manager import SecretsManager, SecretsConfig

# Load from config file
config = SecretsConfig.from_file("secrets.yml")
manager = SecretsManager(config)

# Bootstrap environment
secrets = manager.bootstrap(env="staging", export_to_env=True)

# Set a secret
manager.set_secret(
    env="staging",
    secret="API_KEY",
    value="sk-123456",
    grant_to=["[e-mail]"]
)

# Get a secret
value = manager.get_secret(env="staging", secret="API_KEY")

# List secrets
secrets = manager.list_secrets(env="staging", project="myapp")

# Grant access to all secrets in an environment
result = manager.grant_access_bulk(
    env="staging",
    service_accounts=["[e-mail]", "[e-mail]"]
)

IAM Permissions

The service account running the secrets manager needs:

• secretmanager.admin - To create/update/delete secrets
• iam.serviceAccountUser - To grant access to other service accounts

Grant these roles:

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:[e-mail]" \
  --role="roles/secretmanager.admin"

Runtime service accounts need:

• secretmanager.secretAccessor - To read secrets (granted automatically by the tool)

Troubleshooting

Authentication errors

Make sure you're authenticated to GCP:

gcloud auth application-default login

For CI/CD, use service account key or Workload Identity.

Secret not found

Check that:

1. The secret exists in GSM: gcloud secrets list --project PROJECT_ID
2. Your config file matches the environment name
3. The GCP project ID is correct

Permission denied

Ensure your service account has the required IAM roles listed above.

Releasing

This project uses tag-based releases. When you're ready to publish a new version:

1. Update version and changelog

# Update version in pyproject.toml
vim pyproject.toml  # Change version = "0.1.0" to "0.1.1"

# Update CHANGELOG.md
vim CHANGELOG.md    # Add release notes under [Unreleased]

2. Commit changes

git add pyproject.toml CHANGELOG.md
git commit -m "Bump version to 0.1.1"
git push origin main

3. Create and push tag

# Create an annotated tag
git tag -a v0.1.1 -m "Release v0.1.1: Description of changes"

# Push the tag (this triggers PyPI publish!)
git push origin v0.1.1

What happens automatically:

1. Version verification - Ensures tag matches pyproject.toml version
2. Build package - Creates wheel and source distribution
3. Publish to PyPI - Uploads to PyPI (requires PYPI_API_TOKEN secret)
4. Create GitHub Release - Creates release page with changelog and assets

Manual publish (if needed)

You can also manually trigger the publish workflow from GitHub Actions UI:

• Go to Actions → Publish to PyPI → Run workflow

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details.

Support

For issues and questions:

• GitHub Issues: https://github.com/pawpeer/botmaro-gcp-secret-manager/issues
• Documentation: https://github.com/pawpeer/botmaro-gcp-secret-manager