astronomer-registry-cleanup 0.1.1

A command-line tool for managing Docker registry cleanup operations in Astronomer environments

Astronomer Registry Cleanup Tool

A unified command-line tool for managing Docker registry cleanup operations, including listing repositories and deleting old image tags.

Features

• List Repositories: Browse the registry catalog and list all available repositories
• Delete Old Tags: Remove outdated image tags while preserving the most recent ones
• Flexible Authentication: Support for both local key files and Kubernetes secrets
• Safety Features: Dry-run mode, configurable tag retention, and comprehensive logging
• Kubernetes Integration: Direct integration with Kubernetes secrets for seamless operation

Installation

Prerequisites

• Python 3.10+
• uv for dependency management

Setup

1. Clone and install dependencies:

   git clone <repository-url>
   cd registry-cleanup-master
   uv sync
   

2. Install optional dependencies (for Kubernetes support):

   uv sync --extra k8s
   

Authentication Setup

Choose one of the following authentication methods:

Option A: Local Key File (Recommended for testing)

1. Extract TLS key from Kubernetes secret:

   mkdir -p keys
   kubectl get secret -n astronomer astronomer-tls -o jsonpath='{.data.tls\.key}' | base64 -d > keys/tls.key
   

   Note: On macOS, use base64 -D instead of base64 -d

2. Clean up after use:

   rm keys/tls.key
   

Option B: Direct Kubernetes Secret Access (Recommended for production)

No manual key extraction needed. The tool will automatically access the Kubernetes secret.

Usage

Global Options

uv run astronomer-registry-cleanup [OPTIONS] COMMAND [ARGS]

Global Options:
  -r, --registry REGISTRY     Registry host (e.g., registry.example.com) [REQUIRED]
  -d, --debug                 Enable debug logging
  -n, --namespace NAMESPACE   Kubernetes namespace (required with --k8s-secret)

Authentication (choose one):
  --k8s-secret SECRET_NAME    Get private key from Kubernetes secret
  --key-path KEY_PATH         Path to private key file (default: ./keys/tls.key)

Commands

1. List Repositories

List all repositories in the registry catalog:

# List all repositories
uv run astronomer-registry-cleanup -r registry.example.com list-repos

# List only client names (removes /airflow suffix)
uv run astronomer-registry-cleanup -r registry.example.com list-repos --clients-only

# Using Kubernetes secret
uv run astronomer-registry-cleanup -r registry.example.com --k8s-secret astronomer-tls -n astronomer list-repos

2. Delete Old Image Tags

Remove outdated image tags while preserving recent ones:

# Dry run (recommended first step)
uv run astronomer-registry-cleanup -r registry.example.com delete-tags modern-rocket-1234 -p deploy --dry-run

# Actually delete tags
uv run astronomer-registry-cleanup -r registry.example.com delete-tags modern-rocket-1234 -p deploy

# Using Kubernetes secret with debug logging
uv run astronomer-registry-cleanup -r registry.example.com --k8s-secret astronomer-tls -n astronomer -d delete-tags modern-rocket-1234 -p cli --dry-run

Delete Tags Options:

• client: Airflow deployment release name (e.g., modern-rocket-1234)
• -p, --prefix: Tag prefix pattern (e.g., deploy, cli)
• --dry-run: Show what would be deleted without actually deleting

Tag Retention Policy

• Default retention: 3 most recent tags are always preserved
• Latest tag protection: The newest tag with the specified prefix is never deleted
• Pattern matching: Only tags matching prefix-number pattern are considered for deletion

Examples

Complete Workflow Example

# 1. List all client deployments
uv run astronomer-registry-cleanup -r registry.example.com --k8s-secret astronomer-tls -n astronomer list-repos --clients-only

# 2. Check what would be deleted for a specific client
uv run astronomer-registry-cleanup -r registry.example.com --k8s-secret astronomer-tls -n astronomer delete-tags modern-rocket-1234 -p deploy --dry-run

# 3. Delete old tags if the dry run looks good
uv run astronomer-registry-cleanup -r registry.example.com --k8s-secret astronomer-tls -n astronomer delete-tags modern-rocket-1234 -p deploy

# 4. Run garbage collection on the registry to free disk space
kubectl exec -n astronomer -ti $(kubectl -n astronomer get pods -l component=registry -o jsonpath="{.items[*].metadata.name}") -c registry -- registry garbage-collect /etc/docker/registry/config.yml

Batch Operations

For multiple clients, you can combine commands:

# Get list of clients
CLIENTS=$(uv run astronomer-registry-cleanup -r registry.example.com --k8s-secret astronomer-tls -n astronomer list-repos --clients-only)

# Process each client
for client in $CLIENTS; do
  echo "Processing $client..."
  uv run astronomer-registry-cleanup -r registry.example.com --k8s-secret astronomer-tls -n astronomer delete-tags "$client" -p deploy --dry-run
done

Registry Garbage Collection

After deleting tags, run garbage collection to actually free disk space:

kubectl exec -n astronomer -ti $(kubectl -n astronomer get pods -l component=registry -o jsonpath="{.items[*].metadata.name}") -c registry -- registry garbage-collect /etc/docker/registry/config.yml

Note: This command may take a long time and will pause after "marking blob" messages before starting to delete files.

Troubleshooting

Common Issues

1. "No module named 'jwt'": Run uv sync to install dependencies
2. "kubernetes package required": Install with uv sync --extra k8s
3. "tls.key not found in secret": Verify the secret name and namespace
4. Occasional 404 errors: These are usually safe to ignore during tag deletion

Debug Mode

Enable debug logging for detailed operation information:

uv run astronomer-registry-cleanup -d -r registry.example.com ...

Development

For development and testing:

# Install development dependencies
uv sync --extra dev

# Run linting
uv run ruff check .
uv run ruff format .

Migration from Legacy Scripts

This tool replaces the previous separate scripts:

• delete-old-image-tags.py → main.py delete-tags
• list-catalog-repositories.py → main.py list-repos

The new unified interface provides better error handling, logging, and more flexible authentication options.