pydeb-s3

pydeb-s3 is a Python port of deb-s3, a simple utility to make creating and managing APT repositories on S3.

Most existing guides on using S3 to host an APT repository have you using something like reprepro to generate the repository file structure, and then s3cmd to sync the files to S3.

The annoying thing about this process is it requires you to maintain a local copy of the file tree for regenerating and syncing the next time.

With pydeb-s3, there is no need for this. pydeb-s3 features:

• Downloads the existing package manifest and parses it.
• Updates it with the new package, replacing the existing entry if already there or adding a new one if not.
• Uploads the package itself, the Packages manifest, and the Packages.gz manifest. It will skip the uploading if the package is already there.
• Updates the Release file with the new hashes and file sizes.

Updated Features

pydeb-s3 has been rewritten in Python with modern tooling and additional capabilities:

• Parses .deb files using the official python-debian library
• Updates package manifests, replacing existing entries or adding new ones
• Uploads packages, Packages manifest, and compressed manifests (.gz, .bz2, .xz)
• Updates Release file with new hashes and file sizes
• GPG signing of Release files for secure APT repositories
• S3-compatible storage support (AWS S3, Google Cloud Storage, MinIO, etc.)
• Concurrent operation locking to prevent conflicting uploads
• Cross-component deduplication with --dedupe-component: avoids re-uploading packages that already exist in another component (e.g., upload to main by copying from non-free)
• Acquire-By-Hash support: publishes content-addressed by-hash/ copies of each index (SHA256, SHA512, MD5Sum), so apt can fetch indexes atomically and never sees a Release pointing at a missing or mid-update Packages file
• backfill-by-hash command: repairs repositories published before by-hash support existed by creating the by-hash copies named in the existing Release, without modifying or re-signing it
• Dry-run mode for clean/verify operations
• Configurable timestamps with --timestamps/--no-timestamps flag, auto-detects TTY for clean interactive output
• Modern CLI with Typer, featuring help text and shell completion

Installation

Install via pip:

$ pip install pydeb-s3

For isolated installation, use pipx:

$ pipx install pydeb-s3

Quick Start

Upload a package to S3:

$ pydeb-s3 upload --bucket my-bucket my-deb-package-1.0.0_amd64.deb

For S3-compatible endpoints (e.g., Google Cloud Storage, MinIO):

$ pydeb-s3 upload --bucket my-bucket \
    --endpoint https://storage.googleapis.com \
    --checksum-when-required \
    --visibility nil \
    my-deb-package-1.0.0_amd64.deb

Usage

pydeb-s3 provides the following commands:

$ pydeb-s3 --help
Usage: pydeb-s3 [OPTIONS] COMMAND [ARGS]...

  Easily create and manage an APT repository on S3

Options:
  --quiet                         Only show errors
  --debug                         Enable debug output
  --timestamps / --no-timestamps  Enable/disable timestamps (auto-detects TTY
                                  by default)
  --install-completion            Install completion for the current shell.
  --show-completion               Show completion for the current shell, to
                                  copy it or customize the installation.
  --help                          Show this message and exit.

Commands:
  upload   Upload the given files to a S3 bucket as an APT repository.
  list     List packages in given codename, component, and optionally architecture.
  show     Show information about a package.
  exists   Check if a package exists in the repository.
  copy     Copy a package to another codename and component.
  delete   Remove a package from the repository.
  verify   Verify that the files in the package manifests exist.
  clean    Remove orphaned package files.
  backfill-by-hash  Create missing by-hash index copies for an existing repository.

For detailed options per command, run pydeb-s3 <command> --help.

Output Control

pydeb-s3 provides control over log output:

• Timestamps: Use --timestamps to enable or --no-timestamps to disable timestamp prefixes
• Auto-detection: By default, timestamps are automatically disabled when output is a terminal (TTY) for cleaner interactive use, and enabled when output is piped or redirected for logging purposes
• Log format: When timestamps are enabled, loguru's default format is used (includes timestamp with milliseconds, level, and source location)

Common Command Examples

List packages

$ pydeb-s3 list --bucket my-bucket --codename stable

Show package info

$ pydeb-s3 show mypackage --bucket my-bucket --version 1.0.0

Check if package exists

$ pydeb-s3 exists mypackage --bucket my-bucket --version 1.0.0

Copy package to another codename

$ pydeb-s3 copy mypackage --bucket my-bucket --to-codename jammy --to-component main

Upload with cross-component deduplication

$ pydeb-s3 upload --bucket my-bucket --component main \
    --dedupe-component non-free \
    ollama_0.15.0_amd64.deb libollama-common_0.15.0_amd64.deb

Verify repository integrity

$ pydeb-s3 verify --bucket my-bucket --fix-manifests

Clean orphaned packages (dry-run first!)

$ pydeb-s3 clean --bucket my-bucket --dry-run
$ pydeb-s3 clean --bucket my-bucket  # Actually remove orphans

Backfill by-hash index copies (migration)

For repositories published before Acquire-By-Hash support, create the missing by-hash copies from the existing Release (no re-signing):

$ pydeb-s3 backfill-by-hash --bucket my-bucket --codename stable --dry-run
$ pydeb-s3 backfill-by-hash --bucket my-bucket --codename stable
$ pydeb-s3 backfill-by-hash --bucket my-bucket --all-codenames

Configuration

AWS Credentials

pydeb-s3 uses standard boto3 credential resolution:

1. Command-line options: --access-key-id, --secret-access-key, --session-token
2. Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION
3. AWS config file: ~/.aws/credentials and ~/.aws/config

S3 Bucket

The --bucket option is required for all commands. Use --prefix to add a path prefix to all S3 objects.

Visibility / ACL

Control uploaded file permissions with --visibility:

• public (default): public-read ACL
• private: private ACL
• authenticated: authenticated-read ACL
• nil: No ACL (for S3-compatible storage that doesn't support ACLs)

GPG Signing

Sign Release files with --sign <KEY_ID>. You can specify multiple keys if needed (though repeatable --sign is limited by Typer version constraints).

Development

pydeb-s3 uses hatch for packaging and dependency management.

License

MIT License - see LICENSE file for details.

Credits

• Original deb-s3 by Ken Robertson