s3duct 0.4.0

Chunked, resumable, encrypted pipe to S3-compatible object storage

s3duct

Chunked, resumable, encrypted pipe to S3-compatible object storage.

Stream data from stdin directly to S3 in fixed-size chunks with integrity verification, optional encryption, cache management, and automatic resume on failure.

Features

• Chunked streaming - pipes stdin to S3 without loading the full file into memory
• Resumable uploads - interrupted uploads pick up where they left off using a signature-chained resume log
• Dual-hash integrity - every chunk is verified with both SHA-256 and SHA3-256
• HMAC signature chain - cryptographic chain across all chunks prevents tampering or reordering
• Encryption - optional client-side encryption: AES-256-GCM (symmetric) or age (asymmetric, with optional post-quantum keys)
• Parallel uploads - adaptive multi-threaded upload pipeline with automatic concurrency scaling
• Backpressure - disk-aware flow control prevents scratch directory from filling up
• Deep verification - verify --deep re-downloads and hash-verifies every chunk; upload checksums (SHA-256) are verified server-side by S3 at upload time
• Maintenance - gc collects orphaned chunks, prune rotates old backup streams
• S3-compatible - works with AWS S3, Cloudflare R2, MinIO, Backblaze B2, Wasabi, and any S3-compatible endpoint via --endpoint-url

Installation

pip install s3duct

Shell Completion

Enable tab completion for commands and options:

# Bash (~/.bashrc)
s3duct completion bash >> ~/.bashrc

# Zsh (~/.zshrc)
s3duct completion zsh >> ~/.zshrc

# Fish (~/.config/fish/completions/s3duct.fish)
s3duct completion fish > ~/.config/fish/completions/s3duct.fish

Then restart your shell or source the file.

Development

git clone https://github.com/SiteRelEnby/s3duct
cd s3duct
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# run tests
pytest

AWS Credentials

s3duct uses boto3's standard credential chain. It does not handle authentication itself. Credentials are resolved in this order:

1. Environment variables — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN
2. Shared credentials file — ~/.aws/credentials (set up via aws configure)
3. Config file — ~/.aws/config (profiles, SSO)
4. IAM instance role — automatic on EC2, ECS, Lambda
5. SSO / credential process — configured in ~/.aws/config

For non-AWS S3-compatible services, set the appropriate credentials for that provider and use --endpoint-url.

Quick Start

Upload

# No encryption
tar czf - /my/data | s3duct put --bucket mybucket --name mybackup

# AES-256-GCM encryption (recommended for most users)
tar czf - /my/data | s3duct put --bucket mybucket --name mybackup \
  --key hex:$(openssl rand -hex 32)

# age encryption (asymmetric — see Encryption section below)
tar czf - /my/data | s3duct put --bucket mybucket --name mybackup \
  --age-identity ~/.age/key.txt

# Custom chunk size, storage class, and tags
pg_dump mydb | s3duct put --bucket mybucket --name db-backup \
  --chunk-size 256M --storage-class STANDARD_IA --tag env=prod

Download

# Unencrypted
s3duct get --bucket mybucket --name mybackup > restored.tar.gz

# AES-256-GCM
s3duct get --bucket mybucket --name mybackup --key hex:... | zpool import

# age
s3duct get --bucket mybucket --name mybackup \
  --age-identity ~/.age/key.txt | tar xzf -

List stored streams

s3duct list --bucket mybucket

Verify integrity

# Fast: HEAD each chunk and compare stored ETags
s3duct verify --bucket mybucket --name mybackup

# Deep: download every chunk and verify content hashes + integrity chain
s3duct verify --bucket mybucket --name mybackup --deep --key hex:...

Encryption

s3duct supports two encryption methods. Both encrypt each chunk client-side before upload. The manifest records which method was used, so get auto-detects the decryption path.

The manifest is encrypted by default when encryption is active (both AES-256-GCM and age are supported). Use --no-encrypt-manifest to keep the manifest as readable JSON if needed.

If the manifest is unencrypted, it exposes metadata only (not plaintext): stream name/prefix, chunk count, per-chunk sizes, hashes, ETags, timestamps, storage class, tags, and the encryption method/recipient. With read access to the bucket, many of these (object keys, sizes, storage class, timestamps, and ETags) are already inferable from object listings and HEADs, but the manifest makes it trivial to enumerate and correlate.

AES-256-GCM (symmetric)

Simple, fast, quantum-resistant. You manage one 32-byte key.

# Generate a key (save this — you need it to decrypt)
openssl rand -hex 32
# e.g., a1b2c3d4...64 hex chars

# Upload
tar czf - /data | s3duct put --bucket b --name n --key hex:a1b2c3...

# Download
s3duct get --bucket b --name n --key hex:a1b2c3... > data.tar.gz

Key formats:

• hex:AABBCC... — 64 hex characters (32 bytes)
• file:/path/to/keyfile — raw 32-byte file
• env:VAR_NAME — environment variable containing hex key

Prefer file: or env: in real deployments — a hex: key on the command line is visible to other local users via ps and may end up in your shell history.

The manifest is encrypted by default with AES. To keep it as readable JSON:

s3duct put --bucket b --name n --key hex:... --no-encrypt-manifest

Each chunk is encrypted with a random 96-bit nonce. A nonce collision under the same key (including cross-session key reuse) becomes probable around 2^48 chunks (the birthday bound), which at the default 512MB chunk size is ~128 exabytes. Use a different key per stream to reset the bound. If you anticipate your use case exceeding the birthday bound within a single stream and reasonable block size, age does not suffer from birthday bound issues. If symmetric is a hard requirement, please open an issue and make a donation to Trans Lifeline. Plus we'd just be interested to hear about how well it hyperscales.

age (asymmetric)

Uses age for public-key encryption. Useful when the encryptor shouldn't hold the decryption key (e.g., automated backups encrypting to an offline recovery key).

age uses X25519 (classic) or ML-KEM-768+X25519 (PQ hybrid) to wrap a per-file key, with HKDF-SHA256 and ChaCha20-Poly1305. Payload encryption also uses HKDF-SHA256 and ChaCha20-Poly1305, with an HMAC-SHA256 header MAC.

# Install age
# macOS: brew install age
# Linux: xbps-install age / apt install age / apk add age / pacman -S age / etc. - see https://age-encryption.org/

# Generate a keypair
age-keygen -pq -o ~/.age/key.txt
# Prints the public key (recipient): age1...

# Upload (uses the keypair — extracts public key automatically)
tar czf - /data | s3duct put --bucket b --name n --age-identity ~/.age/key.txt

# Download (needs the private key)
s3duct get --bucket b --name n --age-identity ~/.age/key.txt > data.tar.gz

Post-quantum keys

age supports hybrid post-quantum keys (X25519 + ML-KEM-768) via the -pq flag. These protect against both classical and future quantum attacks. This is recommended if long-term confidentiality matters (e.g., encrypted archives that may sit in storage for years), and is likely to become the default in a future release (note that -pq was already specified in the previous section - the only downside to using post-quantum keys is an overhead of ~2KB/block).

Comparison

	AES-256-GCM	age
Quantum resistant	Yes	Yes, with -pq
Key model	Symmetric	Asymmetric
External dependency	None	age CLI
Best for	Simple backups	Multi-party / offline keys

Storage Classes

# Standard (default)
tar czf - /data | s3duct put --bucket mybucket --name backup --no-encrypt

# Infrequent Access
tar czf - /data | s3duct put --bucket mybucket --name backup --no-encrypt \
  --storage-class STANDARD_IA

# Glacier (retrieval requires thaw — see below)
tar czf - /data | s3duct put --bucket mybucket --name archive --no-encrypt \
  --storage-class GLACIER

# Glacier Deep Archive (cheapest, slowest retrieval)
tar czf - /data | s3duct put --bucket mybucket --name deep-archive --no-encrypt \
  --storage-class DEEP_ARCHIVE

Supported values: STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, GLACIER_IR.

Note: the manifest is always uploaded as STANDARD so it remains immediately accessible regardless of the chunk storage class.

Glacier/Deep Archive retrieval: chunks stored in Glacier or Deep Archive cannot be downloaded directly. Use s3duct restore to initiate the thaw:

# Start restore (Standard tier, 7 days)
s3duct restore --bucket mybucket --name archive

# Wait for completion (polls every 60s)
s3duct restore --bucket mybucket --name archive --wait

# Faster restore (higher cost)
s3duct restore --bucket mybucket --name archive --tier Expedited

# Then download once restored
s3duct get --bucket mybucket --name archive > data.tar.gz

S3-compatible endpoints

# Cloudflare R2
s3duct put --bucket mybucket --name backup --no-encrypt \
  --endpoint-url https://<account-id>.r2.cloudflarestorage.com

# MinIO
s3duct put --bucket mybucket --name backup --no-encrypt \
  --endpoint-url http://localhost:9000

Options

s3duct put

Option	Default	Description
--bucket	(required)	S3 bucket name
--name	(required)	Stream name (used as S3 key prefix)
--chunk-size	512M	Chunk size (e.g., 256M, 1G)
--key		AES-256-GCM key (hex:..., file:..., or env:...)
--age-identity		Path to age identity file (mutually exclusive with --key)
--no-encrypt		Disable encryption even if key/identity provided
--encrypt-manifest/--no-encrypt-manifest	on when encrypted	Encrypt manifest with same key/identity as chunks
--clobber		Overwrite existing stream (fails by default if stream exists)
--scratch-dir	~/.s3duct/scratch	Directory for temporary chunk files
--tag		Custom metadata tag (key=value, repeatable)
--description		Human-readable description stored in manifest
--storage-class	STANDARD	S3 storage class
--region		AWS region
--prefix		S3 key prefix
--endpoint-url		Custom S3 endpoint URL
--expected-size		Expected stream size (warns if actual is shorter)
--diskspace-limit	auto	Max scratch disk usage (e.g., 2G)
--buffer-chunks	auto	Max buffered chunks before backpressure
--strict-resume/--no-strict-resume	on	Fail if stdin ends before all resume-log chunks are re-verified
--retries	10	Max retry attempts per S3 operation
--upload-workers	auto	Parallel upload threads (auto adapts based on throughput, or an integer for fixed concurrency)
--min-upload-workers	2	Minimum workers for auto mode
--max-upload-workers	16	Maximum workers for auto mode
--bandwidth-limit		Max upload bandwidth per worker (e.g., 50M, 1G)
--progress	auto	Progress display: auto, rich, plain, or none
--verbose / -v		Show detailed progress events and timing
--quiet / -q		Suppress all output except errors
--summary	text	Summary output format: text, json, or none

s3duct get

Option	Default	Description
--bucket	(required)	S3 bucket name
--name	(required)	Stream name to restore
--key		AES-256-GCM key (hex:..., file:..., or env:...)
--age-identity		Path to age identity file (mutually exclusive with --key)
--no-decrypt		Skip decryption (download raw encrypted chunks)
--region		AWS region
--prefix		S3 key prefix
--endpoint-url		Custom S3 endpoint URL
--scratch-dir	~/.s3duct/scratch	Directory for temporary chunk files
--retries	10	Max retry attempts per S3 operation
--download-workers	auto	Parallel download threads (auto or integer)
--min-download-workers	2	Minimum workers for auto mode
--max-download-workers	16	Maximum workers for auto mode
--bandwidth-limit		Max download bandwidth per worker (e.g., 50M, 1G)
--progress	auto	Progress display: auto, rich, plain, or none
--verbose / -v		Show detailed progress events and timing
--quiet / -q		Suppress all output except errors
--summary	text	Summary output format: text, json, or none

s3duct list

Option	Default	Description
--bucket	(required)	S3 bucket name
--prefix		Filter by prefix
--region		AWS region
--endpoint-url		Custom S3 endpoint URL

s3duct verify

Option	Default	Description
--bucket	(required)	S3 bucket name
--name	(required)	Stream name to verify
--deep		Download every chunk and verify content hashes (default: ETag-only HEAD check). Without a key, encrypted streams are verified against the recorded ciphertext SHA-256; with --key/--age-identity, plaintext hashes and the full integrity chain are checked
--key		AES-256-GCM key (required if manifest is encrypted with AES)
--age-identity		Path to age identity file (required if manifest is encrypted with age)
--region		AWS region
--prefix		S3 key prefix
--endpoint-url		Custom S3 endpoint URL
--scratch-dir	~/.s3duct/scratch	Directory for temporary chunk files during --deep
--retries	10	Max retry attempts per S3 operation
--progress	auto	Progress display: auto, rich, plain, or none
--verbose / -v		Show detailed progress events and timing
--quiet / -q		Suppress progress and summary output
--summary	text	Summary output format: text, json, or none

s3duct delete

Option	Default	Description
--bucket	(required)	S3 bucket name
--name	(required)	Stream name to delete
--key		AES-256-GCM key (required if manifest is encrypted with AES)
--age-identity		Path to age identity file (required if manifest is encrypted with age)
--dry-run		Show what would be deleted without deleting
--force		Skip confirmation prompt
--region		AWS region
--prefix		S3 key prefix
--endpoint-url		Custom S3 endpoint URL
--retries	10	Max retry attempts per S3 operation
--progress	auto	Progress display: auto, rich, plain, or none
--verbose / -v		Show detailed progress events and timing
--quiet / -q		Suppress progress output

s3duct gc

Delete orphaned chunks not referenced by any manifest — leftovers from interrupted uploads that were never resumed, or --clobber re-uploads. Objects newer than --older-than days are never touched, so a running upload is safe. Streams with unreadable (encrypted) manifests are skipped unless a key is given, and keys s3duct didn't create are never deleted.

Option	Default	Description
--bucket	(required)	S3 bucket name
--older-than	7	Only delete orphans older than this many days
--dry-run		Show what would be deleted without deleting
--force		Skip confirmation prompt
--key		AES-256-GCM key, to read encrypted manifests
--age-identity		Path to age identity file, to read encrypted manifests
--region		AWS region
--prefix		S3 key prefix
--endpoint-url		Custom S3 endpoint URL
--retries	10	Max retry attempts per S3 operation

s3duct prune

Backup rotation: keep the newest N streams (by manifest creation time) matching --stream-prefix, delete the rest.

# Keep the 7 newest daily backups
s3duct prune --bucket b --stream-prefix daily/ --keep 7

Option	Default	Description
--bucket	(required)	S3 bucket name
--keep	(required)	Number of newest streams to keep
--stream-prefix		Only consider streams whose name starts with this prefix
--dry-run		Show what would be deleted without deleting
--force		Skip confirmation prompt
--key		AES-256-GCM key, to read encrypted manifests
--age-identity		Path to age identity file, to read encrypted manifests
--region		AWS region
--prefix		S3 key prefix
--endpoint-url		Custom S3 endpoint URL
--retries	10	Max retry attempts per S3 operation

s3duct restore

Initiate Glacier/Deep Archive restore for archived streams.

Option	Default	Description
--bucket	(required)	S3 bucket name
--name	(required)	Stream name to restore
--key		AES-256-GCM key (required if manifest is encrypted with AES)
--age-identity		Path to age identity file (required if manifest is encrypted with age)
--tier	Standard	Restore tier: Expedited, Standard, or Bulk
--days	7	Days to keep restored copies available
--wait		Wait for restore to complete (polls every 60s)
--poll-interval	60	Seconds between status checks when using --wait
--region		AWS region
--prefix		S3 key prefix
--endpoint-url		Custom S3 endpoint URL
--retries	10	Max retry attempts per S3 operation
--progress	auto	Progress display: auto, rich, plain, or none
--verbose / -v		Show detailed progress events and timing
--quiet / -q		Suppress progress output

How It Works

Upload Pipeline

Main thread                         Thread pool (auto-scaled)
──────────                          ────────────────────────
stdin → chunk (512MB default)
     → SHA-256 + SHA3-256 dual hash
     → HMAC-SHA256 signature chain
     → optional encryption ──────→  S3 upload with retry (parallel)
     → when window full: drain  ←─  result + ETag
       oldest upload
     → append to resume log
     → repeat until EOF
     → drain remaining uploads
     → upload manifest (encrypted by default)

Uploads run in parallel using a sliding window. The main thread reads, hashes, and encrypts sequentially (these depend on ordering), then submits uploads to a thread pool. Completed uploads are drained oldest-first to maintain strict resume log ordering. Worker count auto-scales based on upload-vs-read throughput ratio (override with --upload-workers N).

Integrity Chain

Each chunk's signature is computed as:

chain[0] = HMAC-SHA256(genesis_key, sha256(chunk) || sha3_256(chunk))
chain[n] = HMAC-SHA256(chain[n-1], sha256(chunk) || sha3_256(chunk))

This creates a cryptographic chain where any modification, deletion, or reordering of chunks is detectable.

Threat model: the genesis key is a public constant, so the chain is tamper-evident, not tamper-proof — it protects against accidental corruption, truncation, and reordering, but an attacker with write access to the bucket can recompute a valid chain (and, in an unencrypted manifest, rewrite the recorded chunk hashes themselves). For protection against an adversarial bucket, keep manifest encryption enabled (the default when encryption is active): AES-GCM-encrypted manifests are authenticated, so the hashes and chain cannot be forged without the key. An unencrypted manifest also exposes plaintext chunk hashes, which lets anyone who can read it confirm guesses about the stream's content.

Raw mode (get --no-decrypt)

--no-decrypt downloads the stored (encrypted) chunks and concatenates them to stdout without decrypting. Plaintext hashes can't be verified in this mode; each chunk is instead checked against the encrypted object's size and SHA-256 recorded in the manifest (streams uploaded by s3duct >= 0.4 record both), so raw-mode output is fully integrity-verified without the decryption key. To split the output back into chunks for offline decryption, use the encrypted_size fields from the manifest. For AES-256-GCM each stored chunk is 12-byte nonce || ciphertext || 16-byte tag, i.e. plaintext size + 28 bytes.

Resume

If an upload is interrupted, the resume log (stored locally and in S3) records which chunks were successfully uploaded along with their chain signatures. On restart, s3duct fast-forwards through stdin, verifying each chunk's hash matches the log, then continues uploading from where it left off.

Edge case — truncated resume input: if stdin ends before all previously-uploaded chunks are re-verified (e.g., the source process crashed or you piped a shorter stream), s3duct will by default return an error. --no-strict-resume is available to disable this safety check.

Shell pipefail

s3duct reads stdin until EOF. It cannot detect whether the upstream process exited successfully or crashed — both produce the same EOF. Use set -o pipefail in bash (or setopt PIPE_FAIL in zsh) so your shell reports upstream failures:

set -o pipefail
pg_dump mydb | s3duct put --bucket b --name db-backup || echo "upload or dump failed"

Without pipefail, a crash in pg_dump would go unnoticed — s3duct would upload whatever partial data it received, and the pipeline would exit 0.

If stdin is a regular file (not a pipe), s3duct checks the file size against bytes read and warns on mismatch, as this may indicate the file was modified during upload.

Backpressure

s3duct monitors available disk space in its scratch directory and pauses reading from stdin when approaching the limit. By default it auto-tunes based on available disk (2-10 chunks buffered). Use --diskspace-limit or --buffer-chunks for explicit control.

License

Elastic License 2.0