s3bolt 0.2.0

High-performance S3 file copy tool — concurrent, async, built in Rust

s3bolt

High-performance S3 file copy tool — concurrent, async, built in Rust.

Copy S3 objects between buckets and prefixes at maximum throughput. Uses server-side copy, adaptive concurrency, and async I/O. Available as a Rust crate, a CLI tool, and a Python package.

Why s3bolt?

	aws s3 cp	s3bolt
Concurrency	~10 concurrent transfers	Up to 1024 with adaptive AIMD
Large files (>5 GiB)	Multipart upload (downloads data)	Multipart server-side copy (zero data transfer)
Throttle handling	Fixed retry	Adaptive backoff (halves concurrency on 503)
Resume	No	Checkpoint file for resumable copies
Filtering	Basic --exclude	Glob, regex, size range, date range
Cross-account	Single profile	Separate source/dest profiles

Installation

CLI (Rust)

cargo install s3bolt

Python

pip install s3bolt

From source

git clone https://github.com/cykruss/s3bolt.git
cd s3bolt
cargo build --release
# Binary at target/release/s3bolt

Quick start

CLI

# Copy a single file
s3bolt s3://src-bucket/data/file.parquet s3://dst-bucket/data/file.parquet

# Recursive prefix copy
s3bolt -r s3://src-bucket/data/2024/ s3://dst-bucket/archive/2024/

# Sync (only copy new/changed objects)
s3bolt -r --sync s3://data-lake/raw/ s3://data-lake/curated/

# Filter by pattern
s3bolt -r --include "**/*.parquet" --exclude "_tmp/**" s3://src/ s3://dst/

# Cross-account with different AWS profiles
s3bolt -r --source-profile prod --dest-profile analytics s3://prod/ s3://analytics/

# High concurrency
s3bolt -r -j 512 s3://src/ s3://dst/

# Dry run (see what would be copied)
s3bolt -r --dry-run s3://src/ s3://dst/

# Resumable copy with checkpoint
s3bolt -r --checkpoint /tmp/copy.ckpt s3://src/ s3://dst/
# If interrupted, resume with:
s3bolt -r --checkpoint /tmp/copy.ckpt --resume s3://src/ s3://dst/

Python

from s3bolt import S3CopyEngine

engine = S3CopyEngine(source_profile="prod", dest_profile="analytics")

result = engine.copy(
    "s3://src-bucket/data/",
    "s3://dst-bucket/data/",
    recursive=True,
    include=["**/*.parquet"],
    max_concurrent=512,
)

print(f"Copied {result['copied_objects']} objects ({result['copied_bytes']} bytes)")
print(f"Skipped {result['skipped_objects']} | Failed {result['failed_objects']}")
print(f"Duration: {result['duration_secs']:.1f}s")

Rust

use std::sync::Arc;
use s3bolt::config::{CopyConfig, ConcurrencyConfig, FilterConfig};
use s3bolt::engine::orchestrator;
use s3bolt::progress::reporter::ProgressState;
use s3bolt::types::S3Uri;

#[tokio::main]
async fn main() -> s3bolt::error::Result<()> {
    let config = CopyConfig {
        source: S3Uri::parse("s3://src-bucket/prefix/")?,
        destination: S3Uri::parse("s3://dst-bucket/prefix/")?,
        recursive: true,
        sync_mode: false,
        dry_run: false,
        verify: false,
        filters: FilterConfig::default(),
        concurrency: ConcurrencyConfig::default(),
        checkpoint_path: None,
        resume: false,
        storage_class: None,
        sse: None,
        preserve_metadata: false,
        source_profile: None,
        dest_profile: None,
    };

    let progress = Arc::new(ProgressState::default());
    let manifest = orchestrator::run(config, progress).await?;
    println!("Copied {} objects", manifest.copied_objects);
    Ok(())
}

Architecture

ListObjectsV2 (async paginator stream)
       │
       ▼
[bounded channel, cap=10,000]  ← backpressure: listing pauses when full
       │
       ▼
Filter stage (glob, regex, size, date)
       │
       ▼
[tokio::sync::Semaphore, permits=N]  ← adaptive concurrency (AIMD)
       │
       ▼
Worker tasks (spawn per object)
  ├── CopyObject (≤ 5 GiB) ─── server-side, zero data transfer
  └── UploadPartCopy (> 5 GiB) ─ parallel multipart, server-side
       │
       ▼
Progress reporter + checkpoint writer

Performance design

• Tokio async runtime — tens of thousands of concurrent I/O tasks on a small thread pool. No OS thread overhead.
• Server-side copy — CopyObject and UploadPartCopy move data within S3's network. The client only sends metadata requests (~50-200ms latency each).
• Adaptive concurrency (AIMD) — starts at the configured limit (default 256), ramps up on sustained success, halves on S3 503 SlowDown responses. Respects S3's 3,500 PUT/5,500 GET per-prefix rate limits automatically.
• Bounded backpressure — a 10,000-item channel between the lister and copy workers. If workers fall behind, listing pauses. Memory stays bounded (~2-3 MiB for the queue).
• Multipart for large files — objects > 5 GiB are split into 256 MiB parts, each copied server-side in parallel. Automatic cleanup (abort) on failure.
• Zero unnecessary data transfer — data never flows through the client for same-region copies. Pure metadata orchestration.

CLI reference

s3bolt [OPTIONS] <SOURCE> <DESTINATION>

Arguments:
  <SOURCE>        Source S3 URI (s3://bucket/key or s3://bucket/prefix/)
  <DESTINATION>   Destination S3 URI

Copy options:
  -r, --recursive              Recursively copy all objects under prefix
      --sync                   Only copy new/changed objects
      --dry-run                List objects without copying
      --verify                 Verify ETag after copy
      --storage-class <CLASS>  Override storage class

Filtering:
      --include <GLOB>         Include keys matching glob (repeatable)
      --exclude <GLOB>         Exclude keys matching glob (repeatable)
      --key-regex <REGEX>      Include keys matching regex
      --min-size <BYTES>       Minimum object size
      --max-size <BYTES>       Maximum object size

Concurrency:
  -j, --concurrency <N>       Max concurrent copies [default: 256]
      --no-adaptive            Disable adaptive concurrency

Resume:
      --checkpoint <FILE>      Checkpoint file for resume support
      --resume                 Resume from existing checkpoint

Credentials:
      --source-profile <NAME>  AWS profile for source
      --dest-profile <NAME>    AWS profile for destination

Output:
  -v, --verbose                Debug logging
  -q, --quiet                  Errors only

Prerequisites

• AWS credentials configured via any standard method (env vars, ~/.aws/credentials, IAM role, SSO)
• The executing role must have s3:GetObject + s3:ListBucket on source and s3:PutObject on destination
• For multipart copies: s3:AbortMultipartUpload on destination

Minimal IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::source-bucket",
        "arn:aws:s3:::source-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
      "Resource": [
        "arn:aws:s3:::dest-bucket",
        "arn:aws:s3:::dest-bucket/*"
      ]
    }
  ]
}

Development

# Clone
git clone https://github.com/cykruss/s3bolt.git
cd s3bolt

# Rust tests
cargo test --no-default-features

# Clippy
cargo clippy --no-default-features --lib -- -D warnings

# Build Python package (dev mode)
python -m venv .venv
source .venv/bin/activate
pip install maturin pytest
maturin develop --release
pytest tests/ -v

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT