boto3-s3

A Python library for running every aws s3 operation in-process, with an aws s3 sync-compatible synchronization pipeline at its core. Applications that currently shell out to aws s3 sync can make the same default size/timestamp decisions, transfers, filters, and deletions through S3.sync without starting a CLI process. Every subcommand — cp / ls / mb / mv / presign / rb / rm / sync / website — has a corresponding Python API.

The library provides the building blocks for aws s3 compatibility and is occasionally more permissive for Python callers. The companion CLI is the layer that applies the command's strict argument validation and exit-code behavior.

Each command is a method on a single S3 object, taking ordinary keyword arguments; bring a boto3 client when you need a specific profile, region, or endpoint.

Status: early development (pre-1.0) — all subcommands are implemented; the public API may still change. Python: 3.10+ · License: Apache-2.0

Two packages:

• boto3-s3 — the library. Run aws s3-equivalent operations from Python with your own boto3 clients and credentials.
• boto3-s3-cli — the boto3-s3 command, a drop-in for aws s3.

Why

Much of aws s3 is easy to do straight from boto3 — a one-off cp or rm is a few lines, and recursive copies or multipart via s3transfer take only a bit more effort. What's genuinely hard is preserving the command's path and naming rules, recursive include/exclude semantics, and especially the decisions made by aws s3 sync. Reimplementing those rules in each application invites subtle drift; shelling out keeps the rules but leaves the application parsing process output. boto3-s3 brings those command semantics into a direct Python API.

• An in-process replacement for aws s3 sync. Mirror local trees and buckets in any direction — upload, download, or S3-to-S3 — with the same default size/timestamp comparison, include/exclude behavior, deletion, and dry-run.
• Every aws s3 command. cp / ls / mb / mv / presign / rb / rm / website complete the set.
• A library, not a CLI wrapper. Runs in-process: no subprocess, no scraping stdout, no aws on PATH. You pass boto3 clients directly and get structured, per-item results back — not text to parse.
• Small, pure-Python packaging. boto3-s3 can reuse a compatible existing boto3 / botocore / s3transfer installation, including a Lambda runtime SDK when it satisfies the supported version floor.
• The established AWS transfer engines. Byte transfers use s3transfer or the optional CRT engine, retaining their multipart and concurrency machinery instead of introducing another byte-transfer implementation.
• Familiar behavior. Path rules, options, and default sync decisions follow aws s3; the CLI additionally owns strict validation and exit-code parity.

Install

pip install boto3-s3          # the library
pip install boto3-s3-cli      # the `boto3-s3` command (also installs boto3-s3)

Optional extra — the AWS Common Runtime (CRT) transfer engine and CRT-family checksums:

pip install "boto3-s3[crt]"

Quick start

Create the S3 object once — it holds no connection of its own and needs no cleanup — then call what you need. Its own state is safe to share, but parallel operations require separate, prebuilt clients as described below.

from boto3_s3 import S3

s3 = S3()

# Sync a directory tree up to S3, removing remote extras (mirror).
s3.sync("./site", "s3://my-bucket/site/", delete_filter=True)

# Copy a single object up or down.
s3.cp("./report.csv", "s3://my-bucket/report.csv")
s3.cp("s3://my-bucket/report.csv", "./report.csv")

# List objects; each result is a FileInfo (key, size, …).
s3.ls(
    "s3://my-bucket/site/",
    recursive=True,
    on_result=lambda info: print(info.key, info.size),
)

# Delete everything under a prefix.
s3.rm("s3://my-bucket/tmp/", recursive=True)

# A presigned URL (no request is sent).
url = s3.presign("s3://my-bucket/report.csv", expires_in=900)

For cp / mv / sync the direction is inferred from the two endpoints: local-to-S3 is an upload, S3-to-local a download, S3-to-S3 a copy. A local-to-local pair is rejected, like aws s3.

Sync

sync is the heart of the library: its default pipeline is designed as an in-process replacement for aws s3 sync, in every direction:

s3.sync("./site", "s3://my-bucket/site/")       # upload
s3.sync("s3://my-bucket/site/", "./site")       # download
s3.sync("s3://src/data/", "s3://dest/data/")     # S3-to-S3

sync makes one of three decisions per entry — the same three cases as aws s3 sync, each with its own filter:

• create_filter — whether to create an entry that is new (in the source, not yet at the destination). True (default) creates every one; False creates none; a predicate creates only the ones it keeps. (aws always creates; this is the knob aws does not expose.)
• update_filter — for an entry present on both sides, whether to overwrite it: None (default) decides by size + mtime (equivalently AwsCliComparison(), tuned via AwsCliComparison(size_only=True) / (exact_timestamps=True)); True always, False never (leave existing as-is); or a content strategy EtagComparison(s3) / ChecksumComparison(s3, src, dest) (wrap any lane's filter in ParallelFilter(fn, executor=pool) to decide on a caller-supplied thread pool).
• delete_filter — whether to delete an orphan (at the destination, no longer in the source). False (default) keeps orphans; True deletes every one (aws s3 sync --delete); a predicate deletes only the ones it keeps. Items hidden by filter stay out of deletion too, exactly like aws s3 sync.

create_filter and delete_filter are the two membership knobs (create / delete); update_filter is the overwrite judgment for the entries on both sides. filter= is separate — it decides which entries are visible at all (pruning both sides symmetrically, like --exclude / --include); dryrun=True previews every transfer and deletion first.

# Full mirror: create new, overwrite changed, delete removed.
s3.sync("./site", "s3://my-bucket/site/", delete_filter=True)

# Update-only mirror: refresh existing files and prune deleted ones,
# but never publish brand-new files.
s3.sync("./site", "s3://my-bucket/site/", create_filter=False, delete_filter=True)

The content strategies are opt-in submodule imports — from boto3_s3.etagcompare import EtagComparison / from boto3_s3.checksumcompare import ChecksumComparison (and from boto3_s3.awsclicompare import AwsCliComparison to tune the default). ParallelFilter imports from boto3_s3 itself.

Because it runs in-process, sync provides structured results without parsing console output. on_result receives terminal item outcomes as the run proceeds, plus any additional warning or notice records. Each result carries a transfer_type (upload / download / copy / delete) and an outcome (succeeded / failed / warned / skipped / dryrun / notice):

from boto3_s3 import TransferType, OpOutcome

uploaded = []

def track(r):
    if r.transfer_type is TransferType.UPLOAD and r.outcome is OpOutcome.SUCCEEDED:
        uploaded.append(r.key)

s3.sync("./site", "s3://my-bucket/site/", delete_filter=True, on_result=track)
print(f"{len(uploaded)} files uploaded")

Operations

S3 is the entry point: create one with s3 = S3(), then call the methods below — each mirrors an aws s3 subcommand.

Choosing the client (profile, region, endpoint, cross-account)

A path argument is a str, an os.PathLike, or an S3Storage. A bare "s3://..." string uses the client the S3 instance builds from its own defaults — boto3.client("s3") for a zero-config S3().

For a specific profile, region, or endpoint, hand the S3 object a boto3.Session (and/or an endpoint_url / config); every bare "s3://..." string then inherits it:

import boto3
from boto3_s3 import S3, S3Storage

session = boto3.Session(profile_name="prod", region_name="eu-west-1")
s3 = S3(session=session)
s3.cp("./artifact.tar.gz", "s3://prod-bucket/artifacts/")

When a single operation needs more than one client — a cross-account S3-to-S3 copy is the clearest case — the instance default can't express it. Build each client yourself and wrap the URL in an S3Storage, which is used verbatim with its own client:

s3.cp(
    S3Storage("s3://src-bucket/data/", client=src_client),
    S3Storage("s3://dest-bucket/data/", client=dest_client),
    recursive=True,
)

An S3-compatible endpoint such as MinIO is just a differently-built client — set it on the S3 for every location, or pass a single S3Storage(url, client=minio) when only one side needs it:

minio = boto3.client(
    "s3",
    endpoint_url="http://localhost:9000",
    aws_access_key_id="minioadmin",
    aws_secret_access_key="minioadmin",
)
S3().ls(
    S3Storage("s3://bucket/", client=minio),
    on_result=lambda info: print(info.key),
)

The bucket part of an S3 URI may also be an access-point ARN (plain or Outposts), passed through as the Bucket, like aws s3.

Running operations across threads

An S3 object's own state is safe to share across threads, but parallel operations need care on two fronts: boto3 sessions are not safe for concurrent client construction, and s3transfer's per-transfer setup is not safe on a client shared by concurrent operations. Build the clients sequentially before starting the workers, then give each concurrent operation its own client through S3Storage:

import concurrent.futures
import boto3
from boto3_s3 import S3, S3Storage

s3 = S3()
session = boto3.Session(profile_name="prod", region_name="eu-west-1")
jobs = [
    (
        path,
        S3Storage(
            f"s3://prod-bucket/{path.name}",
            client=session.client("s3"),
        ),
    )
    for path in paths
]

with concurrent.futures.ThreadPoolExecutor() as pool:
    for path, dest in jobs:
        pool.submit(s3.cp, str(path), dest)

Do not use bare "s3://..." arguments for this pattern: they call client() inside each operation and would build clients concurrently.

Options

cp / mv / sync take the aws s3 transfer options as snake_case keyword arguments plus documented library extensions, grouped here by what they control:

• Metadata & headers: metadata, metadata_directive, copy_props, cache_control, content_type / content_disposition / content_encoding / content_language, expires, website_redirect, guess_mime_type
• Access & storage class: acl, grants, storage_class, request_payer
• Encryption: sse, sse_kms_key_id, sse_c / sse_c_key (and the copy-source pair)
• Integrity & write control: checksum_algorithm, checksum_mode, no_overwrite, case_conflict
• Glacier: force_glacier_transfer / ignore_glacier_warnings
• Annotation staging (library-only): annotation_copy_mode

s3.cp(
    "./photo.jpg",
    "s3://my-bucket/photo.jpg",
    storage_class="STANDARD_IA",
    content_type="image/jpeg",
    metadata={"reviewed": "yes"},
    acl="bucket-owner-full-control",
)

Multipart tuning (thresholds, concurrency, bandwidth, the classic/CRT engine choice) is a TransferConfig passed as transfer_config=; its defaults match aws s3. For multipart S3-to-S3 copies under copy_props=ALL, annotations are preloaded in memory by default to match aws-cli's failure timing. Library callers can select AnnotationCopyMode.PRELOAD_TEMPFILE (configured by TransferConfig(annotation_temp_dir=...)) or the lower-overhead AnnotationCopyMode.DEFERRED; the CLI deliberately exposes no corresponding flag.

Filtering

cp / mv / rm / sync take a filter= that decides which items stay in the operation. The common form is an aws s3-style include/exclude matcher (last match wins, default-include):

from boto3_s3 import GlobFilter

keep = GlobFilter().exclude("*").include("*.tar.gz").compile()
s3.cp("./build", "s3://artifacts/", recursive=True, filter=keep)

filter= also accepts a plain Callable[[FileInfo], bool] for arbitrary per-item gating. sync adds one filter per decision — create_filter= (create a new entry?), update_filter= (overwrite one on both sides?), delete_filter= (delete an orphan?). They answer different questions from filter=: filter= decides which items take part at all (visible on either side), while the three lane filters decide what to do with the entries that do — create, overwrite, delete.

Progress, results, cancellation, dry run

Batch operations stream item records instead of returning a list:

• on_result(OpResult) — receives each terminal item outcome as the run proceeds. An item may also emit a WARNED or NOTICE record, so callers must not assume exactly one callback per key. Callbacks may come from the calling thread or worker threads, and may be concurrent; keep them thread-safe, fast, and non-raising.
• on_progress(TransferProgress) — byte-level transfer progress (cp / mv / sync; rm moves no bytes).
• S3.ls(..., on_result=...) — receives each listed FileInfo, in listing order on the calling thread.
• cancel_token — a CancelToken whose cancel() stops new work and drains accepted work (ls / cp / mv / rm / sync). Pass mode=CancelMode.IMMEDIATE to additionally request best-effort cancellation of pending and in-flight work. A later immediate request upgrades an earlier graceful request; cancellation never downgrades.
• dryrun=True — reports every would-be action without any mutating call.

Custom backends

cp / mv / sync aren't limited to local paths and S3: a custom Storage subclass — an HTTP service, an archive, an in-memory store — can be one side of a transfer, the other side always S3 (the built-in IOStorage / StdioStorage stream wrappers are this same seam). A backend declares its capabilities, which a transfer pre-checks, failing fast if it needs more.

See docs/storage.md for the Storage contract, capabilities, and a worked example.

Errors

Recognized operational failures use the Boto3S3Error hierarchy, so catching the root handles the library's translated S3, filesystem, validation, and configuration errors. Programming errors and a small number of documented dependency pass-throughs are not wrapped.

BatchError carries summary counts (succeeded / failed / warned / skipped / total); the per-item detail arrives live through on_result.

Debug logging

boto3 / botocore debug logs leak signed headers, signatures, and session tokens. set_stream_logger mirrors boto3.set_stream_logger but redacts those by default:

from boto3_s3 import set_stream_logger

set_stream_logger("botocore")  # credentials masked unless mask_secrets=False

Compatibility

• Python: 3.10 and later.
• OS: Linux, macOS, Windows (path-separator and case-sensitivity behavior is matched to aws s3 on each).
• AWS SDK: boto3 >= 1.28, botocore >= 1.31, s3transfer >= 0.6.2. Features introduced by newer S3 models degrade rather than being emulated. Notable examples include conditional writes (no_overwrite), CRC64NVME, paginated bucket filters, newer mb tags / account-regional namespace fields, S3 object annotations, copy_props="all", and source-ETag response extras. CRT features need the crt extra. See docs/overview.md for the authoritative version requirements and degradation behavior.

In short

Every aws s3 operation as an in-process Python call — no subprocess, no stdout to scrape, structured per-item results back.

Contributing

Bug reports, questions, and ideas are welcome on the issue tracker. To work on the code, CONTRIBUTING.md covers local setup (uv), the test suite, and the coding and commit conventions.

License

Apache-2.0. See LICENSE.

Source and issues: https://github.com/izumo-m/boto3-s3.