boto3-s3

A complete library version of the aws s3 command, with a faithful aws s3 sync at its core. Every subcommand — cp / ls / mb / mv / presign / rb / rm / sync / website — runs in-process from Python, not by shelling out to the CLI. Byte transfers run on the same s3transfer engine as aws s3, so transfer performance keeps pace with the command.

Each command is a method on a single S3 object, taking ordinary keyword arguments; bring a boto3 client when you need a specific profile, region, or endpoint.

Status: early development (pre-1.0) — all subcommands are implemented; the public API may still change. Python: 3.10+ · License: Apache-2.0

Two packages:

• boto3-s3 — the library. Run aws s3-equivalent operations from Python with your own boto3 clients and credentials.
• boto3-s3-cli — the boto3-s3 command, a drop-in for aws s3 (details).

Why

Much of aws s3 is easy to do straight from boto3 — a one-off cp or rm is a few lines, and recursive copies or multipart via s3transfer take only a bit more effort. What's genuinely hard is matching aws s3 faithfully: its path and naming rules, recursive and include/exclude semantics, and above all a sync that compares and transfers exactly like the command — fiddly to reproduce and easy to get subtly wrong. The usual fallbacks are shelling out to the aws s3 command and parsing its output, or a partial reimplementation that drifts from it. boto3-s3 is neither: it reproduces aws s3's behavior across the whole command set — sync included — as a library you call directly.

• A faithful aws s3 sync. Mirror local trees and buckets in any direction — upload, download, or S3-to-S3 — with --delete, the same size/timestamp comparison as aws-cli, include/exclude, and dry-run.
• Every aws s3 command. cp / ls / mb / mv / presign / rb / rm / website complete the set.
• A library, not a CLI wrapper. Runs in-process: no subprocess, no scraping stdout, no aws on PATH. You pass boto3 clients directly and get structured, per-item results back — not text to parse.
• Light enough for a Lambda. The Python runtime already ships boto3 (and its botocore / s3transfer deps), so adding boto3-s3 costs well under a megabyte for the full aws s3 feature set in-process — where bundling aws-cli (250 MB+) overruns the deployment size limit and shelling out isn't practical.
• Transfer speed on par with aws-cli. Byte transfers use the same engine as aws s3 (s3transfer, or the optional CRT engine), so large transfers run at the same speed — no penalty for being a library.
• Familiar behavior. Path rules, options, and (for the CLI) exit codes follow aws s3, so what you know from the command carries over.

Install

pip install boto3-s3          # the library
pip install boto3-s3-cli      # the `boto3-s3` command (also installs boto3-s3)

Optional extra — the AWS Common Runtime (CRT) transfer engine and CRT-family checksums:

pip install "boto3-s3[crt]"

Quick start

Create the S3 object once — it holds no connection of its own (nothing to close) and is safe to share across threads — then call what you need:

from boto3_s3 import S3

s3 = S3()

# Sync a directory tree up to S3, removing remote extras (mirror).
s3.sync("./site", "s3://my-bucket/site/", delete=True)

# Copy a single object up or down.
s3.cp("./report.csv", "s3://my-bucket/report.csv")
s3.cp("s3://my-bucket/report.csv", "./report.csv")

# List objects lazily; each item is a FileInfo (key, size, …).
for info in s3.ls("s3://my-bucket/site/", recursive=True):
    print(info.key, info.size)

# Delete everything under a prefix.
s3.rm("s3://my-bucket/tmp/", recursive=True)

# A presigned URL (no request is sent).
url = s3.presign("s3://my-bucket/report.csv", expires_in=900)

For cp / mv / sync the direction is inferred from the two endpoints: local-to-S3 is an upload, S3-to-local a download, S3-to-S3 a copy. A local-to-local pair is rejected, like aws s3.

Sync

sync is the heart of the library — a faithful re-creation of aws s3 sync, callable from Python, in every direction:

s3.sync("./site", "s3://my-bucket/site/")       # upload
s3.sync("s3://my-bucket/site/", "./site")       # download
s3.sync("s3://src/data/", "s3://dst/data/")     # S3-to-S3

It supports the flags you know from the command:

• delete=True — remove destination entries the source no longer has. Items hidden by a filter stay out of deletion too, exactly like aws s3 sync.
• copy_filter= — how the source and destination are compared: DefaultCopyFilter(size_only=True) / (exact_timestamps=True) tune the default; True copies everything, False copies nothing.
• filter= — include/exclude matching; dryrun=True to preview every transfer and deletion first.

Because it runs in-process, sync hands back structured results that aws s3 can't: on_result fires once per item as the run proceeds, so you know exactly what changed without parsing any output. Each result carries a kind (upload / download / copy / delete) and an outcome (succeeded / failed / warned / skipped):

from boto3_s3 import OpKind, OpOutcome

uploaded = []

def track(r):
    if r.kind is OpKind.UPLOAD and r.outcome is OpOutcome.SUCCEEDED:
        uploaded.append(r.key)

s3.sync("./site", "s3://my-bucket/site/", delete=True, on_result=track)
print(f"{len(uploaded)} files uploaded")

Operations

S3 is the entry point: create one with s3 = S3(), then call the methods below — each mirrors an aws s3 subcommand.

Choosing the client (profile, region, endpoint, cross-account)

A path argument is a str, an os.PathLike, or an S3Storage. A bare "s3://..." string uses a default boto3.client("s3").

To pick a specific profile, region, or endpoint (e.g. MinIO), or a second account for an S3-to-S3 copy, build the client yourself and wrap the URL in an S3Storage:

import boto3
from boto3_s3 import S3, S3Storage

session = boto3.Session(profile_name="prod")
client = session.client("s3", region_name="eu-west-1")

s3 = S3()
s3.cp("./artifact.tar.gz", S3Storage("s3://prod-bucket/artifacts/", client=client))

An S3-compatible endpoint such as MinIO is just a differently-built client:

minio = boto3.client(
    "s3",
    endpoint_url="http://localhost:9000",
    aws_access_key_id="minioadmin",
    aws_secret_access_key="minioadmin",
)
for info in S3().ls(S3Storage("s3://bucket/", client=minio)):
    print(info.key)

A cross-account S3-to-S3 copy uses one client per side:

s3.cp(
    S3Storage("s3://src-bucket/data/", client=src_client),
    S3Storage("s3://dst-bucket/data/", client=dst_client),
    recursive=True,
)

The bucket part of an S3 URI may also be an access-point ARN (plain or Outposts), passed through as the Bucket, like aws s3.

Options

cp / mv / sync take the aws s3 transfer options as snake_case keyword arguments, grouped here by what they control:

• Metadata & headers: metadata, metadata_directive, copy_props, cache_control, content_type / content_disposition / content_encoding / content_language, expires, website_redirect, guess_mime_type
• Access & storage class: acl, grants, storage_class, request_payer
• Encryption: sse, sse_kms_key_id, sse_c / sse_c_key (and the copy-source pair)
• Integrity & write control: checksum_algorithm, checksum_mode, no_overwrite, case_conflict
• Glacier: force_glacier_transfer / ignore_glacier_warnings

s3.cp(
    "./photo.jpg",
    "s3://my-bucket/photo.jpg",
    storage_class="STANDARD_IA",
    content_type="image/jpeg",
    metadata={"reviewed": "yes"},
    acl="bucket-owner-full-control",
)

Multipart tuning (thresholds, concurrency, bandwidth, the classic/CRT engine choice) is a TransferConfig passed as transfer_config=; its defaults match aws s3.

Filtering

cp / mv / rm / sync take a filter= that decides which items stay in the operation. The common form is an aws s3-style include/exclude matcher (last match wins, default-include):

from boto3_s3 import globsieve

keep = globsieve.compile([
    globsieve.GlobPattern.exclude("*"),
    globsieve.GlobPattern.include("*.tar.gz"),
])
s3.cp("./build", "s3://artifacts/", recursive=True, filter=keep)

filter= also accepts a plain Callable[[FileInfo], bool] for arbitrary per-item gating. sync adds copy_filter= and delete= (True to remove all extras, or a filter to remove only matching ones). The two filters answer different questions: filter= decides which items take part at all, while copy_filter= then decides, for each matched source/destination pair, whether it actually needs copying.

Progress, results, cancellation, dry run

Batch operations stream their per-item outcomes instead of returning a list:

• on_result(OpResult) — fires once per item as the run proceeds. Each OpResult carries a kind (upload / download / copy / delete) and an outcome (succeeded / failed / warned / skipped). It is called from worker threads, so keep it fast and non-raising.
• on_progress(TransferProgress) — byte-level transfer progress.
• cancel_token — a CancelToken whose cancel() cooperatively stops the run.
• dryrun=True — reports every would-be action without any mutating call.

Errors

Every failure is a Boto3S3Error subclass — catch the root to catch them all.

BatchError carries summary counts (succeeded / failed / warned / skipped / total); the per-item detail arrives live through on_result.

Debug logging

boto3 / botocore debug logs leak signed headers, signatures, and session tokens. set_stream_logger mirrors boto3.set_stream_logger but redacts those by default:

from boto3_s3 import set_stream_logger

set_stream_logger("botocore")  # credentials masked unless mask_secrets=False

Compatibility

• Python: 3.10 and later.
• OS: Linux, macOS, Windows (path-separator and case-sensitivity behavior is matched to aws s3 on each).
• AWS SDK: boto3 >= 1.28, botocore >= 1.31, s3transfer >= 0.6.2. A few options need a newer SDK and are simply unavailable below it rather than emulated — conditional writes (no_overwrite), the CRC64NVME checksum, and the ls bucket-name / bucket-region filters. CRT features need the crt extra. Everything else works at the minimum.

In short

Every aws s3 operation as an in-process Python call — no subprocess, no stdout to scrape, structured per-item results back.

Contributing

Bug reports, questions, and ideas are welcome on the issue tracker. To work on the code, CONTRIBUTING.md covers local setup (uv), the test suite, and the coding and commit conventions.

License

Apache-2.0. See LICENSE.

Source and issues: https://github.com/izumo-m/boto3-s3.