Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

theblob

An S3-compatible object store written in Python. It starts as one command on one machine, and is designed to grow into a redundant cluster.

It is meant to replace self-hosted MinIO: single-server installs first, then clusters (not there yet).

Quick start

uv sync
uv run theblob serve --data-dir ./data

The first start creates an access key and prints it once. Use it with any S3 client:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://127.0.0.1:9000",
    aws_access_key_id="...",
    aws_secret_access_key="...",
    region_name="us-east-1",
)
s3.create_bucket(Bucket="my-bucket")
s3.put_object(Bucket="my-bucket", Key="hello.txt", Body=b"hello")

Other commands: theblob key create|list|delete|grant|revoke and theblob fsck. Run theblob serve --help for the server options, including TLS and virtual-hosted addressing.

Presigned URLs need SigV4. boto3 presigns with SigV2 by default, which theblob refuses, so create the client with config=Config(signature_version="s3v4").

Moving from MinIO or Garage

With a rclone remote for each store (provider = Other, force_path_style = true), and the bucket created on theblob:

rclone sync --metadata old:my-bucket theblob:my-bucket
rclone check old:my-bucket theblob:my-bucket

tests/bench/run_migration.py checks this from both: the bytes, Content-Type, Cache-Control and x-amz-meta-* arrive, including multipart objects and unusual keys. These don't, and need setting again: object tags, ACLs, lifecycle rules, CORS, keys, and any version but the current one. rclone also adds mtime and btime to the metadata. An object's time on theblob is when it was copied.

In a container

docker build -t theblob .
docker run -d --name theblob -p 9000:9000 -v theblob-data:/data theblob
docker logs theblob   # the first start prints an access key once

Set --workers after the image name if the default, one per CPU, uses more memory than you want. The data format is upgraded when a newer version starts: back up first, with theblob backup. A store written by an earlier release is tested to open in this one, from samples in tests/upgrade/stores/.

Access keys

A key has full access unless it is created as scoped. A scoped key has no rights until it is granted some, per bucket, at one of three levels: read, write (which includes read) or admin (which also allows deleting the bucket).

uv run theblob key create --scoped --data-dir ./data
uv run theblob key grant ACCESS_KEY my-bucket read --data-dir ./data
uv run theblob key revoke ACCESS_KEY my-bucket --data-dir ./data

A grant or a revocation applies from the next request. A scoped key cannot create buckets, sees only its own buckets when listing, and cannot tell a bucket it has no grant on from one that does not exist. Grants disappear with their bucket, so a bucket deleted and created again starts with none.

Development

make test      # the whole test pyramid
make lint      # ruff, ty, pyrefly, mypy

Tests are in tests/: a_unit for pure logic, b_integration for real SQLite and files, c_e2e for real S3 clients (boto3, AWS CLI, rclone, s5cmd) against a real server over HTTP and HTTPS. The rclone, s5cmd and AWS CLI v2 tests are skipped if those tools are not installed.

License

Apache-2.0: see LICENSE.

Metadata

Release files for theblob 0.4.0.dev1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for theblob 0.4.0.dev1
File Size Uploaded Provenance Actions
theblob-0.4.0.dev1.tar.gz 95.3 kB Without provenance Details for theblob-0.4.0.dev1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for theblob 0.4.0.dev1
File Interpreter ABI Platform Actions
theblob-0.4.0.dev1-py3-none-any.whl Python 3 none any Details for theblob-0.4.0.dev1-py3-none-any.whl

Total release size: 204.1 kB

Release files / theblob-0.4.0.dev1.tar.gz

Download URL theblob-0.4.0.dev1.tar.gz
Size 95.3 kB
Tags Source
SHA-256 checksum
How to use checksums
9dcc3f2a1ce0090000d75d73ca4b99b9eba1e7f1cca0178346d2f293be0ef985
BLAKE2b-256 checksum
How to use checksums
b61476cfd3af8e295701ab4a50059fda2796f9a2490eeda4062e8673aab03845
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / theblob-0.4.0.dev1-py3-none-any.whl

Download URL theblob-0.4.0.dev1-py3-none-any.whl
Size 108.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
aaa1febcc85445a9185c0bae103e8b34f8cae90457578a6a4e964cf70b63d35c
BLAKE2b-256 checksum
How to use checksums
752e1f098e07dcde1ea74b9be5bb6a1da903af1f024da032d567982ccb3d4111
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for theblob 0.4.0.dev1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page