hadro

A small, read-only, S3-compatible server. Point it at a local directory, or at Google Cloud Storage, and use any S3 client (boto3, MinIO, Opteryx, the AWS CLI) to list, download and query the data, including S3 Select over Parquet, JSON Lines and CSV.

It's useful for:

• Tests: give code that reads from S3 a real endpoint with no AWS account or Docker.
• Local development: serve a folder of Parquet/CSV/JSON files as buckets.
• An S3 front-end for GCS: expose GCS buckets to S3-only tools, with an in-memory cache.

hadro evolved from S1 and the cache.opteryx.app Cloud Run service. Releases up to 0.5.0a6 were an unrelated storage engine (hadrodb), which is still in this repository's history.

Install

pip install hadro          # local directories
pip install 'hadro[gcs]'   # plus Google Cloud Storage

hadro needs Python 3.11+ on Linux (x86-64 or aarch64, glibc 2.34+) or macOS on Apple silicon, the platforms rugo publishes wheels for. It does not use pyarrow, pandas or numpy.

Run

hadro ./data               # every sub-directory of ./data is a bucket

data/
├── astronauts/            -> s3://astronauts
│   └── astronauts.parquet -> s3://astronauts/astronauts.parquet
└── planets/
    └── planets.parquet

Then use it like any S3 endpoint (hadro only supports path-style addressing):

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="http://127.0.0.1:8080",
    aws_access_key_id="anything",
    aws_secret_access_key="anything",
    region_name="eu-west-2",
    config=Config(s3={"addressing_style": "path"}),
)
s3.list_objects_v2(Bucket="astronauts")

In tests

hadro.Server runs hadro in a background thread on a free port:

import hadro
import pytest

@pytest.fixture(scope="session")
def s3_endpoint():
    with hadro.Server(data="tests/data") as server:
        yield server.endpoint   # e.g. http://127.0.0.1:53817

hadro.create_app(config) returns the FastAPI app if you would rather use fastapi.testclient.TestClient or mount it yourself.

Serving GCS

hadro --backend gcs --gcs-project my-project

This uses Application Default Credentials, or STORAGE_EMULATOR_HOST for a GCS emulator. Objects up to a quarter of the cache size are kept in memory (256MB by default; see --cache-mb and --cache-ttl).

Docker / Cloud Run

docker build -t hadro .
docker run -p 8080:8080 -v "$PWD/data:/data" hadro
docker run -p 8080:8080 -e HADRO_BACKEND=gcs hadro

Configuration

Settings can be given as CLI flags, as HADRO_* environment variables, or as fields of hadro.Config.

Flag	Environment	Default
DATA (positional)	HADRO_DATA	data	Directory to serve (local backend)
--backend	HADRO_BACKEND	local	local or gcs
--gcs-project	HADRO_GCS_PROJECT		Project used to list buckets
--host	HADRO_HOST	127.0.0.1	Interface to bind
--port	HADRO_PORT, then PORT	8080
--region	HADRO_REGION	eu-west-2	Region reported by GetBucketLocation
--cache-mb	HADRO_CACHE_MB	256 (gcs), 0 (local)	In-memory object cache
--cache-ttl	HADRO_CACHE_TTL	300	Seconds before cached objects are refetched (0 = never)
--access-key	HADRO_ACCESS_KEY		Require SigV4-signed requests...
--secret-key	HADRO_SECRET_KEY		...with this key pair

With no keys set, hadro accepts any request, signed or not. With keys set, it checks SigV4 signatures in the Authorization header and in presigned URLs.

S3 API coverage

Operation
ListBuckets
HeadBucket, GetBucketLocation
ListObjects, ListObjectsV2	prefix, delimiter / CommonPrefixes, pagination, encoding-type=url
GetObject, HeadObject	single Range requests, ETag, Last-Modified, Content-Type
SelectObjectContent	Parquet, JSON Lines and CSV input; see below
Anything that writes	Rejected with 405 MethodNotAllowed
Other sub-resources (?acl, ?versioning...)	501 NotImplemented

Local-backend ETags are derived from each file's size and modification time rather than an MD5 of its contents, so they are stable and change whenever the file does.

S3 Select

response = s3.select_object_content(
    Bucket="astronauts",
    Key="astronauts.parquet",
    Expression="SELECT name, missions FROM S3Object s WHERE s.space_flights > 5 LIMIT 10",
    ExpressionType="SQL",
    InputSerialization={"Parquet": {}},
    OutputSerialization={"JSON": {}},
)
for event in response["Payload"]:
    if "Records" in event:
        print(event["Records"]["Payload"].decode())

Supported SQL:

SELECT * | column [[AS] alias], ...
FROM S3Object [[AS] alias]
[WHERE condition]
[LIMIT n]

• Comparisons = != <> < <= > >= and IS [NOT] NULL, [NOT] IN (...), [NOT] BETWEEN ... AND ..., [NOT] LIKE, combined with AND, OR, NOT and parentheses.
• Literals are converted to the column's type, so birth_date < '1960-01-01' works on date and timestamp columns.
• Unquoted column names are case-insensitive; "quoted" names are exact.
• Aggregates, functions, GROUP BY and ORDER BY are not supported.

Objects are read with rugo, the reader Opteryx uses:

Input
Parquet	Column selection, and simple ANDed conditions (col < 5, col IN (...)) are pushed into rugo, which skips row groups and filters rows as it reads
JSON Lines	<JSON><Type>LINES</Type></JSON>; CompressionType GZIP or BZIP2 allowed
CSV	FileHeaderInfo USE (columns by name) or NONE/IGNORE (_1, _2, ...); single-character FieldDelimiter; GZIP/BZIP2 allowed

Conditions rugo can't apply (OR, LIKE, IS NULL...) are evaluated over the rows it returns, with SQL's NULL semantics. Column types come from the Parquet footer, or are inferred by rugo for JSON Lines and CSV, so WHERE age > 30 compares numbers.

Output can be JSON Lines or CSV (with custom delimiters and quoting), or Parquet. Parquet output is a hadro extension: send <OutputSerialization><Parquet/></OutputSerialization>, optionally with <CompressionAlgorithm> ZSTD (the default) or NONE. SELECT * with Parquet in and out returns the original file untouched.

Results are streamed in 10,000-row Records events, followed by Stats and End.

Development

make install   # creates .venv with test and gcs extras
make test
make lint
make run       # serves ./data on port 8080

Migrating from S1 / cache.opteryx.app

• The package is hadro; start it with hadro or python -m hadro rather than python src/main.py.
• Environment variables now have a HADRO_ prefix: STORAGE_BACKEND → HADRO_BACKEND, LOCAL_STORAGE_PATH → HADRO_DATA, GCS_PROJECT → HADRO_GCS_PROJECT, S1_ACCESS_KEY/S1_SECRET_KEY → HADRO_ACCESS_KEY/HADRO_SECRET_KEY. STORAGE_CACHE_SIZE (a count of objects) is replaced by HADRO_CACHE_MB.
• The default backend is now local, and the default host is 127.0.0.1.
• Errors are S3 XML documents (NoSuchKey, NoSuchBucket, ...) instead of plain text.

License

Apache 2.0; see LICENSE.