Crystalia Collector

Generate dataset descriptors for large S3-hosted datasets as RDF annotations.

For architecture, extensibility, and infrastructure details see docs/architecture.md.

Installation

uv sync

Usage

# List files in an S3 prefix
crystalia-collector list my-bucket/path/to/data

# List and generate task files for parallel processing
crystalia-collector list my-bucket/path/to/data --task-dir ./tasks --method-id md5-8gb

# Compute checksum for a single S3 object
crystalia-collector checksum s3://my-bucket/path/to/file.gz

# Compute checksum for a byte range
crystalia-collector checksum s3://my-bucket/path/to/file.gz --offset 0 --length 8589934592

# Process a task file and generate RDF annotations
crystalia-collector annotate tasks/task_1 --output-file annotations.rdf

# Combine annotations (placeholder — not yet implemented)
crystalia-collector combine

Checksum Methods

Block-based methods split large files into offset/length pairs in task files, enabling parallel checksum computation.

Configuration

Settings load in this order (highest priority first):

1. Environment variables (CRYSTALIA_* prefix, or bare SENTRY_DSN)
2. ./crystalia.toml in the current directory
3. ~/.config/crystalia-collector/config.toml
4. Built-in defaults

Environment Variables

Config File

# crystalia.toml or ~/.config/crystalia-collector/config.toml
default_method_id = "md5-8gb"
default_output_file = "out.rdf"
enable_telemetry = false
log_json = false

Copy .envrc.example to .envrc and adjust as needed.

Docker

docker build -t crystalia-collector .
docker run --rm crystalia-collector --help

Development

uv sync
uv run pytest
uv run ruff check .
uv run mypy src/

License

Apache-2.0