Sunstone Data Platform

A data lakehouse for importing, versioning, and querying external datasets with full provenance tracking. Built on Apache Iceberg, iceberg-catalog, RustFS, and DuckDB.

Data scientists can import CSV/Excel files from local paths or URLs, query them via DuckDB with git-like version references, and track lineage through notebook sessions. All provenance is captured using W3C PROV-DM so metadata can later materialize as a knowledge graph.

Architecture

                        ┌──────────────┐
                        │  sunstone    │
                        │  CLI (Typer) │
                        └──────┬───────┘
                               │
              ┌────────────────┼────────────────┐
              │                │                │
     ┌────────▼───────┐ ┌─────▼──────┐ ┌───────▼────────┐
     │ Import Pipeline │ │   Query    │ │   Versioning   │
     │ fetch → parse → │ │  (DuckDB)  │ │ branches/tags/ │
     │ store → metadata│ │            │ │ history/diff   │
     └────────┬────────┘ └─────┬──────┘ └───────┬────────┘
              │                │                │
              └────────────────┼────────────────┘
                               │
                    ┌──────────▼──────────┐
                    │  iceberg-catalog    │
                    │  (Iceberg REST API) │
                    └──────────┬──────────┘
                               │
                    ┌──────────▼──────────┐
                    │   RustFS (S3)       │
                    │   Iceberg tables +  │
                    │   original files    │
                    └─────────────────────┘

Stack: Python 3.11+ | PyIceberg | DuckDB | iceberg-catalog | RustFS | PostgreSQL 16

Endpoints

• /catalog — DCAT JSON-LD view of the lakehouse, served by graph-server views (see the views design)
• /ui/ — the graph explorer, a 3D view of the data graph, served by graph-server (see docs/graph-explorer.md)

Local Development Setup

Prerequisites

• Python 3.11+
• uv package manager
• Docker and Docker Compose

1. Clone and install

git clone <repo-url>
cd data-platform
uv sync

2. Configure environment

cp .env.example .env
# Edit .env if you want to change default credentials

3. Start the infrastructure

docker compose --profile s3 up -d

This starts the graph stack (Postgres, graph-migrate, oxigraph, graph-server, materializer — see docs/architecture.md) plus:

• iceberg-catalog on port 8181 (Iceberg REST catalog, S3-backed)
• object-store (RustFS) on port 9000 (S3-compatible storage, console on 9001)

For GCS testing, use docker compose --profile gcs up -d instead, which starts fake-gcs-server — note there's no GCS-backed iceberg-catalog wired up yet, so this profile provides the emulator but not a working GCS catalog.

Cloudflare R2 backend

The r2 profile hasn't moved to the graph stack yet — it still runs Nessie directly, against a real Cloudflare R2 bucket (no local object store):

1. Create an R2 bucket in the Cloudflare dashboard.

2. Create an R2 API token (R2 > Manage R2 API Tokens) with Object Read & Write scoped to that bucket. Copy the Access Key ID and Secret Access Key.

3. Copy .env.r2.example values into your .env and fill in R2_ACCOUNT_ID, R2_BUCKET, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY.

4. If your Postgres volume already exists from an earlier s3/gcs run, create the R2 version-store database once (the init script in docker/init-databases.sql only runs on a fresh iceberg-db volume):

   docker compose up -d postgres
   docker compose exec postgres \
     psql -U iceberg -d iceberg \
     -c "CREATE DATABASE nessie_r2; GRANT ALL PRIVILEGES ON DATABASE nessie_r2 TO iceberg;"
   

   (A fresh stack — or docker compose down -v && docker compose --profile r2 up -d — creates it automatically. The reset destroys all local data.)

5. Start the R2 profile:

   docker compose --profile r2 up -d
   

   Nessie listens on port 19122. Point your client at http://localhost:19122.

R2-specific Nessie settings the profile sets for you:

• region=auto (R2 ignores region but the SDK requires a value).
• endpoint=https://<R2_ACCOUNT_ID>.r2.cloudflarestorage.com (account-style host).
• path-style-access=true so requests are <endpoint>/<bucket>/<key> — R2 accepts both styles but path-style avoids virtual-host DNS issues with some older clients.

The local PyIceberg/DuckDB client reads Iceberg data files straight from R2, so it needs the same bucket and credentials. It derives those from the same four R2_* vars automatically — warehouse (s3://<R2_BUCKET>/), endpoint, access keys, region=auto, and a default CATALOG_URI of http://localhost:19122. No separate S3_*/ICEBERG_WAREHOUSE settings are required; if you set them explicitly they take precedence.

4. Verify the stack

uv run scripts/smoke_test.py

5. Set up your environment

The sunstone CLI comes from sunstone-py. data-platform registers as a plugin and adds the data subcommand group:

# Add your local Docker stack as an environment
sunstone env add local --catalog-url http://localhost:19120 --s3-endpoint http://localhost:9000
sunstone env use local

6. Try it out

# Import a local CSV
sunstone data import path/to/data.csv

# Import from a URL
sunstone data import https://example.com/dataset.csv

# List datasets
sunstone data info

# View dataset metadata
sunstone data info namespace.table_name

# Query history
sunstone data history namespace.table_name

# Compare versions
sunstone data diff namespace.table_name @v1 @v2

Running Tests

# Unit tests (fast, no Docker required)
uv run pytest tests/unit/

# Integration tests (requires Docker stack running)
uv run pytest tests/integration/

# All tests
uv run pytest

# Specific test file with verbose output
uv run pytest tests/unit/test_query.py -v

CLI Commands

Command	Description
sunstone data import <source>	Import CSV/Excel from path or URL
sunstone data info [table]	View metadata for one or all datasets
sunstone data history <table>	Show snapshot history (git-log style)
sunstone data tag <table> <tag>	Tag a snapshot version
sunstone data untag <table> <tag>	Remove a tag
sunstone data branches	List branches and tags
sunstone data diff <table> <from> [to]	Compare two versions
sunstone data lineage <table>	Show provenance and lineage
sunstone data original <table>	Download the original imported source file

All commands support --branch for working on version branches and --json for machine-readable output. Run sunstone data <command> --help for full options.

Key Concepts

• Table references use git-like syntax: namespace.table@branch, @v1.0, @2026-01-25, @snapshot_id
• Deduplication: Imports are checksum-based (SHA256). Re-importing the same file is a no-op.
• Schema evolution: Safe type promotions (e.g. int32 to int64) are allowed automatically. Unsafe changes require --force-schema.
• Provenance: Every import captures source metadata, schema info, and W3C PROV-DM provenance as Iceberg table properties.
• Original files: Source files are stored content-addressable in S3 alongside the Iceberg data.

sunstone: URLs

Datasets in the lakehouse can be referenced as URLs anywhere sunstone-py accepts a data source:

import sunstone

df = sunstone.read("sunstone:namespace/table@v1.0")

Supports the full ref syntax: @branch, @tag, @2026-01-25, @snapshot_id.

Project Structure

src/sunstone_data/
├── plugin.py          # sunstone-py plugin (CLIProvider + URLHandler)
├── url.py             # sunstone: URL parser
├── config.py          # Settings from DataEnvironment or .env
├── catalog.py         # PyIceberg RestCatalog connection to the Iceberg REST catalog
├── query.py           # DuckDB QuerySession with time-travel support
├── lineage.py         # Lineage in Iceberg table properties
├── cli/               # Typer CLI commands + Rich formatters
├── import_/           # Pipeline: fetch → parse → store → metadata
├── metadata/          # Frozen dataclasses, MetadataBuilder, W3C PROV-DM
├── versioning/        # Version branches/tags, refs, schema evolution, diff
└── utils/             # Checksums, encoding detection, naming

Documentation

• Getting Started - Detailed setup instructions
• Architecture - System design and data flow
• CLI Reference - Complete command documentation
• Import Pipeline - How imports work
• Development Guide - Contributing and testing

Resetting the Stack

To wipe all data and start fresh:

docker compose --profile s3 --profile gcs down -v
docker compose --profile s3 up -d

License

MIT

Graph server authentication

graph-server requires a Keycloak bearer token on every route except /healthz//readyz. Access is role-based: the client roles in your token determine which environment you can use. See docs/specs/2026-07-05-graph-server-auth-design.md.

No hostnames or client ids ship in this package. Each environment's graph_url comes from config: Sunstone-managed machines get /etc/sunstone/data_platform.toml from the fleet-config repo, and other machines set GRAPH_URL or add an environment with sunstone env add. graph-server publishes the OIDC issuer and public client id at $GRAPH_URL/ui/config.json; the CLI reads them from there.

Humans (CLI, local agents)

sunstone env init          # cache discovery for configured environments
sunstone data login        # browser sign-in (authorization code + PKCE)

A browser window opens for Keycloak SSO; the refresh token renews silently until the SSO session expires.

Services (CI, automation)

Set SUNSTONE_CLIENT_ID and SUNSTONE_CLIENT_SECRET (or client_id and client_secret in the environment's data-platform section). The client then uses the OAuth2 client-credentials flow against the discovered issuer instead of a cached human login.