Sunstone Data Platform

A data lakehouse for importing, versioning, and querying external datasets with full provenance tracking. Built on Apache Iceberg, an Iceberg REST catalog, a versioned RDF graph in Postgres, RustFS, and DuckDB.

Data scientists import CSV/Excel files from local paths or URLs into Iceberg tables, query them with DuckDB using git-like version references, and publish dataset metadata and W3C PROV-DM provenance into the graph. The graph is the system of record for metadata, branches and tags.

Architecture

              ┌─────────────────────────────────┐
              │ sunstone_data (Python)          │
              │ sunstone data CLI, sunstone:    │
              │ URLs, import pipeline, DuckDB   │
              └───────┬─────────────────┬───────┘
         Iceberg REST │                 │ GSP, refs, SPARQL
                      ▼                 ▼
          ┌─────────────────┐   ┌─────────────────┐
          │ iceberg-catalog │──▶│ graph-server    │
          │ :8181           │   │ :8080           │
          └────────┬────────┘   └────────┬────────┘
                   ▼                     ▼
          ┌─────────────────┐   ┌─────────────────┐
          │ RustFS (S3)     │   │ Postgres graph  │
          │ Iceberg tables, │   │ (SoR), copied   │
          │ original files  │   │ to Oxigraph     │
          └─────────────────┘   └─────────────────┘

On every table commit, iceberg-catalog also writes the table's descriptor into the graph. See docs/architecture.md for the full data flow.

Stack: Python 3.11+ | PyIceberg | DuckDB | iceberg-catalog | graph-server | Oxigraph | RustFS | PostgreSQL 16

Endpoints

• /catalog — DCAT JSON-LD view of the lakehouse, served by graph-server views (see the views design)
• /ui/ — the graph explorer, a 3D view of the data graph, served by graph-server (see docs/graph-explorer.md)

Local Development Setup

Prerequisites

• Python 3.11+
• uv package manager
• Docker and Docker Compose

1. Clone and install

git clone <repo-url>
cd data-platform
uv sync

2. Configure environment

cp .env.example .env
# Edit .env if you want to change default credentials

3. Start the infrastructure

docker compose --profile s3 up -d

This starts the graph stack (Postgres, graph-migrate, oxigraph, graph-server, materializer — see docs/architecture.md) plus:

• iceberg-catalog on port 8181 (Iceberg REST catalog, S3-backed)
• object-store (RustFS) on port 9000 (S3-compatible storage, console on 9001)

For GCS testing, use docker compose --profile gcs up -d instead, which starts fake-gcs-server — note there's no GCS-backed iceberg-catalog wired up yet, so this profile provides the emulator but not a working GCS catalog.

Cloudflare R2 backend

The r2 profile hasn't moved to the graph stack yet — it still runs Nessie directly, against a real Cloudflare R2 bucket (no local object store):

1. Create an R2 bucket in the Cloudflare dashboard.

2. Create an R2 API token (R2 > Manage R2 API Tokens) with Object Read & Write scoped to that bucket. Copy the Access Key ID and Secret Access Key.

3. Copy .env.r2.example values into your .env and fill in R2_ACCOUNT_ID, R2_BUCKET, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY.

4. If your Postgres volume already exists from an earlier s3/gcs run, create the R2 version-store database once (the init script in docker/init-databases.sql only runs on a fresh iceberg-db volume):

   docker compose up -d postgres
   docker compose exec postgres \
     psql -U iceberg -d iceberg \
     -c "CREATE DATABASE nessie_r2; GRANT ALL PRIVILEGES ON DATABASE nessie_r2 TO iceberg;"
   

   (A fresh stack — or docker compose down -v && docker compose --profile r2 up -d — creates it automatically. The reset destroys all local data.)

5. Start the R2 profile:

   docker compose --profile r2 up -d
   

   Nessie listens on port 19122. Point your client at http://localhost:19122.

R2-specific Nessie settings the profile sets for you:

• region=auto (R2 ignores region but the SDK requires a value).
• endpoint=https://<R2_ACCOUNT_ID>.r2.cloudflarestorage.com (account-style host).
• path-style-access=true so requests are <endpoint>/<bucket>/<key> — R2 accepts both styles but path-style avoids virtual-host DNS issues with some older clients.

The local PyIceberg/DuckDB client reads Iceberg data files straight from R2, so it needs the same bucket and credentials. It derives those from the same four R2_* vars automatically — warehouse (s3://<R2_BUCKET>/), endpoint, access keys, region=auto, and a default CATALOG_URI of http://localhost:19122. No separate S3_*/ICEBERG_WAREHOUSE settings are required; if you set them explicitly they take precedence.

4. Verify the stack

uv run scripts/smoke_test.py

5. Set up your environment

The sunstone CLI comes from sunstone-py. data-platform registers as a plugin and adds the data subcommand group. You configure one value per environment, the base IRI (base_iri). graph-server is served at the base, the Iceberg catalog under {base_iri}iceberg/, and datasets and organizations are named under it. The client discovers sign-in settings from graph-server.

# Add your local Docker stack as an environment
sunstone env add local data-platform.base_iri=https://data.example.org/
sunstone env use local

base_iri must equal the iceberg-catalog service's BASE_IRI. In the local Docker stack, graph-server (port 8080) and the catalog (port 8181) listen on different ports, so also set the service overrides GRAPH_URL=http://localhost:8080 and CATALOG_URI=http://localhost:8181 (see .env.example). They never affect IRIs.

6. Try it out

# Import a local CSV
sunstone data import path/to/data.csv

# Import from a URL
sunstone data import https://example.com/dataset.csv

# List namespaces and tables
sunstone data ls

# View dataset metadata from the graph
sunstone data show namespace.table_name

# Snapshot history
sunstone data history namespace.table_name

# Publish the current snapshot as a named version
sunstone data publish namespace.table_name --version v1.0

Running Tests

# Unit tests (fast, no Docker required)
uv run pytest tests/unit/

# Integration tests (run inside Docker automatically; needs Docker)
uv run pytest tests/integration/

# All tests
uv run pytest

# Specific test file with verbose output
uv run pytest tests/unit/test_query.py -v

CLI Commands

Command	Description
sunstone data login, logout, whoami	Sign in to graph-server (Keycloak)
sunstone data ls [namespace]	List namespaces and tables
sunstone data show <table>	Dataset metadata from the graph (table, Turtle or JSON-LD)
sunstone data info [table]	Iceberg table properties for one or all datasets
sunstone data import <source>	Import CSV/Excel from a path or URL
sunstone data publish <table>	Publish dataset metadata to the graph; --version names the current snapshot
sunstone data history <table>	Iceberg snapshot history
sunstone data lineage <table>	Provenance from the published PROV facts
sunstone data sparql <query>	Run a SPARQL query through graph-server
sunstone data branches, branch create|delete	List, create and delete graph branches
sunstone data tags, tag create|delete	List, create and delete graph tags

Every command takes --json for machine-readable output. import, publish, show, history and tag create take --branch. Run sunstone data <command> --help for full options.

Key Concepts

• Table references use the form namespace.table@ref. A ref is checked in this order:

  1. a branch name
  2. a snapshot id (13 or more digits)
  3. a date or timestamp (UTC if it has no timezone), which selects the last snapshot at or before that time
  4. a named version created with sunstone data publish --version

  Snapshot ids, timestamps and named versions resolve against main. Graph tags (sunstone data tag create) are not valid refs.

• Deduplication: Imports are checksum-based (SHA-256). Re-importing the same file is a no-op.

• Schema evolution: Safe type promotions (e.g. int32 to int64) are allowed automatically. Unsafe changes require --force-schema.

• Provenance: Every import records source metadata and schema info as Iceberg table properties, and asserts W3C PROV-DM provenance into the dataset's graph.

• Original files: Source files are stored content-addressed in object storage next to the Iceberg data.

sunstone: URLs

Datasets in the lakehouse can be referenced as URLs anywhere sunstone-py accepts a data source:

import sunstone

df = sunstone.read("sunstone:namespace/table@v1.0")

The @ref part follows the same rules as table references above. Here v1.0 is a named version.

Project Structure

src/sunstone_data/
├── plugin.py          # sunstone-py plugin (CLIProvider + URLHandler)
├── url.py             # sunstone: URL parser
├── resolve.py         # Resolves a @ref to a branch or snapshot
├── config.py          # Settings from DataEnvironment or .env
├── discovery.py       # Sign-in settings from graph-server /ui/config.json
├── auth/              # Keycloak login and token handling
├── graph/             # graph-server client and RDF serialization
├── publish.py         # Publishes dataset metadata to the graph
├── catalog.py         # PyIceberg RestCatalog connection to the Iceberg REST catalog
├── query.py           # DuckDB QuerySession with time-travel support
├── cli/               # Typer CLI commands
├── import_/           # Pipeline: fetch → parse → store → metadata
├── metadata/          # Frozen dataclasses, MetadataBuilder, W3C PROV-DM
├── versioning/        # Schema evolution checks
└── utils/             # Checksums, encoding detection, naming

Documentation

• docs/README.md - Index of all documentation
• docs/architecture.md - System overview and data flow
• docs/specs2/ - Current design, one file per subsystem. The Python client and CLI are in 05-python-client.md.

Resetting the Stack

To wipe all data and start fresh:

docker compose --profile s3 --profile gcs down -v
docker compose --profile s3 up -d

License

MIT

Graph server authentication

graph-server requires a Keycloak bearer token on every route except /healthz//readyz. Access is role-based: the client roles in your token determine which environment you can use. See docs/specs/2026-07-05-graph-server-auth-design.md.

No hostnames or client ids ship in this package. Each environment's base_iri comes from config: Sunstone-managed machines get /etc/sunstone/data_platform.toml from the fleet-config repo, and other machines set BASE_IRI or add an environment with sunstone env add. graph-server publishes the OIDC issuer and public client id at {base_iri}ui/config.json; the CLI reads them from there.

Humans (CLI, local agents)

sunstone env init          # cache discovery for configured environments
sunstone data login        # browser sign-in (authorization code + PKCE)

A browser window opens for Keycloak SSO; the refresh token renews silently until the SSO session expires.

Services (CI, automation)

Set SUNSTONE_CLIENT_ID and SUNSTONE_CLIENT_SECRET (or client_id and client_secret in the environment's data-platform section). The client then uses the OAuth2 client-credentials flow against the discovered issuer instead of a cached human login.