SuperTable

SuperTable — versioned data lake library for SQL analytics.

SuperTable stores structured data as immutable Parquet snapshots on object storage (S3, MinIO, Azure Blob, GCP Cloud Storage, or local disk), keeps metadata, locks, and audit state in Redis, and queries everything through DuckDB (embedded) or Spark SQL. It is a Python library — there is no separate server process.

Installation

pip install supertable                # core + local storage
pip install "supertable[s3]"          # AWS S3
pip install "supertable[minio]"       # MinIO
pip install "supertable[azure]"       # Azure Blob
pip install "supertable[gcp]"         # Google Cloud Storage
pip install "supertable[all]"         # everything

Requirements: Python 3.10+, a POSIX host (the durable monitoring spool uses fcntl locking), a reachable Redis 6+, and a configured storage backend (or local disk for development). See docs/02_configuration.md for environment variables.

Architecture

┌──────────────────────────────────────────────────┐
│                Python application                 │
│   (notebooks, ETL jobs, FastAPI handlers, etc.)   │
└──────────┬─────────────────────────┬──────────────┘
           │ DataWriter / DataReader │
           ▼                         ▼
   ┌───────────────┐        ┌────────────────────┐
   │  RedisCatalog │        │  StorageInterface  │
   │  metadata     │        │  Parquet files     │
   │  locks        │        │  S3 / MinIO /      │
   │  audit chain  │        │  Azure / GCP /     │
   └───────────────┘        │  Local             │
                            └────────────────────┘

Data is organised as Organization → SuperTable → SimpleTable. Each SimpleTable is a versioned, append-only collection of Parquet files backed by a snapshot linked list — every write produces a new immutable snapshot whose previous_snapshot points at the predecessor.

Quick example

from supertable import SuperTable, DataWriter, DataReader, engine

# Bootstrap catalogue + storage
SuperTable(super_name="example", organization="my-org")

# Write
dw = DataWriter(super_name="example", organization="my-org")
columns, rows, inserted, deleted = dw.write(
    role_name="superadmin",
    simple_name="facts",
    data=arrow_table,
    overwrite_columns=["day", "client"],
    lineage={"source_type": "manual", "source_id": "my-job"},
)

# Read
dr = DataReader(
    super_name="example",
    organization="my-org",
    query="SELECT day, sum(value) FROM facts GROUP BY day LIMIT 10",
)
df, status, message = dr.execute(role_name="superadmin", engine=engine.AUTO)

Demos

The package ships three runnable demos under supertable.demo:

# Numbered tutorial — runs the full lifecycle end-to-end.
supertable-demo-quickstart
# or
python -m supertable.demo.quickstart

# Synthetic webshop dataset.
supertable-demo-webshop-generate    # build ~1.2M rows on disk
supertable-demo-webshop-load        # load them into SuperTable
supertable-demo-webshop-topup       # continuous incremental refresh

# Medical-center finance demo (raw → staging → mart, fully synthetic).
supertable-demo-medcenter-run       # generate → load → transform → tests →
                                    # accounting export → idempotency proof

All demos are also runnable as module steps. Examples:

python -m supertable.demo.quickstart.s01_01_01_create_super_table
python -m supertable.demo.quickstart.s03_08_read_snapshot_history
python -m supertable.demo.webshop.generate

See supertable/demo/README.md for the full script index.

What's included

• Versioned tables with snapshot isolation, upsert (overwrite_columns), soft deletes (delete_only=True), schema evolution, and staleness filtering
• DuckDB and IslandDB query engines — embedded reads with adaptive selection
• Spark SQL via Thrift — for queries exceeding DuckDB memory limits
• RBAC — role types (superadmin, admin, writer, reader, meta) with row-level and column-level security enforced through view chains
• Audit logging — tamper-evident SHA-256 hash chain in Redis Streams with Parquet export
• Monitoring — MonitoringWriter pushes read/write/metric payloads to Redis lists; structured JSON logging with correlation IDs
• Ingestion — staging areas (Staging) and automated ingestion pipes (SuperPipe)
• Mirroring — optional Delta Lake / Iceberg / Parquet export after every write
• Snapshot history — every write chains to previous_snapshot, enabling point-in-time inspection without separate historical tables

Documentation

See docs/00_index.md for the full table of contents.

License

SuperTable is licensed under the Functional Source License, Version 1.1, ALv2 Future License (FSL-1.1-ALv2).

You may use, copy, modify, create derivative works, publicly perform, publicly display, and redistribute the software for any permitted purpose other than a Competing Use.

A Competing Use means making the software available to others in a commercial product or service that:

1. substitutes for SuperTable;
2. substitutes for another product or service offered by Kladna Soft Kft. using SuperTable; or
3. offers the same or substantially similar functionality as SuperTable.

Permitted purposes include internal use, non-commercial education, non-commercial research, and professional services provided to a licensee using the software in accordance with the license.

Each version of the software becomes available under the Apache License 2.0 on the second anniversary of the date that version is made available.

See LICENSE for the full license terms.

Copyright © 2024-2026 Kladna Soft Kft.