Lakebench

Run a real data workload through a composable lakehouse architecture on Kubernetes, and get evidence of how it behaved.

Deploy a complete lakehouse stack from a single YAML, generate a workload corpus, run its pipeline and queries at any scale, and get a scorecard that records exactly what produced it. Two runs are compared only when they returned the same workload results.

Why Lakebench?

• Compare stacks. Swap catalogs (Hive, Polaris), query engines (Trino, Spark Thrift, DuckDB), and table formats (Iceberg, Delta) -- same data, same queries, different architecture. lakebench compare checks that both sides returned the same results before it shows any performance difference, and says NOT COMPARABLE when they did not.
• Test at scale. Run the same workload at 10 GB, 100 GB, and 1 TB to find where throughput plateaus or resources saturate on your hardware.
• Measure freshness. Continuous mode keeps data arriving through the pipeline and benchmarks query performance under ongoing ingest load.

Workloads

Lakebench ships two workload schemas. The stack is the same either way; the schema flag routes datagen and the pipeline scripts.

• Customer 360 (default). Retail interactions from ~8 channels flow through bronze / silver / gold into an executive dashboard and the 8-query analytical benchmark. workload.schema: customer360.
• Financial crime (AML). pacs.008 wire messages from a frozen generator (datagen-v2-rs-0.3) flow through bronze / silver / gold. Nine detection rules, including a fuzzy sanctions and PEP screen against a synthetic, dated watchlist, are scored for recall and precision against planted typologies (structuring, round-tripping, layering, dormant reactivation, high-risk corridors and others) by spark/scripts/score_financial.py, which lakebench run invokes inline. A pre-registered reference model (the fidelity gate) and a band leakage report ship separately as lakebench financial reference-score; they diagnose whether a rule could read high recall from a label proxy, and they are not run by lakebench run. Iceberg recipes only. workload.schema: financial. See AML Scoring for what precision and recall measure here vs what an AML ops team cares about.

Quick Start

pip install lakebench-k8s

Pre-built binaries (no Python required) are available on GitHub Releases.

pip install lakebench-k8s
lakebench init                                 # quick setup (4 questions), writes lakebench.yaml
lakebench run lakebench.yaml --generate --yes  # deploy + generate + pipeline + benchmark
lakebench results lakebench.yaml               # view scorecard
lakebench destroy lakebench.yaml               # tear down what this deployment owns

--yes lets run deploy the namespace and components when they do not exist yet; without it run stops and asks you to run lakebench deploy first. The Spark Operator is shared cluster infrastructure: a cluster admin installs it once with lakebench admin install-spark-operator (see Getting Started).

Minimum config -- 4 lines:

# lakebench.yaml
endpoint: http://s3.example.com:80
access_key: YOUR_KEY
secret_key: YOUR_SECRET
scale: 10                          # 1 = ~10 GB, 10 = ~100 GB, 100 = ~1 TB

Name is auto-generated. Buckets default to <name>-bronze, <name>-silver and <name>-gold, so they are unique on stores where bucket names are global (FlashBlade, AWS S3). Recipe defaults to hive-iceberg-spark-trino. Override anything with flat fields or nested YAML:

# lakebench.yaml (with overrides)
name: flashblade-polaris
recipe: polaris-iceberg-spark-trino
endpoint: http://10.0.1.50:80
access_key: ${S3_ACCESS_KEY}       # env var substitution
secret_key: ${S3_SECRET_KEY}
scale: 50
mode: batch
spark_image: apache/spark:4.1.1-python3
architecture:
  catalog:
    polaris:
      client_secret: ${LAKEBENCH_POLARIS_CLIENT_SECRET}   # required for Polaris

Eleven recipes are available -- see Recipes for the full list. Support is judged per workload x recipe x mode: supported means a live run on the release tree validated it end to end, unverified means it is valid but not release-validated, and unsupported is refused before a run (rules). Every run records its state in metrics.json. As computed by this release:

• unsupported, refused at config load: AML (financial) on hive-delta-spark-none, hive-delta-spark-thrift, hive-delta-spark-trino. The financial (AML) workload supports table_format iceberg, not delta. Its stage scripts and table DDL are written for iceberg only, so this combination would not run the workload it names. Set architecture.table_format.type to iceberg (for example recipe: polaris-iceberg-spark-trino).
• Any catalog, table format and query engine combination that is not a recipe above is refused at config load for every workload.
• AML (financial) continuous: AML continuous runs detection rules W2, W3, W4, W17 each tick and records W1, W5, W6, W7, W8 as not run. Its results depend on when detection ran relative to arrival, so no end-of-run result check is recorded.

Compare two configurations side-by-side:

lakebench deploy config-hive.yaml && lakebench deploy config-polaris.yaml
lakebench compare config-hive.yaml config-polaris.yaml --generate

compare runs each config through the pipeline and benchmark in turn, then destroys that deployment unless you pass --keep. Deploy both configs first; --generate fills each bronze bucket before its run. The two configs need different names and bucket names, or the first destroy empties the second run's data.

For all recipes, see examples/ or run lakebench init --advanced for the full interactive wizard.

What You Get

After lakebench run completes, the terminal prints a scorecard (the numbers below are illustrative, not a measurement):

 ─ Pipeline Complete ──────────────────────────────
  bronze-verify         142.0 s
  silver-build          891.0 s
  gold-finalize         234.0 s
  benchmark              87.0 s

  Scores
    Time to Value:        1354.0 s
    Throughput:           0.782 GB/s
    Efficiency:           3.41 GB/core-hr
    Scale:                100.0% verified
    QpH:                  2847.3

  Full report: lakebench report
 ──────────────────────────────────────────────────

Each run writes lakebench-output/runs/run-<id>/report.html; lakebench report prints the run's summary and points at it (--render writes a fresh copy). The report shows per-query latencies, bottleneck analysis, and optional platform metrics (CPU, memory, S3 I/O per pod). Every run's metrics.json carries an experiment block: workload and generator version, seed, recipe and component versions, scale, mode, the maintenance that actually ran, the Lakebench-imposed limits that applied and whether they bound, the support state, and a fingerprint of each query's result.

Upgrading from 1.5: most earlier numbers are not comparable with 1.6 (for example, no Iceberg snapshot expiry or Delta VACUUM ever ran before 1.6.0). See the CHANGELOG.

How It Works

                    ┌──────────────────────────────────┐
                    │         lakebench.yaml           │
                    └────────────┬─────────────────────┘
                                 │
                    ┌────────────▼─────────────────────┐
                    │   deploy (Kubernetes namespace,   │
                    │   S3 secrets, PostgreSQL, catalog, │
                    │   query engine, observability)     │
                    └────────────┬─────────────────────┘
                                 │
     Raw Parquet ──► Bronze (validate) ──► Silver (enrich) ──► Gold (aggregate)
         S3              Spark                Spark               Spark
                                                                    │
                                                        ┌───────────▼──────────┐
                                                        │  8-query benchmark   │
                                                        │  (Trino / DuckDB /   │
                                                        │   Spark Thrift)      │
                                                        └──────────────────────┘

Prerequisites

• kubectl and helm on PATH
• Kubernetes 1.26+ with capacity for the Spark job profiles. Customer360 or AML batch at scale 1 requests a peak of 36 cores and 512 GB RAM (driven by silver-build), and a single executor pod needs 60 GB on one node. AML continuous at scale 1-10 requests 118 cores and 980 GB, because its three stream jobs run at once; AML batch at scale 100 peaks at 76 cores for the pipeline, but its data generation ran about 350 cores (44 pods x 8 cores, measured in run-20260925-104703-c02890). The request figures come from compute_peak_requirements(). See Getting Started for the full table. lakebench run fails fast if the cluster is too small.
• S3-compatible object storage. FlashBlade and Garage are validated; others are expected to work. Run lakebench config storage to check yours. See Storage Backends.
• Kubeflow Spark Operator 2.5.1+, installed once per cluster with lakebench admin install-spark-operator. deploy adds its namespace to the operator's watch list itself, under the cluster lock; do not edit spark.jobNamespaces by hand.
• Stackable Hive Operator for Hive recipes (not needed for Polaris)

Commands

See CLI Reference for flags and options.

Component Versions

All versions are overridable in the YAML config. See Supported Components.

Documentation

• Getting Started -- prerequisites, install, first run
• Configuration -- full YAML reference
• Recipes -- catalog + format + engine combinations
• Compatibility Matrix -- Spark, Iceberg, and Delta version support
• Running Pipelines -- batch and continuous modes
• Benchmarking -- scorecard and query benchmark
• AML Scoring -- financial-crime / AML workload: rule recall, reference detector, leakage gate
• Architecture -- system design
• Storage Backends -- validated S3 backends and conformance checks
• Troubleshooting -- common errors and fixes

License

Apache 2.0