Lakebench

Run a real data workload through a composable lakehouse architecture on Kubernetes, and get evidence of how it behaved.

Deploy a complete lakehouse stack from a single YAML, generate a workload corpus, run its pipeline and queries at any scale, and get a scorecard that records exactly what produced it, including a fingerprint of each query's result, so a reader can check that two runs returned the same answers before comparing their numbers.

Why Lakebench?

• Compare stacks. Swap catalogs (Hive, Polaris), query engines (Trino, Spark Thrift, DuckDB), and table formats (Iceberg, Delta) -- same data, same queries, different architecture. Each run's report records a result fingerprint per query, so you can see that two runs returned the same answers before you compare their performance.
• Test at scale. Run the same workload at 10 GB, 100 GB, and 1 TB to find where throughput plateaus or resources saturate on your hardware.
• Measure freshness. Continuous mode keeps data arriving through the pipeline and benchmarks query performance under ongoing ingest load.

Workloads

Lakebench ships two workload schemas. The stack is the same either way; the schema flag routes datagen and the pipeline scripts.

• Customer 360 (default). Retail interactions from ~8 channels flow through bronze / silver / gold into an executive dashboard and the 8-query analytical benchmark. workload.schema: customer360.
• Financial crime (AML). pacs.008 wire messages from a frozen generator (datagen-v2-rs-0.3) flow through bronze / silver / gold. Nine detection rules, including a fuzzy sanctions and PEP screen against a synthetic, dated watchlist, are scored for recall and precision against planted typologies (structuring, round-tripping, layering, dormant reactivation, high-risk corridors and others) by spark/scripts/score_financial.py, which lakebench run invokes inline. A pre-registered reference model (the fidelity gate) and a band leakage report ship separately as lakebench financial reference-score; they diagnose whether a rule could read high recall from a label proxy, and they are not run by lakebench run. Iceberg recipes only. workload.schema: financial. See AML Scoring for what precision and recall measure here vs what an AML ops team cares about.

Quick Start

pip install lakebench-k8s

Pre-built binaries (no Python required) are available on GitHub Releases.

lakebench init                                 # writes lakebench.yaml (--workload financial for AML): set its S3 endpoint, export the two key variables it names
lakebench admin install --component all lakebench.yaml  # once per cluster (cluster admin)
lakebench plan lakebench.yaml                  # what the run needs and whether the cluster has room; changes nothing
lakebench run lakebench.yaml --generate --yes  # deploy + generate + pipeline + benchmark
lakebench report lakebench.yaml                # view scorecard
lakebench destroy lakebench.yaml               # tear down what this deployment owns (asks first; --yes to skip)

--yes lets run deploy the namespace and components when they do not exist yet; without it run stops and asks you to run lakebench deploy first. The Spark Operator, the Stackable operators (Hive recipes), the scratch StorageClass and the observability stack are shared cluster infrastructure: deploy only checks them, and a cluster admin installs them once with lakebench admin install --component all lakebench.yaml (see Getting Started).

Minimum config:

# lakebench.yaml, as `lakebench init` writes it
name: lb-alice-7f3c                # names the namespace and the buckets
recipe: polaris-iceberg-spark-trino
workload:
  schema: customer360
  datagen:
    scale: 1                       # 1 = ~10 GB, 10 = ~100 GB, 100 = ~1 TB
platform:
  storage:
    s3:
      endpoint: http://s3.example.com:80
      access_key: "${LAKEBENCH_S3_ACCESS_KEY}"
      secret_key: "${LAKEBENCH_S3_SECRET_KEY}"

The name is required by every command that changes data (deploy, generate, run and the rest); a config without one is refused with the name to add. Buckets default to <name>-bronze, <name>-silver and <name>-gold, so they are unique on stores where bucket names are global (FlashBlade, AWS S3). A config with no recipe: (or recipe: default) still resolves to hive-iceberg-spark-trino with a deprecation note; v1.8 requires the recipe. A component written against the recipe (say catalog.type: hive under a Polaris recipe) is refused at load. Override anything in the nested YAML:

# lakebench.yaml (with overrides)
name: flashblade-polaris
recipe: polaris-iceberg-spark-trino
images:
  spark: apache/spark:4.1.1-python3
platform:
  storage:
    s3:
      endpoint: http://10.0.1.50:80
      access_key: "${S3_ACCESS_KEY}"   # env var substitution; quote it
      secret_key: "${S3_SECRET_KEY}"
architecture:
  pipeline:
    mode: batch
workload:
  datagen:
    scale: 50

The flat top-level spellings (endpoint:, scale: and the others in Configuration) still load, with a deprecation note naming the nested key.

Polaris needs no client secret in the config: deploy generates one per deployment and stores it in the namespace.

Eleven recipes are available -- see Recipes for the full list. Support is judged per workload x recipe x mode: supported means a live run on the release tree validated it end to end, unverified means it is valid but not release-validated, and unsupported is refused before a run (rules). Every run records its state in metrics.json. As computed by this release:

• [1] unverified: not in this release's validation matrix.
• [2] unverified: in this release's validation matrix; no validation run is listed yet.
• unsupported, refused at config load: AML (financial) on hive-delta-spark-none, hive-delta-spark-thrift, hive-delta-spark-trino. The financial (AML) workload supports table_format iceberg, not delta. Its stage scripts and table DDL are written for iceberg only, so this combination would not run the workload it names. Use an iceberg recipe (for example recipe: polaris-iceberg-spark-trino), or with no recipe set architecture.table_format.type to iceberg.
• Any catalog, table format and query engine combination that is not a recipe above is refused at config load for every workload.
• A supported cell names the Spark minor and table format version its validation runs used; the same cell on any other Spark minor or format version is unverified. Spark 3.5 is unverified and gets no v1.7 features.
• AML (financial) continuous: AML continuous runs detection rules W2, W3, W4, W17 each tick and records W1, W5, W6, W7, W8 as not run. Its results depend on when detection ran relative to arrival, so no end-of-run result check is recorded.

Compare two configurations by running each and reading the two reports side by side:

lakebench deploy config-hive.yaml && lakebench run config-hive.yaml
lakebench deploy config-polaris.yaml && lakebench run config-polaris.yaml
lakebench report config-hive.yaml
lakebench report config-polaris.yaml

report prints each run's scorecard and the path of the HTML report the run wrote. Each HTML report's Experiment section states the corpus, datagen image, components, maintenance, the stages and rules that ran, a result fingerprint per benchmark query and, for AML batch runs, the alert set. Compare performance only when the corpus, the result fingerprints and the alert set match: a difference means the two stacks returned different answers. A number bounded by a Lakebench cap is labelled as such.

For all recipes, see examples/, lakebench config recipes, or lakebench init --recipe <name>.

What You Get

Each lakebench run prints a scorecard summary (per-stage seconds and the run's scores) and writes lakebench-output/runs/run-<id>/report.html with per-query latencies, bottleneck analysis and, when observability is enabled, pod-level CPU, memory and S3 I/O. lakebench report prints the summary and points at the HTML; --render writes a fresh copy. Every run's metrics.json carries an experiment block: workload and generator version, seed, recipe and component versions, scale, mode, the maintenance that actually ran, the Lakebench-imposed limits that applied and whether they bound, the support state, and a fingerprint of each query's result.

Upgrading from 1.6: every change that can break a 1.6 config, command line or script, with what to do, is in UPGRADING-1.7.md; redeploy each deployment once after upgrading. Numbers from 1.5 and earlier are not comparable with 1.7: no Iceberg snapshot expiry or Delta VACUUM ran before 1.6, and the v1.7 datagen image changes the corpus. See the CHANGELOG.

How It Works

                    ┌──────────────────────────────────┐
                    │         lakebench.yaml           │
                    └────────────┬─────────────────────┘
                                 │
                    ┌────────────▼─────────────────────┐
                    │   deploy (Kubernetes namespace,   │
                    │   S3 secrets, PostgreSQL, catalog, │
                    │   query engine, observability)     │
                    └────────────┬─────────────────────┘
                                 │
     Raw Parquet ──► Bronze (validate) ──► Silver (enrich) ──► Gold (aggregate)
         S3              Spark                Spark               Spark
                                                                    │
                                                        ┌───────────▼──────────┐
                                                        │  8-query benchmark   │
                                                        │  (Trino / DuckDB /   │
                                                        │   Spark Thrift)      │
                                                        └──────────────────────┘

Prerequisites

• kubectl and helm on PATH
• Kubernetes 1.26+ with room for the minimum below: what must fit at once, for the default recipe (hive-iceberg-spark-trino), from the sizing function behind lakebench config show, config recommend and the run capacity preflight. Batch datagen pods that do not fit wait their turn, so the batch minimum is the Spark peak plus the always-on pods. See Getting Started for what each figure is built from. lakebench run fails fast if the cluster is too small.

• S3-compatible object storage. FlashBlade and Garage are validated; others are expected to work. Run lakebench config storage to check yours. See Storage Backends.
• Kubeflow Spark Operator 2.5.1+, installed once per cluster with lakebench admin install --component spark-operator. deploy adds its namespace to the operator's watch list itself, under the cluster lock; do not edit spark.jobNamespaces by hand.
• Stackable Hive Operator for Hive recipes (not needed for Polaris)

Commands

See CLI Reference for flags and options.

Component Versions

All versions are overridable in the YAML config. See Supported Components.

Documentation

• Getting Started -- prerequisites, install, first run
• Configuration -- full YAML reference
• Recipes -- catalog + format + engine combinations
• Compatibility Matrix -- Spark, Iceberg, and Delta version support
• Running Pipelines -- batch and continuous modes
• Benchmarking -- scorecard and query benchmark
• AML Scoring -- financial-crime / AML workload: rule recall, reference detector, leakage gate
• Architecture -- system design
• Storage Backends -- validated S3 backends and conformance checks
• Troubleshooting -- common errors and fixes

License

Apache 2.0