Lakebench

A/B testing for lakehouse architectures on Kubernetes.

Deploy a complete lakehouse stack from a single YAML, run a medallion pipeline at any scale, and get a scorecard you can compare across configurations.

Why Lakebench?

• Compare stacks. Swap catalogs (Hive, Polaris), query engines (Trino, Spark Thrift, DuckDB), and table formats -- same data, same queries, different architecture. Side-by-side scorecard comparison.
• Test at scale. Run the same workload at 10 GB, 100 GB, and 1 TB to find where throughput plateaus or resources saturate on your hardware.
• Measure freshness. Sustained mode streams data through the pipeline and benchmarks query performance under sustained ingest load.

Quick Start

pip install lakebench-k8s

Pre-built binaries (no Python required) are available on GitHub Releases.

pip install lakebench-k8s
lakebench init                           # quick setup (4 questions)
lakebench run lakebench.yaml --generate  # deploy + generate + pipeline + benchmark
lakebench results                        # view scorecard
lakebench destroy lakebench.yaml         # tear down everything

Minimum config -- 4 lines:

# lakebench.yaml
endpoint: http://s3.example.com:80
access_key: YOUR_KEY
secret_key: YOUR_SECRET
scale: 10                          # 1 = ~10 GB, 10 = ~100 GB, 100 = ~1 TB

Name is auto-generated. Recipe defaults to hive-iceberg-spark-trino. Override anything with flat fields or nested YAML:

# lakebench.yaml (with overrides)
name: flashblade-polaris
recipe: polaris-iceberg-spark-trino
endpoint: http://10.21.227.93:80
access_key: ${S3_ACCESS_KEY}       # env var substitution
secret_key: ${S3_SECRET_KEY}
scale: 50
mode: batch
spark_image: apache/spark:4.1.1-python3

Eleven recipes are available -- see Recipes for the full list.

Compare two configurations side-by-side:

lakebench compare config-hive.yaml config-polaris.yaml

For all recipes, see examples/ or run lakebench init --advanced for the full interactive wizard.

What You Get

After lakebench run completes, the terminal prints a scorecard:

 ─ Pipeline Complete ──────────────────────────────
  bronze-verify         142.0 s
  silver-build          891.0 s
  gold-finalize         234.0 s
  benchmark              87.0 s

  Scores
    Time to Value:        1354.0 s
    Throughput:           0.782 GB/s
    Efficiency:           3.41 GB/core-hr
    Scale:                100.0% verified
    QpH:                  2847.3

  Full report: lakebench report
 ──────────────────────────────────────────────────

lakebench report generates an HTML report with per-query latencies, bottleneck analysis, and optional platform metrics (CPU, memory, S3 I/O per pod).

How It Works

                    ┌──────────────────────────────────┐
                    │         lakebench.yaml           │
                    └────────────┬─────────────────────┘
                                 │
                    ┌────────────▼─────────────────────┐
                    │   deploy (Kubernetes namespace,   │
                    │   S3 secrets, PostgreSQL, catalog, │
                    │   query engine, observability)     │
                    └────────────┬─────────────────────┘
                                 │
     Raw Parquet ──► Bronze (validate) ──► Silver (enrich) ──► Gold (aggregate)
         S3              Spark                Spark               Spark
                                                                    │
                                                        ┌───────────▼──────────┐
                                                        │  8-query benchmark   │
                                                        │  (Trino / DuckDB /   │
                                                        │   Spark Thrift)      │
                                                        └──────────────────────┘

Prerequisites

• kubectl and helm on PATH
• Kubernetes 1.26+ with capacity for the Spark job profiles. Scale 1 requests a peak of 36 cores and 512 GB RAM (driven by silver-build), and a single executor pod needs 60 GB on one node. See Getting Started for the full table. Lakebench fails fast if the cluster is too small.
• S3-compatible object storage. FlashBlade and Garage are validated; others are expected to work. Run lakebench config storage to check yours. See Storage Backends.
• Kubeflow Spark Operator 2.5.1+ (or set spark.operator.install: true)
• Stackable Hive Operator for Hive recipes (not needed for Polaris)

Commands

See CLI Reference for flags and options.

Component Versions

All versions are overridable in the YAML config. See Supported Components.

Documentation

• Getting Started -- prerequisites, install, first run
• Configuration -- full YAML reference
• Recipes -- catalog + format + engine combinations
• Compatibility Matrix -- Spark, Iceberg, and Delta version support
• Running Pipelines -- batch and sustained modes
• Benchmarking -- scorecard and query benchmark
• Architecture -- system design
• Storage Backends -- validated S3 backends and conformance checks
• Troubleshooting -- common errors and fixes

License

Apache 2.0