lakebench-k8s 1.0.5

Deploy and benchmark lakehouse stacks on Kubernetes

Lakebench

CLI tool for deploying and benchmarking lakehouse architectures on Kubernetes.

Note: This package is published as lakebench-k8s on PyPI. Install with pip install lakebench-k8s. The CLI command is lakebench.

Choosing between Hive and Polaris, sizing Spark for 100 GB vs 10 TB, or comparing batch and continuous pipelines shouldn't require weeks of manual setup. Lakebench deploys a complete lakehouse stack from a single YAML file, generates realistic data at any scale, runs the pipeline, benchmarks query performance, and tears everything down --so you can focus on comparing architectures, not plumbing.

Installation

pip install lakebench-k8s

Or with pipx: pipx install lakebench-k8s

Pre-built binaries (no Python required) are available on GitHub Releases.

Prerequisites

• Python 3.10+
• kubectl and helm on PATH
• Kubernetes cluster (1.26+, minimum 8 CPU / 32 GB RAM for scale 1)
• S3-compatible object storage (FlashBlade, MinIO, AWS S3, etc.)
• Kubeflow Spark Operator 2.4.0+ (or set spark.operator.install: true in config to auto-install)
• Stackable Hive Operator if using a Hive recipe (the default). Not needed for Polaris recipes.

See Getting Started for detailed install instructions.

Quick Start

A quick-recipe selects the catalog + table format + query engine combination in one line (e.g. recipe: hive-iceberg-spark-trino). The scale factor controls data volume: 1 = ~10 GB, 10 = ~100 GB, 100 = ~1 TB. Every recipe default can be overridden individually -- see the Configuration Reference for advanced configuration options.

# 1. Generate config (interactive prompts for S3 details)
lakebench init --interactive

# 2. Validate config and cluster connectivity
lakebench validate lakebench.yaml

# 3. Deploy infrastructure
lakebench deploy lakebench.yaml

# 4. Generate test data
lakebench generate lakebench.yaml --wait

# 5. Run the pipeline + benchmark
lakebench run lakebench.yaml

# 6. View results
lakebench report

# 7. Tear down
lakebench destroy lakebench.yaml

Deploy and generate will prompt for confirmation. Add --yes to skip (e.g. lakebench deploy lakebench.yaml --yes).

Commands

Command	Description
lakebench init	Generate a starter configuration file
lakebench validate <config>	Validate config and test connectivity
lakebench info <config>	Show configuration summary
lakebench recommend	Recommend cluster sizing for a scale factor
lakebench deploy <config>	Deploy all infrastructure
lakebench generate <config>	Generate synthetic data to bronze bucket
lakebench run <config>	Execute the medallion pipeline with metrics
lakebench benchmark <config>	Run 8-query benchmark against the active engine
lakebench query <config>	Execute SQL queries against the active engine
lakebench status [config]	Show deployment status
lakebench logs <component> [config]	Stream logs from a component
lakebench report	Generate HTML benchmark report
lakebench destroy <config>	Tear down all resources

How It Works

Lakebench deploys a three-layer stack on Kubernetes:

1. Platform -- Kubernetes namespace, S3 secrets, PostgreSQL (metadata store)
2. Data architecture -- catalog (Hive or Polaris), table format (Iceberg), query engine (Trino, Spark Thrift, or DuckDB), all wired together via recipes
3. Observability -- optional Prometheus + Grafana stack for platform metrics

Once deployed, the pipeline runs three Spark jobs in sequence:

Raw Parquet (S3)  -->  Bronze (validate, deduplicate)
                  -->  Silver (normalize, enrich -- Iceberg table)
                  -->  Gold (aggregate -- Iceberg table)
                  -->  Benchmark (8 analytical queries via query engine)

The benchmark produces an HTML report with query latencies, throughput scores, and optional platform metrics (CPU, memory, S3 I/O per pod). See the Architecture doc for the full picture.

Component Versions

Component	Version
Apache Spark	3.5.4
Spark Operator	2.4.0 (Kubeflow)
Apache Iceberg	1.10.1
Hive Metastore	3.1.3 (Stackable 25.7.0)
Apache Polaris	1.3.0-incubating
Trino	479
DuckDB	bundled (Python 3.11)
PostgreSQL	17

All versions are configurable. See Supported Components for the full matrix of components, recipes, and override options.

Documentation

Full documentation is in the docs/ directory:

• Getting Started -- prerequisites, install, first deployment
• Configuration -- full YAML reference
• CLI Reference -- all commands and flags
• Recipes -- supported component combinations
• Supported Components -- versions, images, and recipe matrix
• Deployment -- deploy lifecycle and status checks
• Data Generation -- scale factors, parallelism, and monitoring
• Running Pipelines -- batch and streaming modes
• Scoring and Benchmarking -- pipeline scorecard and query engine benchmark
• Polaris Quick Start -- use Apache Polaris instead of Hive
• Architecture -- system design and component layers
• Troubleshooting -- common errors and fixes

License

Apache 2.0