MiniLake

Free, open-source local Databricks emulator for offline development and testing.

Real SQL & Spark execution · Unity Catalog hierarchy · Databricks SDK compatible · Terraform compatible · MIT licensed

Documentation · GitHub · Container Image (GHCR)

MiniLake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.

Quick Start

# Option 1: PyPI
pip install minilake
minilake --port 8000

# Option 2: GitHub Container Registry
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest

# Option 3: Clone and build
git clone https://github.com/dmux/minilake && cd minilake
docker compose up -d

# Verify (any option)
curl http://localhost:8000/_minilake/health

Then open http://localhost:8000/ui/ for the built-in SQL workspace — an Athena-style query editor with a data catalog, saved queries and query history (details).

No account, no API key, no sign-up — and the container image downloads nothing at runtime: DuckDB's delta extension and the Delta / Unity Catalog Spark jars are baked in at build time, so it works air-gapped (details).

Then point any Databricks client at it:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="vendas")

More in Getting Started.

Documentation

Supported Services

Known Gaps

These are deliberate, not oversights — minilake targets one developer running it locally, not a shared or multi-tenant server:

• No real authentication — any token is accepted; there's only ever one real user.
• No access-control enforcement — the Permissions API is real CRUD but always allow-all, so a test that passes here says nothing about grants in a real workspace.
• No real Spark compute for Clusters — state machine only; real compute happens through Jobs' sibling containers instead.
• Single process, no HA — and DuckDB's single-writer model means concurrent load contends on locks.
• Uneven test coverage — jobs.py, sql_statements.py and unity_catalog.py are covered mostly on happy paths, not edge cases.
• Secrets ACLs not implemented — scope/secret CRUD is real, ACL endpoints aren't.

Contributing

See CONTRIBUTING.md for the project structure, how to add a new API group, and the PR checklist.

License

MIT — see LICENSE.