minilake — Local Databricks API Emulator

MiniLake

Free, open-source local Databricks emulator for offline development and testing.

Real SQL & Spark execution · Unity Catalog hierarchy · Databricks SDK compatible · Terraform compatible · MIT licensed

Python

Website · Documentation · GitHub · Container Image (GHCR)

MiniLake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.

Quick Start

# Option 1: PyPI
pip install minilake
minilake --port 8000

# Option 2: GitHub Container Registry
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest

# Option 3: Clone and build
git clone https://github.com/dmux/minilake && cd minilake
docker compose up -d

# Verify (any option)
curl http://localhost:8000/_minilake/health

Then open http://localhost:8000/ui/ for the built-in SQL workspace — an Athena-style query editor with a data catalog, saved queries and query history (details).

No account, no API key, no sign-up — and the container image downloads nothing at runtime: DuckDB's delta extension and the Delta / Unity Catalog Spark jars are baked in at build time, so it works air-gapped (details).

Then point any Databricks client at it:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="vendas")

Documentation


Getting Started	Install, first catalog and query, internal endpoints
Web UI	The built-in SQL workspace at `/ui`
Configuration	Every environment variable, persistence, HTTPS/TLS
Databricks SDK	Unity Catalog, warehouses, SQL and jobs from Python
Terraform & Asset Bundles	The provider, and `bundle deploy` / `bundle run`
Spark & Delta Lake	Real Delta files, real Spark jobs, `spark.table()` by name
MCP Server	67 tools for LLM agents — examples, tool reference, troubleshooting
Testing & development	Running the suite, adding an API group
Releases & CI/CD	How a tag becomes a published image
Feature status	Endpoint-by-endpoint status and design rationale

Supported Services

Service	Status	Notes
Unity Catalog (catalogs, schemas, tables, volumes)	✅ Real	Each catalog = its own DuckDB database (`ATTACH`), native `catalog.schema.table` addressing
EXTERNAL Delta Tables	✅ Real	Real Delta files; `INSERT`/`UPDATE`/`DELETE` via a generated Spark job, reads via `delta_scan()`
SQL Statement Execution	✅ Real	Real DuckDB; `JSON_ARRAY`/`ARROW_STREAM`/`CSV`, `INLINE`/`EXTERNAL_LINKS`; result manifest carries column types
SQL Warehouses	✅ Real	Full CRUD + lifecycle
Query History	✅ Real	`w.query_history.list()` over everything executed, failures included
Saved Queries	✅ Real	`w.queries.*` CRUD with `update_mask`
Web UI	✅ Real	Athena-style SQL workspace at `/ui` — see Web UI
Jobs	✅ Real	Sibling Docker container execution (Spark) or subprocess fallback; real DAG scheduling (`depends_on`/`run_if`); `sql_task.file`
Workspace	✅ Real	File-backed notebook/script storage; raw-bytes `workspace-files` sync powers `databricks bundle deploy` / `bundle run`
DBFS & Files API	✅ Real	File-backed storage, chunked upload
Secrets	✅ Real	Real CRUD; values only resolvable inside job env vars, never via direct API (matches real Databricks)
Clusters	✅ Real state machine	CRUD + timed lifecycle transitions; no real Spark compute (by design)
Permissions	✅ Real CRUD	Single-user "allow-all" default (by design — see Gaps)
Identity (SCIM)	✅ Static	Fake current-user endpoint
Persistence (`MINILAKE_PERSIST=1`)	✅ Real	JSON snapshot on shutdown, restored on startup
Unity Catalog protocol for Spark	✅ Real	`spark.table("cat.sch.tbl")` resolves against minilake — see Spark & Delta Lake
JupyterLab + PySpark + Delta (optional)	✅ Real	`docker compose --profile notebook up`
MCP Server (optional, `MINILAKE_MCP=1`)	✅ Real	67 tools + resources + prompts at `/mcp` — see MCP Server
Secrets ACLs, Repos/Git, multi-language notebooks, DBT/pipeline tasks, Model Registry, Vector Search, Dashboards	🚫 Not implemented	Returns `501 NOT_IMPLEMENTED`

Known Gaps

These are deliberate, not oversights — minilake targets one developer running it locally, not a shared or multi-tenant server:

No real authentication — any token is accepted; there's only ever one real user.
No access-control enforcement — the Permissions API is real CRUD but always allow-all, so a test that passes here says nothing about grants in a real workspace.
No real Spark compute for Clusters — state machine only; real compute happens through Jobs' sibling containers instead.
Single process, no HA — and DuckDB's single-writer model means concurrent load contends on locks.
Uneven test coverage — jobs.py, sql_statements.py and unity_catalog.py are covered mostly on happy paths, not edge cases.
Secrets ACLs not implemented — scope/secret CRUD is real, ACL endpoints aren't.

Contributing

See CONTRIBUTING.md for the project structure, how to add a new API group, and the PR checklist.

License

MIT — see LICENSE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

minilake-1.7.4.tar.gz (7.6 MB view details)

Uploaded Aug 31, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

minilake-1.7.4-py3-none-any.whl (6.9 MB view details)

Uploaded Aug 31, 2026 Python 3

File details

Details for the file minilake-1.7.4.tar.gz.

File metadata

Download URL: minilake-1.7.4.tar.gz
Upload date: Aug 31, 2026
Size: 7.6 MB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for minilake-1.7.4.tar.gz
Algorithm	Hash digest
SHA256	`426a661f59e1b6ce97330ea5e3c6b4b9a2d56499263c65860e50f2e012c35987`
MD5	`3288229c473c39f461d29fbb688ae04f`
BLAKE2b-256	`8924ddec58f20ca7c699d78d1fd3b4e8813cb040e8d31ce41ff151799110305c`

See more details on using hashes here.

Provenance

The following attestation bundles were made for minilake-1.7.4.tar.gz:

Publisher: release.yml on dmux/minilake

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: minilake-1.7.4.tar.gz
- Subject digest: 426a661f59e1b6ce97330ea5e3c6b4b9a2d56499263c65860e50f2e012c35987
- Sigstore transparency entry: 2669458265
- Sigstore integration time: Aug 31, 2026
Source repository:
- Permalink: dmux/minilake@4b3d280514aaf40aa0b2e7f82bfd01514f5e6844
- Branch / Tag: refs/tags/v1.7.4
- Owner: https://github.com/dmux
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@4b3d280514aaf40aa0b2e7f82bfd01514f5e6844
- Trigger Event: push