minilake — Local Databricks API Emulator

MiniLake

Free, open-source local Databricks emulator for offline development and testing.

Real SQL & Spark execution · Unity Catalog hierarchy · Databricks SDK compatible · Terraform compatible · MIT licensed

GitHub release CI Release GHCR image License Python

Documentation · GitHub · Container Image (GHCR)

MiniLake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.

Quick Start

# Option 1: PyPI
pip install minilake
minilake --port 8000

# Option 2: GitHub Container Registry
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest

# Option 3: Clone and build
git clone https://github.com/dmux/minilake && cd minilake
docker compose up -d

# Verify (any option)
curl http://localhost:8000/_minilake/health

Then open http://localhost:8000/ui/ for the built-in SQL workspace — an Athena-style query editor with a data catalog, saved queries and query history (details).

No account, no API key, no sign-up — and the container image downloads nothing at runtime: DuckDB's delta extension and the Delta / Unity Catalog Spark jars are baked in at build time, so it works air-gapped (details).

Then point any Databricks client at it:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="vendas")

More in Getting Started.

Documentation
Getting Started Install, first catalog and query, internal endpoints
Web UI The built-in SQL workspace at /ui
Configuration Every environment variable, persistence, HTTPS/TLS
Databricks SDK Unity Catalog, warehouses, SQL and jobs from Python
Terraform & Asset Bundles The provider, and bundle deploy / bundle run
Spark & Delta Lake Real Delta files, real Spark jobs, spark.table() by name
MCP Server 67 tools for LLM agents — examples, tool reference, troubleshooting
Testing & development Running the suite, adding an API group
Releases & CI/CD How a tag becomes a published image
Feature status Endpoint-by-endpoint status and design rationale

Supported Services

Service Status Notes
Unity Catalog (catalogs, schemas, tables, volumes) ✅ Real Each catalog = its own DuckDB database (ATTACH), native catalog.schema.table addressing
EXTERNAL Delta Tables ✅ Real Real Delta files; INSERT/UPDATE/DELETE via a generated Spark job, reads via delta_scan()
SQL Statement Execution ✅ Real Real DuckDB; JSON_ARRAY/ARROW_STREAM/CSV, INLINE/EXTERNAL_LINKS; result manifest carries column types
SQL Warehouses ✅ Real Full CRUD + lifecycle
Query History ✅ Real w.query_history.list() over everything executed, failures included
Saved Queries ✅ Real w.queries.* CRUD with update_mask
Web UI ✅ Real Athena-style SQL workspace at /ui — see Web UI
Jobs ✅ Real Sibling Docker container execution (Spark) or subprocess fallback; real DAG scheduling (depends_on/run_if); sql_task.file
Workspace ✅ Real File-backed notebook/script storage; raw-bytes workspace-files sync powers databricks bundle deploy / bundle run
DBFS & Files API ✅ Real File-backed storage, chunked upload
Secrets ✅ Real Real CRUD; values only resolvable inside job env vars, never via direct API (matches real Databricks)
Clusters ✅ Real state machine CRUD + timed lifecycle transitions; no real Spark compute (by design)
Permissions ✅ Real CRUD Single-user "allow-all" default (by design — see Gaps)
Identity (SCIM) ✅ Static Fake current-user endpoint
Persistence (MINILAKE_PERSIST=1) ✅ Real JSON snapshot on shutdown, restored on startup
Unity Catalog protocol for Spark ✅ Real spark.table("cat.sch.tbl") resolves against minilake — see Spark & Delta Lake
JupyterLab + PySpark + Delta (optional) ✅ Real docker compose --profile notebook up
MCP Server (optional, MINILAKE_MCP=1) ✅ Real 67 tools + resources + prompts at /mcp — see MCP Server
Secrets ACLs, Repos/Git, multi-language notebooks, DBT/pipeline tasks, Model Registry, Vector Search, Dashboards 🚫 Not implemented Returns 501 NOT_IMPLEMENTED

Known Gaps

These are deliberate, not oversights — minilake targets one developer running it locally, not a shared or multi-tenant server:

  • No real authentication — any token is accepted; there's only ever one real user.
  • No access-control enforcement — the Permissions API is real CRUD but always allow-all, so a test that passes here says nothing about grants in a real workspace.
  • No real Spark compute for Clusters — state machine only; real compute happens through Jobs' sibling containers instead.
  • Single process, no HA — and DuckDB's single-writer model means concurrent load contends on locks.
  • Uneven test coverage — jobs.py, sql_statements.py and unity_catalog.py are covered mostly on happy paths, not edge cases.
  • Secrets ACLs not implemented — scope/secret CRUD is real, ACL endpoints aren't.

Contributing

See CONTRIBUTING.md for the project structure, how to add a new API group, and the PR checklist.

License

MIT — see LICENSE.

Project links

Data verified by PyPI on Aug 19, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.7.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for rfsales from gravatar.com rfsales

Credits

PyPI data
Data sourced directly from PyPI's database.

Author: Rafael Sales

GitHub Statistics

Data verified by PyPI on Aug 19, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.11

Provides Extra

mcp notebook

Tags

databricks delta-lake duckdb emulator mcp spark testing unity-catalog

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

minilake-1.7.1.tar.gz (7.2 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

minilake-1.7.1-py3-none-any.whl (6.8 MB view details)

Uploaded Python 3

File details

Details for the file minilake-1.7.1.tar.gz.

File metadata

  • Download URL: minilake-1.7.1.tar.gz
  • Upload date:
  • Size: 7.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for minilake-1.7.1.tar.gz
Algorithm Hash digest
SHA256 2234f7f00d858a1ec53b08b8f9c1a617e54670e8fbcc8cf5caa0c46bfe509e03
MD5 f1fae3ad3b7a21a09cbd074343103dd5
BLAKE2b-256 030e225b6f24de155ab4f773d820676ff6b33a0ad323275a45bd9568ee635ca1

See more details on using hashes here.

Provenance

The following attestation bundles were made for minilake-1.7.1.tar.gz:

Publisher: release.yml on dmux/minilake

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minilake-1.7.1-py3-none-any.whl.

File metadata

  • Download URL: minilake-1.7.1-py3-none-any.whl
  • Upload date:
  • Size: 6.8 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for minilake-1.7.1-py3-none-any.whl
Algorithm Hash digest
SHA256 ba053f7dcb9a0740e1832f9cd61d49a1ae0a108bca61b685eeb9d49a78f407ba
MD5 dd5c7bcd5947741c249d328a8dc6613a
BLAKE2b-256 b301f06c0cd481674e5216ada7291ad3d77a6f0e924e5841cf4470455a922791

See more details on using hashes here.

Provenance

The following attestation bundles were made for minilake-1.7.1-py3-none-any.whl:

Publisher: release.yml on dmux/minilake

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.7.4

2 files

1.7.3

2 files

1.7.2

2 files

This release

1.7.1 This release

2 files

1.7.0

1 file

1.6.0

2 files

1.5.1

2 files

1.5.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page