minilake — Local Databricks API Emulator

MiniLake

Free, open-source local Databricks emulator for offline development and testing.

Real SQL & Spark execution · Unity Catalog hierarchy · Databricks SDK compatible · Terraform compatible · MIT licensed

GitHub release CI Release GHCR image License Python

Documentation · GitHub · Container Image (GHCR)

MiniLake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.

Quick Start

# Option 1: PyPI
pip install minilake
minilake --port 8000

# Option 2: GitHub Container Registry
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest

# Option 3: Clone and build
git clone https://github.com/dmux/minilake && cd minilake
docker compose up -d

# Verify (any option)
curl http://localhost:8000/_minilake/health

No account, no API key, no sign-up — and the container image downloads nothing at runtime: DuckDB's delta extension and the Delta / Unity Catalog Spark jars are baked in at build time, so it works air-gapped (details).

Then point any Databricks client at it:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="vendas")

More in Getting Started.

Documentation
Getting Started Install, first catalog and query, internal endpoints
Configuration Every environment variable, persistence, HTTPS/TLS
Databricks SDK Unity Catalog, warehouses, SQL and jobs from Python
Terraform & Asset Bundles The provider, and bundle deploy / bundle run
Spark & Delta Lake Real Delta files, real Spark jobs, spark.table() by name
MCP Server 67 tools for LLM agents — examples, tool reference, troubleshooting
Testing & development Running the suite, adding an API group
Releases & CI/CD How a tag becomes a published image
Feature status Endpoint-by-endpoint status and design rationale

Supported Services

Service Status Notes
Unity Catalog (catalogs, schemas, tables, volumes) ✅ Real Each catalog = its own DuckDB database (ATTACH), native catalog.schema.table addressing
EXTERNAL Delta Tables ✅ Real Real Delta files; INSERT/UPDATE/DELETE via a generated Spark job, reads via delta_scan()
SQL Statement Execution ✅ Real Real DuckDB; JSON_ARRAY/ARROW_STREAM/CSV, INLINE/EXTERNAL_LINKS
SQL Warehouses ✅ Real Full CRUD + lifecycle
Jobs ✅ Real Sibling Docker container execution (Spark) or subprocess fallback; real DAG scheduling (depends_on/run_if); sql_task.file
Workspace ✅ Real File-backed notebook/script storage; raw-bytes workspace-files sync powers databricks bundle deploy / bundle run
DBFS & Files API ✅ Real File-backed storage, chunked upload
Secrets ✅ Real Real CRUD; values only resolvable inside job env vars, never via direct API (matches real Databricks)
Clusters ✅ Real state machine CRUD + timed lifecycle transitions; no real Spark compute (by design)
Permissions ✅ Real CRUD Single-user "allow-all" default (by design — see Gaps)
Identity (SCIM) ✅ Static Fake current-user endpoint
Persistence (MINILAKE_PERSIST=1) ✅ Real JSON snapshot on shutdown, restored on startup
Unity Catalog protocol for Spark ✅ Real spark.table("cat.sch.tbl") resolves against minilake — see Spark & Delta Lake
JupyterLab + PySpark + Delta (optional) ✅ Real docker compose --profile notebook up
MCP Server (optional, MINILAKE_MCP=1) ✅ Real 67 tools + resources + prompts at /mcp — see MCP Server
Secrets ACLs, Repos/Git, multi-language notebooks, DBT/pipeline tasks, Model Registry, Vector Search, Dashboards 🚫 Not implemented Returns 501 NOT_IMPLEMENTED

Known Gaps

These are deliberate, not oversights — minilake targets one developer running it locally, not a shared or multi-tenant server:

  • No real authentication — any token is accepted; there's only ever one real user.
  • No access-control enforcement — the Permissions API is real CRUD but always allow-all, so a test that passes here says nothing about grants in a real workspace.
  • No real Spark compute for Clusters — state machine only; real compute happens through Jobs' sibling containers instead.
  • Single process, no HA — and DuckDB's single-writer model means concurrent load contends on locks.
  • Uneven test coverage — jobs.py, sql_statements.py and unity_catalog.py are covered mostly on happy paths, not edge cases.
  • Secrets ACLs not implemented — scope/secret CRUD is real, ACL endpoints aren't.

Contributing

See CONTRIBUTING.md for the project structure, how to add a new API group, and the PR checklist.

License

MIT — see LICENSE.

Project links

Data verified by PyPI on Aug 8, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.7.4)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for rfsales from gravatar.com rfsales

Credits

PyPI data
Data sourced directly from PyPI's database.

Author: Rafael Sales

GitHub Statistics

Data verified by PyPI on Aug 8, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.11

Provides Extra

mcp

Tags

databricks delta-lake duckdb emulator mcp spark testing unity-catalog

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

minilake-1.6.0.tar.gz (451.0 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

minilake-1.6.0-py3-none-any.whl (117.3 kB view details)

Uploaded Python 3

File details

Details for the file minilake-1.6.0.tar.gz.

File metadata

  • Download URL: minilake-1.6.0.tar.gz
  • Upload date:
  • Size: 451.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for minilake-1.6.0.tar.gz
Algorithm Hash digest
SHA256 55749ec40cdaa53ee4cf3e824c629fa28b33666f414370183dfdcb0c16bc1dd0
MD5 643a7d84f5e5a8a356ed775cc3fef29f
BLAKE2b-256 3ced35317177f7476d37b33d5d4d182fa7c7d663b7a47cac145986a2b947be8b

See more details on using hashes here.

Provenance

The following attestation bundles were made for minilake-1.6.0.tar.gz:

Publisher: release.yml on dmux/minilake

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file minilake-1.6.0-py3-none-any.whl.

File metadata

  • Download URL: minilake-1.6.0-py3-none-any.whl
  • Upload date:
  • Size: 117.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for minilake-1.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 a0819bcdac6e478cff77cadccba0c7563c55ad6b2c1b90c081a576bdc48ae6c9
MD5 32d11121407f0db54cdeff358302e9f3
BLAKE2b-256 0c90ba52d246c520f99405f819756e50667752fa3de85323a024d537ba7771fa

See more details on using hashes here.

Provenance

The following attestation bundles were made for minilake-1.6.0-py3-none-any.whl:

Publisher: release.yml on dmux/minilake

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.7.4

2 files

1.7.3

2 files

1.7.2

2 files

1.7.1

2 files

1.7.0

1 file

This release

1.6.0 This release

2 files

1.5.1

2 files

1.5.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page