BrickHouse

Run selected Databricks workflows locally, without a JVM.

BrickHouse gives you a persistent local workspace for Python jobs, SQL, Delta tables, and optional notebooks. Sparkless executes DataFrames and selected Spark SQL; delta-rs stores real Delta tables. Every execution surface uses the same catalog and table service, so a job's committed output is available to SQL and other sessions.

Use it to develop transformations, exercise selected Databricks client workflows, and design local integration tests. The base workflow needs no Java, Spark installation, Docker, Databricks account, or cloud compute.

0.1.0 alpha: compatibility is partial and versioned. Compute resource lifecycles and access modes are simulated; jobs and queries execute locally. Check the compatibility scope before adopting a workflow.

Quickstart · Capabilities · Compare · Benchmarks · Compatibility · Development · Documentation

Quickstart

Use Python 3.11 or 3.12. The qualified platforms are Ubuntu 22.04 x86_64 and macOS 26 arm64. The commands below install from the source checkout.

1. Install BrickHouse and the SDK

git clone https://github.com/eddiethedean/brickhouse.git
cd brickhouse
python -m venv .venv
source .venv/bin/activate
python -m pip install . databricks-sdk==0.153.0

The SDK is a client for the example; it is separate from BrickHouse's base runtime. To enable notebook jobs, also install the optional extra:

python -m pip install '.[notebooks]'

2. Start a local workspace

brickhouse init --workspace ./workspace
brickhouse serve --workspace ./workspace --host 127.0.0.1 --port 8000

The server accepts IPv4 loopback connections only. Use a disposable workspace and trusted code: the API has no inbound authentication, and workers run as your local user with access to your files and processes.

3. Run a job and query its Delta output

In another terminal, from the repository root:

source .venv/bin/activate
python examples/phase_0_1/run_orders.py --host http://127.0.0.1:8000

The example uses the real SDK to create a managed Delta table, upload and run a Python worker, append typed order rows, and query the committed data through SQL. Its JSON output includes resource IDs and this result value:

[["alice", "2", "15.75"]]

See the example walkthrough, SDK orchestration, and worker code. The run leaves its table and resource history available for inspection. Stop the server with Ctrl-C; restarting with the same workspace preserves metadata and committed tables.

Connect your own SDK code

With the server running, point the qualified SDK at the local endpoint:

from databricks.sdk import WorkspaceClient

client = WorkspaceClient(
    host="http://127.0.0.1:8000",
    token="brickhouse-local",
    auth_type="pat",
)

print([catalog.name for catalog in client.catalogs.list()])  # ['main'] in a fresh workspace

The token is a local compatibility value. Jobs are submitted through the SDK Jobs API; brickhouse run is reserved and currently reports an unavailable feature.

What you can do

Each row describes the selected 0.1.0 subset. Exact operations, limits, and evidence are recorded in the capability registry.

The runtime has one authority for catalog metadata and table commits. Sparkless handles the selected DataFrame and SQL semantics; native delta-rs owns table mutations; SQLite persists workspace metadata and run state. DuckDB is reserved for a future explicit execution option. A Sparkless failure is surfaced to the caller.

Supported pyspark, databricks, and delta import compatibility is scoped to job workers. It does not replace packages in the host environment. Worker table writes are limited to bounded appends on existing registered tables; embedded and SQL surfaces have their own declared operation sets.

Feature comparison

MiniLake and calvinchengx/databricks-emulator also expose local Databricks workspace APIs. This comparison uses source snapshots reviewed on 2026-10-22. ◐ Partial means a selected implemented subset; ⚙ Simulated means infrastructure metadata; 🗓 Planned marks future BrickHouse work.

Other columns summarize upstream documentation and source, rather than a complete independent conformance audit. Sources: MiniLake's README and configuration; the Go emulator's parity ledger and engine setup. BrickHouse's planned entries follow the roadmap.

Local benchmarks

Measured 2026-10-22 on an Apple M4 Pro, 12 logical CPUs, 24 GiB RAM, macOS 26.5.2 arm64, CPython 3.11.15, using native host processes. Two complete runs provide 10 fresh trials and 250 timed samples per warm workload per product. Latency cells show median / p95; RSS is a single-process median.

A follow-up SQL profile attributes 71% of BrickHouse's instrumented request time to SQLite calls and 17% to Sparkless SQL/Arrow boundaries. It retains raw traces, an uninstrumented baseline and measured profiling overhead; no optimization is included in these results.

A separate SQLite storage experiment measured 27.15 ms median with ordinary disk metadata, 11.64 ms with disk/shared cache, and 9.34 ms with shared in-memory metadata (2.91× faster than ordinary disk). Memory metadata is benchmark-only and temporary; Delta files stay on disk. These configurations do not change the comparison table above.

Startup includes workspace initialization and the first verified script round trip. Startup/RSS cover the core API process before SQL work; they exclude optional notebooks and the Go SQL engine/agent processes. The SQL case reads the same immutable Delta fixture and verifies all 100 aggregate result rows. MiniLake uses DuckDB; the Go profile attaches native Sail 0.7.1 and a pinned upstream agent. Table registration, engine startup, installation, and builds are outside warm query timings.

These are small local workflow measurements with warm filesystem caches. Engines, durability models, and API paths differ; the figures do not establish general Spark performance or complete Databricks parity. Both runs, every timed sample, source/artifact hashes, setup probes, and reproduction instructions are in the benchmark report and raw evidence.

Compatibility

The 0.1.0 registry contains 24 partial, 2 simulated, and 41 planned capabilities. No broad capability is marked fully supported. Unsupported operations raise explicit errors.

The qualified platform matrix is Ubuntu 22.04 x86_64 and macOS 26 arm64, each on Python 3.11 and 3.12, with separate base and notebook installations. Windows and other OS versions are outside the 0.1.0 qualification scope.

These versions identify tested client subsets. They do not imply that every method or protocol in those clients is implemented. SQL connector Thrift, Arrow-stream/LZ4 results, broad metadata methods, and remote authentication remain outside the qualified subset.

The base runtime pins Sparkless 4.15.2, deltalake 1.2.1, PyArrow 23.0.1, and SQLGlot 30.24.0. Sparkless 4.15.2 has a documented SQL COUNT output type difference from DataFrame count; see upstream issue #1723. Cluster access modes do not enforce production security policies. BrickHouse provides local trusted-code execution; distributed Spark, Databricks Runtime, and hostile-code isolation are outside this release.

Inspect your installation and the declared capability surface with:

brickhouse --version
brickhouse doctor --workspace ./workspace
brickhouse capabilities --format json

The frozen 0.1.0 candidate passed 1,368 installed-wheel offline cases and 1,832 full-source cases across the platform matrix, with zero required skips. Its bundle retains exact artifact hashes, interpreter and dependency versions, and CI reports.

Development

From the checkout and an activated virtual environment:

python -m pip install --require-hashes -r requirements/dev.txt
python scripts/validate_planning.py
python -m ruff check src tests scripts
python -m ruff format --check src tests scripts
python -m mypy
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest

Opt-in external-client cases skip in an unprovisioned developer environment. They are mandatory in CI, which provisions exact client versions and qualifies one built candidate across the full platform matrix. The offline gates enforce loopback-only networking and fail on required skips. See the release readiness guide and qualification workflow for the complete setup; Linux requires a non-root runner with passwordless sudo, unshare, and ip, and macOS uses sandbox-exec.

Ordinary tests and required CI use local sanitized fixtures. They never load the checkout's .env or discover real-workspace credentials. Occasional personal Databricks reference captures are separate and bounded by document 15.

Documentation

• Try a workflow: orders example.
• Check current behavior: project status and capability registry.
• Understand the design: architecture and serverless/classic compute contracts.
• Follow future work: release roadmap and task ledger.
• Review the release: changelog and 0.1.0 evidence bundle.
• Report a problem: GitHub issues.

License

Apache-2.0. See LICENSE and third-party notices.