Dreamlake

A simple and flexible SDK for ML experiment tracking and data storage.

Features

• Three Usage Styles: Decorator, context manager, or direct instantiation
• Dual Operation Modes: Remote (API server) or local (filesystem)
• Auto-creation: Automatically creates namespace, workspace, and folder hierarchy
• Upsert Behavior: Updates existing episodes or creates new ones
• Simple API: Minimal configuration, maximum flexibility
• Time-Based Queries: MCAP-like API for querying track data by timestamp ranges
• Multi-Modal Sync: Timestamp inheritance for synchronizing pose, images, and sensor data

Installation

uv add dreamlake@0.15.0

pip install dreamlake==0.19.0

CLI (deprecated — use the standalone DreamLake CLI)

The Python CLI bundled in this package is deprecated, and this package no longer installs a dreamlake console script. Install the standalone DreamLake CLI instead — same commands, flags, and env vars, plus environment switching (dreamlake env use):

curl -fsSL https://dl.dreamlake.ai/install.sh | bash

upload, download, list, create, delete, update and vectorize have been removed from this package; they live in the standalone CLI. What is still here, reachable through python -m dreamlake.cli:

The append-local writers are not deprecated

artifact append-local and workflow append-local are the canonical DreamDB writers, and dreamlake-server spawns them as a subprocess — routes/workflows.ts for workflow apply, routes/artifacts.ts for artifact apply. Their stdout is a machine contract: exactly one JSON line, which the server parses as the last non-empty line of stdout. Errors go to stdout too, as JSON. Do not reformat either.

workflow append-local  →  {"version": N, "meta": {description, stageCount, nodeCount, edgeCount}}
artifact append-local  →  {"version": N}            # no "meta" key — the two are not symmetric
error (either)         →  {"error": "...", "message": "..."}   # exit 1

The wrapper bin

The server spawns ${WORKFLOWS_APPLY_BIN:-dreamlake} workflow append-local … and ${ARTIFACTS_APPLY_BIN:-…} artifact append-local …. The dreamlake on PATH is the standalone TS CLI, which has no DreamDB writer, and pip install dreamlake deliberately installs no bin of its own. So point both env vars at the wrapper shipped in this repo:

export WORKFLOWS_APPLY_BIN=/path/to/dreamlake-py/bin/dreamlake-append-local
export ARTIFACTS_APPLY_BIN=/path/to/dreamlake-py/bin/dreamlake-append-local

It is a one-line exec python3 -m dreamlake.cli "$@" and must stay one — anything the wrapper prints would corrupt the JSON contract above. It is not registered in [project.scripts], on purpose: the dreamlake name on PATH belongs to the standalone CLI.

Quick Start

Remote Mode (with API Server)

from dreamlake import Episode

with Episode(
    name="my-experiment",
    workspace="my-workspace",
    remote="https://cu3thurmv3.us-east-1.awsapprunner.com",
    api_key="your-jwt-token"
) as episode:
    print(f"Episode ID: {episode.id}")

Local Mode (Filesystem)

from dreamlake import Episode

with Episode(
    name="my-experiment",
    workspace="my-workspace",
    local_path=".dreamlake"
) as episode:
    pass  # Your code here

See examples/ for more complete examples.

Development Setup

Installing Dev Dependencies

To contribute to Dreamlake or run tests, install the development dependencies:

uv sync --extra dev

pip install -e ".[dev]"

This installs:

• pytest>=8.0.0 - Testing framework
• pytest-asyncio>=0.23.0 - Async test support
• sphinx>=7.2.0 - Documentation builder
• sphinx-rtd-theme>=2.0.0 - Read the Docs theme
• sphinx-autobuild>=2024.0.0 - Live preview for documentation
• myst-parser>=2.0.0 - Markdown support for Sphinx
• ruff>=0.3.0 - Linter and formatter
• mypy>=1.9.0 - Type checker

Running Tests

uv run pytest

pytest

Building Documentation

Documentation is built using Sphinx with Read the Docs theme.

uv run python -m sphinx -b html docs docs/_build/html

uv run sphinx-autobuild docs docs/_build/html

rm -rf docs/_build

The live preview command starts a local server and automatically rebuilds when files change.

Alternatively, you can use the Makefile from within the docs directory:

cd docs
make html          # Build HTML documentation
make clean         # Clean build files

For maintainers, to build and publish a new release: uv build && uv publish

Automated publish

.github/workflows/publish.yml publishes dreamlake to PyPI on a GitHub runner. It fires on one thing only — a push to the fixed branch publish-production — and there is no workflow_dispatch, no pull-request trigger and no approval gate. Pushing an exact SHA to publish-production publishes it. Treat that branch as the release button it is, and restrict who can push to it.

The version comes from the committed pyproject.toml; the workflow never writes it, and refuses anything that is not a stable x.y.z. Everything is built from github.sha, so a moving main cannot change what ships. Concurrency is a repository-wide lock with cancel-in-progress: false.

One-time account setup, which this repository cannot do for itself:

There is no API token and no secret: trusted publishing authenticates the job's OIDC token, which is why id-token: write is granted and why the runner must stay GitHub-hosted.

Partial publish, and why there is no skip

--skip-existing is deliberately not used. A version already on PyPI is a clear stop — scripts/check-pypi-version.py asks before anything is built — not a green run that published nothing. Failing to reach PyPI is also a stop.

A partial upload is reconcilable, but only by uploading the missing file; PyPI will not accept a second file under a name it already has. That is why the build job keeps dist/ as an artifact for three days: if the sdist landed and the wheel did not, upload the wheel from those retained bytes, not from a rebuild — a rebuild is not guaranteed to reproduce them. Re-running the whole workflow will not do this for you: it stops at the version pre-check. If the retained artifact is gone, yank the incomplete version and release the next one.

The build job holds no publishing credential, so the bytes exist before any credential is in play, and the workflow refuses a dist/ holding anything other than exactly this version's wheel and sdist.