Elementary Runtime

Early development.

Elementary Runtime is a small service that customers run in their own environment. It uses local warehouse credentials to execute SQL prepared by Elementary Cloud, so Cloud never holds credentials that can read customer business tables.

Warehouse drivers will be optional extras. Installing the base package for task models will not pull in every warehouse driver or the full runtime.

A license has not been chosen yet.

Security-critical code

Local safety controls live under src/elementary_runtime/execution/. That tree is the audited surface: SQL validation, query execution with limits applied while rows are fetched and encoded, and redaction of SQL, row values, credentials, and unrestricted warehouse errors from logs and telemetry.

Repository layout

src/elementary_runtime/
  protocol/      Task and result wire formats
  config/        Versioned YAML configuration and local limits
  common/        Shared helpers used by protocol and config
  warehouses/    Warehouse clients (drivers behind extras)
  execution/     Audited local safety controls
    sql/         SQL validation
    worker/      Worker pool, query execution, limit enforcement
    redaction/   Error, log, and telemetry scrubbing
  coordinator/   Register, task streams, heartbeat, submit, retries, shutdown
  cli/           Process entrypoint: commands, instance id, logging

Configuration

version: 1
cloud:
  token_file: /run/secrets/elementary_runtime_token
connections:
  - warehouse_connection_id: connection-id
    type: snowflake
    account: ab12345.eu-central-1
    user: elementary_runtime
    password_file: /run/secrets/snowflake_password
    role: ELEMENTARY_RUNTIME
    warehouse: ELEMENTARY_WH
    database: ANALYTICS

Running

elementary-runtime loads the YAML configuration and runs until it is stopped.

elementary-runtime run --config /etc/elementary-runtime/config.yml
elementary-runtime validate --config /etc/elementary-runtime/config.yml
elementary-runtime validate --check-connections --config /etc/elementary-runtime/config.yml
elementary-runtime --version

A container can start with no arguments. Each flag falls back to an environment variable:

• ELEMENTARY_RUNTIME_CONFIG: configuration file, default /etc/elementary-runtime/config.yml
• ELEMENTARY_RUNTIME_INSTANCE_ID: explicit instance id, unset by default
• ELEMENTARY_RUNTIME_LOG_LEVEL: root log level, default INFO
• ELEMENTARY_RUNTIME_POD_UID: Helm pod UID from the Downward API
• ELEMENTARY_RUNTIME_POD_NAME: pod name, used as the display name when the config has none

The instance id is the first match:

1. --instance-id or ELEMENTARY_RUNTIME_INSTANCE_ID, used as given.
2. helm:<pod uid> when ELEMENTARY_RUNTIME_POD_UID is set. instance.id_prefix is not applied, because a pod UID is already unique.
3. docker:<hostname> when /.dockerenv or /run/.containerenv exists, otherwise host:<hostname>. instance.id_prefix is prepended when set, for example eu-prod:docker:runtime-1.

Docker assigns a new hostname when a container is recreated. Set one explicitly (docker run --hostname or Compose hostname) so the instance id stays stable, and keep it unique among replicas.

Exit codes:

• 0: stopped gracefully
• 1: runtime failure, such as Cloud rejecting the token
• 2: usage or configuration error
• 3: drain timed out with work still in flight

The first SIGTERM or SIGINT drains in-flight work for 30 seconds. A second signal exits immediately. Set the container stop grace period above that drain (docker stop -t, or Compose stop_grace_period), for example 45 seconds.

Docker images

Published images live at ghcr.io/elementary-data/elementary-runtime and are built for linux/amd64 and linux/arm64.

A pre-release such as 0.1.0rc3 is tagged with its version only. It does not move latest, core, or the bare driver tags.

docker run --rm \
  --hostname runtime-1 \
  --stop-timeout 45 \
  -v /path/to/config.yml:/etc/elementary-runtime/config.yml:ro \
  ghcr.io/elementary-data/elementary-runtime:{version}

The image runs elementary-runtime run. Mount the configuration where the process expects it, set --hostname so the instance id stays stable, and keep the stop timeout above the 30 second drain.

Development

Requires uv and Python 3.10–3.14. Local development uses Python 3.12.

After a clone or a new Git worktree, install the environment and hooks:

./script/setup.sh

Then:

uv run pytest
uv run mypy src tests
uvx prek run --all-files

The package version lives in pyproject.toml. At runtime, elementary_runtime.__version__ is read from the installed distribution metadata.

Releasing

Run the Prepare release workflow from the Actions tab. Choose a bump (patch, minor, major, rc, stable, patch-rc, minor-rc, or major-rc) or set an explicit version. minor-rc turns 0.1.0 into 0.2.0rc1. rc advances an existing candidate. It opens a pull request that updates pyproject.toml and uv.lock.

Merging that pull request publishes the package to PyPI and the container images, then tags v{version} and creates a GitHub Release. Pre-releases do not move floating image tags.

If publishing fails before that tag exists, re-run the Release package workflow from main. A version that already has a tag is skipped.