Cereyan

Cereyan is a minimal, local-first orchestrator for Python data pipelines. A Rust core (SQLite store, state machine, scheduler, HTTP server) sits behind a thin layer of Python decorators, and the whole thing ships as one wheel with no runtime dependencies. Decorate functions as flows and tasks, run them as plain scripts, and every run is recorded locally; cereyan serve adds live monitoring, schedules, retries, backfills, rules, and a built-in MCP server for agents on top of the same store.

pip install cereyan

# pipeline.py
from datetime import date
from cereyan import flow, task, get_run_logger

@task
def extract(day: date) -> list[int]:
    get_run_logger().info("extracting %s", day)
    return [1, 2, 3]

@task
def load(rows: list[int]) -> int:
    return sum(rows)

@flow(run_name="etl-{day}")
def etl(day: date = date(2026, 9, 6)) -> int:
    return load(extract(day))

if __name__ == "__main__":
    print(etl())

python pipeline.py                                    # records a run in ~/.cereyan/db.sqlite
cereyan run pipeline.py:etl --param day=2026-01-02    # same, with parameters and a summary
cereyan serve .                                       # API, UI, scheduler, engines at http://127.0.0.1:4200

_{The dashboard after cereyan serve — take the full tour}

• Offline first. A script records runs into a local SQLite file; nothing else needs to run.
• One process to serve. The API, the web UI, the scheduler, a warm pool of engine processes, rules, and MCP, in one cereyan serve.
• Data-pipeline semantics. Targets make reruns idempotent, backfills cover date ranges or listed values, resources are named semaphores that can be keyed by parameter, flows chain and fan in by key, unique keys stop duplicate runs, and rules react to events or to their absence.
• Durable runs. Every completed task is checkpointed, so a crash rerun or a retry from failure resumes after the last finished task; a run can sleep, wait for an event or a target, or snooze without holding an engine; and a task keeps notes that survive its retries.

Screenshots

Every page below is described in the tour; click a shot to jump to it.

Runs — scope by project and group, filter, act in bulk	Run detail — logs, tasks and retries
Flows — groups, schedules, skips, dependencies, history	Timeline — the task graph of a run
Events — what happened, as it happens	Rules — react to events, or to their absence

The UI follows your system theme; the documentation shows the dark variants too.

Documentation

The site is at https://sercanatalik.github.io/cereyan/ (built from docs/ with just docs):

• Quickstart: from install to a scheduled, retried, backfilled pipeline in ten minutes.
• Concepts: the model behind flows, runs, states, schedules, targets, resources, backfills, dependencies, events, rules, artifacts, and variables.
• Guides: one goal per page, from retries to running the server as a service and using cereyan with an AI agent.
• Reference: the Python API, CLI, HTTP API, MCP tools, events, states, and configuration.
• Design and limitations: what cereyan does not do, and why.
• For agents: llms.txt and llms-full.txt.
• Changelog.

Status

Production ready, and published to PyPI. The pieces described here work and are tested on macOS, Linux and Windows, and the interfaces — names, signatures, defaults, routes, event payloads and the database schema — are settled. Windows lacks the Unix socket and engine niceness, and cannot interrupt a flow blocked in most I/O; Install lists what that changes. Read the changelog before upgrading. Issues and questions are welcome.

Development

Rust stable, Python 3.11 or newer with uv, Node 22, and just. just ui builds the UI, just dev builds the extension in place, just test and just lint run everything, just demo serves the examples. Details, the repository layout, and the release checklist are in the contributing guide. Design decisions are on the Design pages, and changes not yet released are under Unreleased in CHANGELOG.md.

Credits

Cereyan owes its shape to two projects that came first.

Luigi contributed the idea the data-pipeline semantics rest on: a task declares the target it produces, and work is skipped when that target already exists. Target and LocalTarget keep Luigi's names because they are Luigi's concept, and idempotent reruns and backfills follow from it.

Prefect contributed the authoring model — flows and tasks as decorated functions, runs carrying explicit states, a server watching them — and the look of the UI. Six UI components were adapted from Prefect's, rewritten in React from the Vue originals

Neither is a dependency, and cereyan deliberately does far less than either: one machine, one process, one wheel, no remote workers and no database to run. If you need what they do, use them. Migrating from Prefect or Luigi says what carries over and what does not.

License

MIT, see LICENSE.