mini-atlas-simulator

This is the Mini Atlas simulator library (mini_atlas_simulator). It is the only installable simulator package. The user’s Python pipeline owns analytics, formulas, rule execution, and emitted results. The library owns the UTC clock, catalog identity, metric/table declarations, atomic publication, and a read-only ResultReader.

M1d adds installable-wheel examples: a user pipeline can publish a live run and a second process can inspect it through ResultReader with no API, frontend, Docker, Node, or repository checkout required at runtime.

Cadences are fixed elapsed UTC durations. 1d / P1D is 24 hours, not a local calendar day across daylight-saving changes.

Install (local development)

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
python -m pip install -e ".[dev]"

Install (local wheel)

The package is not published to PyPI in this milestone. Build and install a local wheel:

python -m pip wheel . -w dist --no-deps
python -m venv /tmp/sim-venv
source /tmp/sim-venv/bin/activate
python -m pip install dist/mini_atlas_simulator-*.whl

Runtime dependencies are the storage stack (PyArrow for Parquet publication, DuckDB for table pages). Installing mini-atlas-simulator does not pull Dash, Flask, Flask-RESTX, FastAPI, Waitress, NiceGUI, React, Node, Docker, or the graph packages.

Flask/API, React, Node, and Docker are not required to log or read results. Dash is not required to log or read results. A later Dash application will open the same store directly through ResultReader.

Storage

• SQLite (catalog.sqlite) stores catalog/scalar metadata and metric points: projects, runs, declarations, committed steps, and coverage scalars.
• Parquet stores logged tabular results for a step.
• ResultReader provides read-only access to both through the registered catalog. Callers do not query SQLite or Parquet themselves.

Clock

from mini_atlas_simulator import SimulationClock

clock = SimulationClock(
    start="2025-01-01T00:00:00Z",
    stop="2025-01-03T00:00:00Z",
    cadence="1d",
    lookback="1d",
)

for bounds in clock:
    # bounds.step_n starts at 1
    # bounds.previous_as_of = start + (n-1)*cadence
    # bounds.as_of = start + n*cadence  (final as_of == stop)
    # bounds.window_start = as_of - lookback
    ...

Start and stop must be timezone-aware UTC (Z or +00:00). Naive timestamps and non-UTC offsets are rejected. (stop - start) must be a positive integer multiple of cadence.

Catalog

from mini_atlas_simulator import open_catalog

with open_catalog("./results") as catalog:
    assert catalog.result_format_version == 1

A fresh {store}/catalog.sqlite is created in WAL mode with result_format_version = 1. Reopening a compatible catalog succeeds. An unknown or unsupported format version fails closed.

Writer

from mini_atlas_simulator import create_project, start_run

create_project(store="./results", name="sample-project", description="Sample simulation")

with start_run(
    store="./results",
    project="sample-project",
    name="baseline-30d",
    schedule=clock,
    description="Baseline configuration for the weekly study",
) as run:
    run.declare_metric(
        "slice_node_count",
        label="Slice nodes",
        unit="count",
        grain="per_step",
        direction="none",
    )
    run.declare_table("subject_scores", role="scores", subject_column="subject_id")
    for bounds in clock:
        with run.step(bounds) as step:
            step.log_metrics({"definition_id": "slice_node_count", "value": 7})
            step.log_table("subject_scores", [{"subject_id": "A-99", "score": 0.91}])

Project and run creation do not allocate a store-wide commit_id. A step is visible in the catalog only after the short publication transaction commits. Metrics are declared by user Python; there are no built-in evaluation formulas.

Optional start_run(..., description=...) stores a run description in a side table. Omit the argument to leave the description absent (None). An empty or whitespace-only string is invalid and is not stored as missing. The established runs table is unchanged. ResultReader.get_run and run-list rows return description as a string or None.

list_projects includes run_count. list_runs includes description, committed_steps (count of committed steps at the reader's observed_cursor), and scheduled_steps. Those committed values are not silently clamped and are not substituted from later state. Run-row summaries add declaration metadata declared, label, unit, grain, and display. Missing, logged-null, explicit zero, no-cohort, and undeclared remain distinct; the simulator does not derive presentation strings.

declare_metric(..., description="...") is an optional free-text extra. The library stores extra keys on the declaration and returns them; it does not require or interpret description.

Lineage declarations and selected-step observations

Lineage is bounded graph metadata for a run: a user-declared set of stages and, for each committed step, at most one user-recorded observation per stage. The library does not execute, schedule, reorder, infer, or time stages. It is not a workflow engine, event log, per-subject trace, or trend series.

with start_run(
    store="./results",
    project="sample-project",
    name="baseline",
    schedule=clock,
    description="Baseline configuration for the weekly study",
) as run:
    run.declare_lineage_stage(
        "ingest",
        label="Ingest inputs",
        description="Load step inputs",
        counts=["input_n"],
    )
    run.declare_lineage_stage(
        "prepare",
        label="Prepare cohort",
        parents=["ingest"],
        counts=["kept_n", "dropped_n"],
    )
    run.declare_lineage_stage(
        "evaluate",
        label="Evaluate subjects",
        parents=["prepare"],
        counts=["evaluated_n", "flagged_n"],
    )
    run.declare_lineage_stage("review", label="Optional review", parents=["evaluate"])
    run.declare_lineage_stage(
        "publish",
        label="Publish outputs",
        parents=["evaluate", "review"],
        counts=["output_n"],
    )

    for bounds in clock:
        kept_n = 0
        dropped_n = 0
        for group in (("s1", "s2", "s3"), ("s4",)):
            for subject in group:
                if subject == "s2":
                    dropped_n += 1
                else:
                    kept_n += 1
        with run.step(bounds) as step:
            step.log_lineage(
                [
                    {"stage_id": "ingest", "status": "completed", "counts": {"input_n": 8240}},
                    {
                        "stage_id": "prepare",
                        "status": "completed",
                        "counts": {"kept_n": kept_n, "dropped_n": dropped_n},
                    },
                    {
                        "stage_id": "evaluate",
                        "status": "completed",
                        "counts": {"evaluated_n": kept_n, "flagged_n": 0},
                    },
                    {"stage_id": "review", "status": "skipped"},
                    {"stage_id": "publish", "status": "completed"},
                ]
            )

A stage_id may be observed once in every different committed step; it names one stable graph node for the run. Nested loops inside a single run.step(...) must not call log_lineage per subject. User Python aggregates during those loops and records each stage once before the step commits. Repeating a stage_id on the same step is rejected rather than added, overwritten, or accumulated. Lineage is not an event log and must never scale with evaluated subjects.

Every logged observation requires stage_id and status. status is exactly completed, skipped, or failed (running and pending are not valid). counts is optional. Unknown keys, unknown stages, and unknown count names are rejected. A missing observation is not skipped, failed, completed, or zero; a missing declared count is not zero; explicit zero round-trips as zero.

Declarations use a short catalog transaction and are readable at zero commits through ResultReader.list_lineage_stages. Observations publish in the same transaction and commit_id as the rest of the committed step. Failed or poisoned steps leave no visible observations. A frozen through_cursor excludes later observations; there is no independent Lineage cursor. read_lineage_at_step requires a committed step_n at that cursor (no nearest step, no clamping, no series API).

Lineage lives in optional catalog tables only (no Lineage Parquet). Stores written by 0.1.1 remain readable: a new reader returns an empty declaration list and description is None. An old 0.1.1 reader ignores the new optional tables and continues to read established run, metric, table, rule, evidence, and decision data. result_format_version remains 1.

Rule declarations (Task 1)

User Python evaluates rules. The library stores declarations (Task 1), per-step aggregates / set-aggregates (Task 2), and triggered-subject evidence with receipts (Task 3). Each logic_ref is claimed identity supplied by your pipeline; the library does not read source files, execute rule logic, or derive aggregate counts from evidence. Only subjects that the caller says were triggered are logged; evaluated populations belong in user-computed aggregates, not evidence rows.

All declare_rule calls must finish before the first committed step. Definitions are written in a short catalog transaction (no commit_id) and are visible through ResultReader.list_rule_definitions even when the run has zero committed steps.

with start_run(store="./results", project="sample-project", name="baseline", schedule=clock) as run:
    run.declare_rule(
        "R12",
        label="Velocity",
        description="Velocity 14d",
        version="1",
        target={"kind": "entity", "type": "subject", "key": "subject_id"},
        parameters={"window": "P14D", "k": 11},
        logic_ref={"module": "pipeline.rules.velocity", "digest": "sha256:..."},
        category="evaluation",
        tags=["velocity", "monitoring"],
    )

definition_fingerprint is SHA-256 of canonical JSON over rule_id, version, target.kind, target.type, target.key, parameters, and logic_ref (not label, description, category, tags, or run-level code_ref). Optional Rule Foundation catalog tables are additive at result_format_version = 1.

Rule aggregates (Task 2)

All aggregate and set-aggregate values are calculated in user Python. The library stores what you log and does not reconcile counts, compute set overlap, or inspect the evaluated population.

Call step.log_rule_aggregates(...) and/or step.log_rule_set_aggregates(...) only inside an active run.step(...) context. Rows become visible only when that step commits (same short publication transaction as metrics and tables). Missing is not zero: a declared rule with no aggregate row for a step is Not logged, not numeric zero.

with run.step(clock.bounds_for(1)) as step:
    step.log_rule_aggregates(
        {
            "rule_id": "R12",
            "evaluated_n": 100,
            "unique_triggered_n": 4,
            "first_triggered_n": 3,
            "repeat_triggered_n": 2,
            "extras": {"note": "user-computed"},
        }
    )
    step.log_rule_set_aggregates(
        {
            "op": "intersection",
            "rule_ids": ["R12", "R13"],
            "unique_n": 2,
        }
    )

Set operations require every rule_id to be declared on the run and to share the same (target.kind, target.type, target.key). intersection and union require at least two unique rule ids (stored in sorted order). difference requires exactly two rule ids in operand order (A, B) meaning A minus B.

Read aggregates with ResultReader.read_rule_aggregates_at_step, read_rule_aggregate_series, and read_rule_set_aggregates_at_step. All of these require through_cursor and a committed step_n (or step range) at that cursor. Selected-step reads keep the full extras object. Trend series fields are the standardized scalar core measures only (evaluated_n, trigger_event_n, unique_triggered_n, first_triggered_n, repeat_triggered_n, cumulative_distinct_n); extras is not a series field.

read_rule_aggregate_series is cadence-neutral and bounded:

• MAX_RULE_AGGREGATE_SERIES_STEPS (1000) caps to_n - from_n + 1, which covers a 600-step run without assuming a 365-day calendar;
• MAX_RULE_AGGREGATE_SERIES_COUNT (8) caps the number of returned series;
• a request is either one rule with multiple measures or one measure with multiple rules, never a rules × measures matrix.

Triggered-subject evidence (Task 3)

Rule logic stays in user Python. A hit is not a decision and is not evaluation truth. The writer records only triggered subjects, never the evaluated population:

with run.step(clock.bounds_for(1)) as step:
    step.log_rule_aggregates({
        "rule_id": "R12",
        "evaluated_n": 1_000_000,
        "unique_triggered_n": 1,
    })
    step.log_rule_hits(
        rule_id="R12",
        rows=[{
            "target_id": "A-19",
            "evidence": {"score": 0.91, "reason": "velocity"},
        }],
    )

with run.step(clock.bounds_for(2)) as step:
    step.log_rule_hits(rule_id="R12", rows=[])

with ResultReader(store="./results") as reader:
    cursor = reader.latest_cursor()
    status = reader.read_rule_evidence_status(
        run_id, through_cursor=cursor, step_n=1, rule_id="R12"
    )
    page = reader.read_table(
        run_id,
        "rule_hits",
        through_cursor=cursor,
        step_n=1,
        limit=100,
        offset=0,
        filter=[("rule_id", "eq", "R12")],
    )

status distinguishes not_recorded, logged_empty, and recorded. Evidence is visible only when the step commits. Evidence pages are bounded: the default is 100 rows and the maximum is 1,000; callers must supply exactly one rule_id equality predicate. Evidence objects are stored as opaque, canonical UTF-8 JSON text. When callers provide mappings, Python has already resolved any duplicate object keys, so duplicate keys that are no longer observable cannot be detected. filtered_empty and page_past_end describe only a page query for stored evidence; they do not replace stored status.

First-class decisions (Task 4)

Decisions are user-supplied records, not rule-engine output. Declare one decision stream before the first committed step. Its target namespace uses the same kind, type, and normalized, case-preserving key rules as rule targets. A run may use an entity or action namespace, but it must not mix namespaces in one stream.

with start_run(store="./results", project="sample-project", name="baseline", schedule=clock) as run:
    run.declare_rule(
        "R12",
        label="Velocity",
        description="Velocity 14d",
        version="1",
        target={"kind": "entity", "type": "subject", "key": "subject_id"},
        parameters={"window": "P14D"},
        logic_ref={"module": "pipeline.rules.velocity"},
    )
    run.declare_table(
        "decisions",
        role="decisions",
        target={"kind": "entity", "type": "subject", "key": "subject_id"},
    )
    with run.step(clock.bounds_for(1)) as step:
        step.log_decisions({
            "decision_id": "D-2025-01-02-A-19",
            "target_id": "A-19",
            "contributing_rules": [{
                "rule_id": "R12",
                "version": "1",
                # Supply the fingerprint recorded for the R12 declaration.
                "definition_fingerprint": "sha256:<R12-fingerprint>",
            }],
            "payload": {"action": "review", "priority": 2},
        })

decision_id and target_id are NFC-normalized, trimmed, nonempty, case-preserving UTF-8 strings of at most 1,024 characters. Decision IDs are unique within the decision stream and step; reusing one on a later step is allowed. contributing_rules is optional, but when supplied it is a nonempty, canonicalized list whose declared rule ID, version, fingerprint, and target namespace all match the run declarations. This is claimed provenance, not proof of a matching hit or causal contribution.

Read configuration, stored status, and bounded rows separately:

with ResultReader(store="./results") as reader:
    target = reader.get_decision_target(run_id)
    status = reader.read_decision_status(
        run_id, through_cursor=reader.latest_cursor(), step_n=1
    )
    page = reader.read_table(
        run_id, "decisions", through_cursor=reader.latest_cursor(), step_n=1,
        limit=100, offset=0, sort=[("decision_id", "asc")],
    )

status distinguishes not_recorded, logged_empty, and recorded; page results additionally distinguish filtered_empty and page_past_end. Decision pages default to 100 rows and reject more than 1,000. Reads are pinned to through_cursor and require an exactly committed step_n; they do not clamp or select a nearest step. A zero-commit declaration is readable but does not invent step 1. Arbitrary decision payload is opaque canonical JSON; finite numbers are required and integers at or above 2**53 are stored as decimal strings.

A rule hit is not automatically a decision, and a decision is not automatically evaluation truth. The simulator records artifacts explicitly logged by user Python; it does not execute rule or decision logic, infer contributing rules from hits, materialize evaluated-subject populations, or perform analytic evaluation.

Reader

from mini_atlas_simulator import ResultReader

with ResultReader(store="./results") as reader:
    cursor = reader.latest_cursor()
    run = reader.get_run(run_id)
    steps = reader.list_committed_steps(run_id, through_cursor=cursor)
    metrics = reader.read_metrics(
        run_id,
        definition_ids=["slice_node_count"],
        through_cursor=cursor,
        step_n=steps[-1].step_n,
    )
    page = reader.read_table(
        run_id,
        "subject_scores",
        through_cursor=cursor,
        step_n=steps[-1].step_n,
        limit=100,
        offset=0,
        sort=[("score", "desc")],
    )
    rules = reader.list_rule_definitions(run_id)
    agg = reader.read_rule_aggregates_at_step(
        run_id, through_cursor=cursor, step_n=steps[-1].step_n
    )

list_rule_definitions is not step-scoped and returns [] for catalogs that predate Rule Foundation tables. Aggregate reads return empty/logged=False entries when rule tables are absent; they still require a committed step at the cursor when a step_n is requested.

ResultReader does not create schema, set WAL, or mutate writer files. A missing store directory, empty directory, or missing catalog.sqlite raises CatalogError and creates nothing. A format-v1 catalog that contains only catalog_info (no writer tables) is a valid initialized empty store: discovery methods return empty results and latest_cursor() is 0. A nonempty partial writer schema is CatalogError and is never repaired.

Composing reads take through_cursor and see only committed steps at or before that cursor. Format v1 stores metric and stream declarations without a declaration commit_id, so a definition declared after later commits may still be visible when reading an earlier cursor; points remain hidden until their publication commit_id. That is a format-v1 limitation, not a reader pin bug.

Opening a live WAL catalog read-only may create or map SQLite's catalog.sqlite-shm index and may create an empty catalog.sqlite-wal. Those sidecars are not schema changes; the reader does not write catalog pages or change journal_mode.

Project and run name search treats %, _, and \ in q as literals. Table pages order nulls with DuckDB NULLS LAST for both asc and desc. _commit_id is injected after the Parquet query and cannot be used as a sort or filter column.

A missing as_of or metric point is missing / Not logged, never numeric zero. Table coverage.status is one of ok, not_recorded, logged_empty, filtered_empty, or page_past_end. presentation_status is derived from stored status plus the writer heartbeat; it never authorizes GC or a second writer.

Examples

See examples/README.md. The first writer invocation creates the example project. A later invocation reuses that project and adds a distinctly named run. read_results.py needs --run-id when more than one run exists.

python examples/tabular_live.py --store ./results
python examples/tabular_live.py --store ./results --run-name daily-tabular-2
python examples/read_results.py --store ./results --run-id <run_id>

To watch commits arrive from a second terminal:

python examples/tabular_live.py --store ./results --step-delay 0.5

python examples/read_results.py --store ./results --poll-seconds 60

examples/tabular_live.py logs illustrative user metrics (coverage, reference count, observed count). Those labels are not built-in product behavior. Tests may pass --steps for a short run; the default remains the 365-day demonstration.

Current limitations

Rule declarations, aggregates, set aggregates, triggered-subject evidence, and decisions exist in the simulator. Optional run descriptions, additive list fields, and bounded Lineage graph metadata are also present. The simulator remains independently usable without Flask, React, Node, Docker, or the monitor. The separately installable monitor is read-only.

Still deferred: investigation/lineage UI, resume and stale-lock takeover, GC, export, run comparison, graph integration, evaluation-helper suite, later roadmap passes, Dash, and PyPI publication. Windows support is provisional and gated on CI. This does not claim that Windows is fully supported. Ubuntu CI jobs are provisional evidence for ubuntu-latest only. They are not evidence that native Linux in general was tested, that every Linux distribution was tested, or that desktop-browser behavior was tested on Linux.

Tests

python -m pytest