Civic-Digital-Twins Modeling Framework

This repository contains a Python package implementing a Civic-Digital-Twins modeling framework. The framework is designed to support defining digital twins models and evaluating them in simulated environments with varying contextual conditions. We develop this package at @fbk-most, a research unit at Fondazione Bruno Kessler.

Note: this package is currently in an early development stage.

Conceptual Overview

The framework is organised in three layers.

Engine layer

The engine (civic_digital_twins.dt_model.engine) is an embedded DSL compiler. The programmer builds a computation graph (DAG) by composing typed nodes — constants, placeholders, and operations — using ordinary Python expressions. The graph is then linearised by topological sorting and evaluated by a NumPy-based interpreter that maps each node to the corresponding numpy operation.

import numpy as np

from civic_digital_twins.dt_model.engine.frontend import graph, linearize
from civic_digital_twins.dt_model.engine.numpybackend import executor

a = graph.placeholder("a")
b = graph.placeholder("b")
c = a * 2 + b

state = executor.State(values={a: np.asarray(3.0), b: np.asarray(1.0)})
executor.evaluate_nodes(state, *linearize.forest(c))
print(state.get_node_value(c))  # 7.0

See docs/design/dd-cdt-engine.md for a full description of the engine.

Model layer

The model layer (civic_digital_twins.dt_model.model) provides typed building blocks for defining a digital-twin model on top of the engine:

• Index / TimeseriesIndex — named wrappers around graph nodes. An index can be a constant, a distribution (sampled at evaluation time), or a formula.
• Model — a typed computation unit. Use the @define decorator to declare a Model subclass via a compute() method; @inputs, @outputs, and @expose decorators mark the contractual interface. Sub-models are wired via constructor arguments in compute(), producing a composable pipeline.
• ModelVariant — selects among pre-constructed Model implementations sharing the same I/O contract. The active variant is resolved by a string key (static) or a CategoricalIndex/graph node (runtime dispatch).

from scipy import stats

from civic_digital_twins.dt_model import DistributionIndex, Index, Model, define, inputs, outputs

@define("example")
class ExampleModel(Model):
    @inputs
    class Inputs:
        x: DistributionIndex
        y: DistributionIndex

    @outputs
    class Outputs:
        result: Index

    def compute(self, inputs: Inputs) -> Outputs:
        result = Index("result", inputs.x + inputs.y)
        return ExampleModel.Outputs(result=result)

model = ExampleModel(inputs=ExampleModel.Inputs(
    x=DistributionIndex("x", stats.uniform, {"loc": 0.0, "scale": 1.0}),
    y=DistributionIndex("y", stats.uniform, {"loc": 0.0, "scale": 1.0}),
))

See docs/design/dd-cdt-model.md for the full reference: index types, Model API, ModelVariant, and the domain modeling pattern; see docs/design/dd-cdt-modularity.md for multi-model composition and decomposition patterns.

Simulation layer

The simulation layer (civic_digital_twins.dt_model.simulation) runs a model against one or more scenarios:

• Scenario — wraps a model with optional value overrides and parameter axes; the canonical first argument to Evaluation and all ensemble classes.
• Evaluation — evaluates a model over a sequence of weighted scenarios, each of which maps every abstract index to a concrete value, and returns an EvaluationResult.
• Ensemble / WeightedScenario — a protocol and type alias that define the scenario contract consumed by Evaluation; concrete implementations (DistributionEnsemble, CrossProductEnsemble, …) draw or enumerate scenarios.

See docs/design/dd-cdt-simulation.md for the full reference: Scenario, ensembles, Evaluation, EvaluationResult, EvaluationHandle, and ModelEvaluator — the higher-level runner used by the worked examples (see Usage patterns below).

Besides the three layer subpackages, civic_digital_twins.dt_model hosts two top-level modules (axes, graph) with narrowly-scoped, deliberate roles — see civic_digital_twins/dt_model/README.md for the package-layout policy.

Usage patterns

The examples/ directory contains two worked examples, both driven through a domain-specific ModelEvaluator subclass (civic_digital_twins.dt_model.simulation.runner) rather than calling Evaluation directly: evaluator.evaluate(Scenario(model, ...), EvaluationConfig(...)) runs the engine internally and returns a domain-specific ModelOutput — a JSON-serialisable, optionally resumable summary — rather than a raw EvaluationResult. Both use the @define/compute() API (@inputs, @outputs, @expose, ModelVariant) — see docs/design/dd-cdt-modularity.md.

They differ in whether the model has context variables: categorical scenario factors outside the modeller's control (e.g. season, weather), as opposed to DistributionIndex parameters, which represent uncertainty the modeller chooses to sample directly.

Direct pattern (examples/mobility_bologna/) — no context variables, only DistributionIndex parameters. DistributionEnsemble draws S Monte-Carlo samples to produce weighted scenarios.

Context-variable pattern (examples/overtourism_molveno/) — the model has categorical context variables (season, weather, …), expressed as CategoricalIndex, and quantities whose distribution depends on that context, expressed as ConditionalDistributionIndex. Internally, CrossProductEnsemble enumerates the context combinations into weighted scenarios, and presence quantities are swept over a multi-dimensional grid via Evaluation.evaluate(parameters={pv: array, …}).

Installation

The package name is civic-digital-twins on PyPi. Install using pip:

pip install civic-digital-twins

or, using uv:

uv add civic-digital-twins

The main package name is civic_digital_twins:

import civic_digital_twins

or

from civic_digital_twins import dt_model

Minimum Python Version

Python 3.12. Tested against Python 3.12, 3.13, and 3.14.

API Stability Guarantees

The package is currently in an early development stage. We do not anticipate breaking APIs without a good reason to do so, yet, breaking changes may occur from time to time. We generally expect subpackages within the top-level package to change more frequently.

Development Setup

We use uv for managing the development environment.

To get started, run:

git clone https://github.com/fbk-most/civic-digital-twins
cd civic-digital-twins
uv venv
source .venv/bin/activate
uv sync --dev

We use pytest for testing. To run tests use this command (from inside the virtual environment):

pytest

Pull requests are automatically tested using GitHub Actions. PRs targeting dev run the fast CI (dev) workflow (format, lint, type-check, tests on Python 3.12). PRs targeting main run the full CI (release) workflow (all Python versions, doc examples, domain examples, SPDX check, dependency audit, and build smoke test).

Significant portions of this codebase — implementation, tests, and documentation — are developed with AI coding assistants (primarily Claude/Claude Code, also GitHub Copilot and OpenAI tools), directed and reviewed by the project's maintainers, who remain responsible for the correctness and design of everything merged.

Updating Dependencies

uv self update
uv sync --upgrade

Development model

This project follows a simplified GitHub Flow with an explicit dev branch:

feature/* ──PR─▶ dev ──PR─▶ main ──tag─▶ PyPI
           (CI dev)   (CI release)  (publish)

• Feature work happens on short-lived branches cut from dev.
• dev is the integration branch. It always carries a +dev version marker (e.g. 0.11.0+dev).
• main contains only released commits. Merging dev into main is always immediately followed by a version tag and a PyPI release.

Releasing

Step 1 — Merging a feature PR into dev

Before opening the PR, verify locally:

• Tests pass: uv run pytest
• Format, lint, and type-check pass: uv run ruff format --check ., uv run ruff check ., uv run pyright
• CHANGELOG.md [Unreleased] section updated (Added / Changed / Removed / Fixed; breaking changes flagged).
• Design docs (docs/design/) updated if public API or architecture changed.
• SPDX licence header present on any new .py or .md file.

Open the PR targeting dev. The CI (dev) workflow runs automatically; merge once it is green.

Step 2 — Preparing a release (promoting dev to main)

Perform the following steps on the dev branch before opening the dev → main PR:

1. Set the final version in pyproject.toml (remove the +dev suffix):

   version = "<version>"
   

2. Regenerate the lockfile:

   uv lock
   

3. Update CHANGELOG.md: promote [Unreleased] to [<version>] - <date> and add the corresponding comparison link at the bottom.

4. Check that documentation Last-Updated dates are in sync with actual commit dates:

   git log -1 --format="%ai" -- docs/design/dd-cdt-engine.md
   git log -1 --format="%ai" -- docs/design/dd-cdt-model.md
   git log -1 --format="%ai" -- docs/design/dd-cdt-modularity.md
   git log -1 --format="%ai" -- docs/design/dd-cdt-simulation.md
   git log -1 --format="%ai" -- docs/getting-started.md
   

   Update any Last-Updated fields that are out of date.

5. Verify that the runnable doc scripts are in sync with the documentation and execute without errors (also enforced by CI (release)):

   uv run python examples/doc/doc_engine.py
   uv run python examples/doc/doc_model.py
   uv run python examples/doc/doc_modularity.py
   uv run python examples/doc/doc_simulation.py
   uv run python examples/doc/doc_getting_started.py
   uv run python examples/doc/doc_overtourism_getting_started.py
   uv run python examples/doc/doc_readme.py
   

6. Verify that the full domain examples run end-to-end without errors (also enforced by CI (release); output images are written to examples/*/output/):

   uv run python examples/mobility_bologna/mobility_bologna.py
   uv run python examples/overtourism_molveno/overtourism_molveno.py
   

7. Verify that every tracked Python and Markdown file carries an SPDX header (also enforced by CI (release)):

   # Python files — should print nothing
   git ls-files '*.py' | xargs grep -rL "SPDX-License-Identifier"
   # Markdown files — should print nothing
   git ls-files '*.md' | xargs grep -rL "SPDX-License-Identifier"
   

   Add # SPDX-License-Identifier: Apache-2.0 (Python) or <!-- SPDX-License-Identifier: Apache-2.0 --> (Markdown) to any file that is missing the header.

8. Commit the release preparation. The Release-PR: true trailer tells CI (dev) to skip the +dev marker check for this specific commit — see ci-dev.yml:

   git add pyproject.toml uv.lock CHANGELOG.md docs/
   git commit -m "$(cat <<'EOF'
   chore: prepare v<version> release
   
   Release-PR: true
   EOF
   )"
   git push origin dev
   

Open the PR from dev to main. The CI (release) workflow runs the full verification suite automatically (all Python versions, doc examples, domain examples, SPDX headers, dependency audit, build smoke test). Merge once it is green, using "Create a merge commit" (the only method main's ruleset allows) — this keeps dev's commit history intact and attached to main, so the next release's dev → main diff only shows genuinely new commits.

Step 3 — Tagging and publishing

After the dev → main PR is merged, GitHub may have auto-deleted dev (if delete_branch_on_merge is enabled). Restore it from the merge commit before continuing — either click "Restore branch" on the merged PR page, or recreate it locally:

git checkout main && git pull
git push origin main:refs/heads/dev

Then tag and push the release:

git tag v<version> && git push origin main v<version>

Go to the repository's Releases page, review the auto-created draft, write release notes, and click Publish release. This triggers the publish.yml workflow, which builds the sdist + wheel, runs twine check, and publishes to PyPI via OIDC — no manual build or upload step is needed.

Step 4 — Post-release: bump dev back to development

After the release is published, switch back to dev and prepare it for the next development cycle:

git checkout dev && git pull

Edit pyproject.toml to bump to the next planned version with the +dev marker:

version = "<next-version>+dev"

Then:

uv lock

Add a fresh [Unreleased] section at the top of CHANGELOG.md:

## [Unreleased]

Commit and push:

git add pyproject.toml uv.lock CHANGELOG.md
git commit -m "chore: start v<next-version> development"
git push origin dev

Hotfix and backport releases

Two situations don't fit the normal dev → main flow above:

• Hotfix — a bug needs fixing in the version main currently sits at, while dev has independently moved on to unfinished next-version work that isn't ready to ship. Branch directly off main's current tip.
• Backport — a bug needs fixing in an older, already-superseded release line (main has moved past it by one or more versions). Branch off the old release tag instead.

Both use a dedicated hotfix/vX.Y.Z or backport/vX.Y.Z branch as a short-lived staging branch: individual pieces of work are PR'd and merged into it (never committed to directly), then it is either merged onward (hotfix) or tagged and released directly (backport).

Hotfix procedure

1. git checkout -b hotfix/v<version> origin/main, push it: git push -u origin hotfix/v<version>.
2. For each piece of work (bug fixes, release prep), branch off hotfix/v<version>, PR into it, merge. CI (release) runs automatically on these PRs (see below).
3. Once hotfix/v<version> contains everything for the release, open a PR from it into main — this is the actual release PR (version bump, changelog, etc. is typically already in place from step 2) and triggers CI (release) normally, exactly like a dev → main PR.
4. Tag and publish from main's new tip, exactly as in Step 3 above.
5. Forward-port the same fix into dev as its own small PR, so the next regular release doesn't reintroduce it. Don't touch dev's in-progress work otherwise.

Backport procedure

1. git checkout -b backport/v<version> v<old-version>, push it.
2. For each piece of work, branch off backport/v<version>, PR into it, merge — same as the hotfix flow.
3. Tag and publish directly from backport/v<version>'s tip. This branch never merges into main or dev — their code has diverged too far for that to be meaningful. On the Releases page, review the draft and explicitly set "Set as the latest release" to No before publishing, so the repository's "latest" designation stays on the current release.
4. Separately, port the same fix forward into whichever line is actually current (typically as a hotfix per above, or folded into normal dev work).

CI on maintenance branches

CI (release) (.github/workflows/ci-release.yml) triggers on push/pull_request against main, hotfix/**, and backport/**, plus workflow_dispatch for manual runs. A brand-new hotfix/*/backport/* branch created straight from a historical tag may carry an older snapshot of the CI workflow that lacks this pattern — add it (and a workflow_dispatch: {} fallback) as part of setting up the branch.

The pattern trigger only fires once a branch's own committed copy of the workflow already includes it, so the very first PR into a brand-new branch (before that trigger exists there) won't auto-fire. Verify it manually instead, dispatching against the PR's source branch directly (this works even if the target branch doesn't have the trigger yet):

gh workflow run <workflow-file> --ref <branch-name>

One-time setup

PyPI Trusted Publisher: must be configured before the first release. See the PyPI Trusted Publishers documentation.

Branch protection: configure GitHub Rulesets (Settings → Rules → Rulesets) to require CI (dev) to pass before merging into dev, and all CI (release) jobs to pass before merging into main. Direct pushes to main should be blocked; maintainers should be allowed to bypass dev protection for post-release bump commits. main's ruleset must allow only the merge method (no squash/rebase) and must not require linear history — squashing or rebasing the dev → main PR mints new commit SHAs unrelated to dev's own commits, which detaches dev from main's history and makes every subsequent dev → main diff re-show already-released commits.

Documentation

License

SPDX-License-Identifier: Apache-2.0