Skip to main content

sdflow

Open-source Python framework for sequential decision problems.

A sequential decision problem is a loop. State S_t is observed, a decision a_t = A^π(S_t) is taken, exogenous information W_{t+1} is revealed, a new state S_{t+1} = S^M(S_t, a_t, W_{t+1}) is reached and a contribution C(S_t, a_t, W_{t+1}) is earned. The goal is a policy π that maximises the expected total contribution.

sdflow gives that loop a concrete form:

  • Problems built from point-in-time datasets, environments and objectives: every field is bitemporal, so a model physically cannot read data that was not knowable at the decision origin.
  • Policies built from predictors, optimizers and simulators, connected by agents.
  • One engine that runs a policy the same way in a backtest over history, a simulation over synthetic exogenous information, and live operation.
  • Verifiable scoring: official splits, a Verifier that fits a fresh submission itself and scores the hidden holdout, bundled benchmarks with reference baselines.

The core is domain-neutral. Energy, the best-supported domain, is an extra: sdflow[energy] adds the asset classes built on energydatamodel. sdflow is the successor of emflow; emflow 0.3 code ports by renaming the import.

Installation

pip install sdflow            # the core
pip install "sdflow[energy]"  # plus the energy asset classes

Quick start

import sdflow as sd

problem = sd.load_problem("gefcom2014:solar")      # a bundled benchmark
from sdflow.benchmarks.gefcom2014.baseline import ClimatologyQuantiles

result = sd.evaluate(problem, ClimatologyQuantiles(), split="validation")
print(result.score, result.n_origins)

A predictor is fit(train) on a leakage-safe training view and predict(obs) per decision origin; the environment computes the objective from what it returns. See docs/ for the walkthrough: datasets and feeds, environments, features, models, problems and the run engine.

Agents

A policy is a predictor and an optimizer. The predictor forecasts, the optimizer turns the forecast and the state into the decision, the environment settles it:

from sdflow.models import Agent, QuantileDecision

agent = Agent(my_quantile_predictor, QuantileDecision(c_under=3, c_over=1))
result = sd.evaluate("heftcom2024:trading", agent)        # revenue, not pinball loss

A predictor on its own is the agent whose optimizer is the identity, so the same predictor can be scored on a forecast leaderboard and, behind an optimizer, on a revenue one.

Stateful problems

A forecast problem is the stateless case of the loop. When a decision changes what the next decision faces, give the environment a simulator and a settlement:

from sdflow.envs import ColumnsAction, DecisionEnv, Settlement
from sdflow.models import StorageSimulator

class StorageRevenue(Settlement):
    fields = ("price",)                      # settle once the realised price is knowable
    def settle(self, ctx):                   # price what the store actually did
        return float(-(ctx.series("price") * ctx.state_after.realized).sum())

env = DecisionEnv(
    feed, origins, settlement=StorageRevenue(), objective=objective,
    action_spec=ColumnsAction("flow"),       # the decision: a flow per target period
    simulator=StorageSimulator(capacity=10, max_inflow=5, max_outflow=5),
)
obs, _ = env.reset()                         # obs.state.level == 0.0, obs.pending == ()

The observation carries the state after the last settled decision and the decisions still pending, which is exactly what a live participant knows at that clock time: committed, not yet metered. ForecastEnv and TradingEnv are this environment with a scoring and a market settlement.

Backtest, simulate, live

The same policy, the same environment, three sources of exogenous information:

result  = sd.evaluate(problem, agent)                                   # history
sim     = sd.simulate(problem, lambda: MyAgent(), n_scenarios=20)        # sampled draws
session = sd.LiveSession(problem, agent)                                 # the clock
step    = session.step(dataset_as_of_now)                                # one wake-up

A live session settles what has become knowable, decides for the origin open now and keeps its pending decisions and state between wake-ups. Fed the dataset truncated at each origin, it reproduces the backtest exactly. Fetching the data as of now is the runner's job (the Rebase toolkit's, in production); sdflow defines what it is handed and what comes back.

Leak checks

The feed makes look-ahead hard by construction. For what construction cannot catch, run the truncation check: the decision at an origin must be identical when everything not knowable at that origin is physically deleted from the dataset.

report = sd.truncation_invariance(problem, lambda: MyAgent(seed=0), k=6)
assert report.ok, report

signal-game:w8 is the framework's own check: a problem with exact reference scores (chance, Bayes, oracle) and no energy in it.

Registering your own problems

Problems in other packages register through the sdflow.problems entry-point group and then load by name anywhere the package is installed:

[project.entry-points."sdflow.problems"]
icing = "my_signals.icing.problem"
sd.load_problem("icing:se1")

Data can live on the Hugging Face Hub: a dataset repo with a rebase.yaml marker is readable through the rb:// filesystem sdflow registers with fsspec (pd.read_parquet("rb://dataset/<owner>/<repo>/<path>")).

Development

uv sync --group dev
uv run pytest

MIT licensed. Built by rebase.energy.

Metadata

Release files for sdflow 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sdflow 0.1.1
File Size Uploaded Provenance Actions
sdflow-0.1.1.tar.gz 5.5 MB Without provenance Details for sdflow-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sdflow 0.1.1
File Interpreter ABI Platform Actions
sdflow-0.1.1-py3-none-any.whl Python 3 none any Details for sdflow-0.1.1-py3-none-any.whl

Total release size: 10.9 MB

Release files / sdflow-0.1.1.tar.gz

Download URL sdflow-0.1.1.tar.gz
Size 5.5 MB
Tags Source
SHA-256 checksum
How to use checksums
ac54fe20e6bf82ac30c525753f826db171469c4ea484afc3bbf652dcdd50df80
BLAKE2b-256 checksum
How to use checksums
a24172449d45b1299e7dd37b9e883aca16fb9efac771d5d784278271a7b4c116
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.2

Release files / sdflow-0.1.1-py3-none-any.whl

Download URL sdflow-0.1.1-py3-none-any.whl
Size 5.4 MB
Tags Python 3
SHA-256 checksum
How to use checksums
f5efae44f487089a8432f421aa9ac1f7c50b9b267bb82b142d0cef4b6896a1f6
BLAKE2b-256 checksum
How to use checksums
521e093158bcfd8f3e70a08ac2e01fc12898c50e9dfc1f2ec3b48adf4227827d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.2

Security for sdflow 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page