datarecord

Dimensioned attribute data with a declared schema.

A record holds components (named members of a type), connections between components and buses, attribute values over both, and the axes those values vary along. A schema declares what may exist; the data says what does.

Records stack: a layer is a partial record on top of a parent, resolved last-writer-wins, so a scenario variant costs the rows it changes rather than a copy of everything. On disk a record is a plain parquet directory that a tool knowing nothing about this package can read.

datarecord depends only on duckdb, narwhals and pydantic. It names no modelling framework — a framework consumes a record, a workflow engine produces one, and neither needs to know how the other works.

Documentation

Full documentation is at https://energy-models.github.io/datarecord/:

• Usage — reading, editing, layering and writing a record
• Design — what a record is and why, the authoritative design
• API Reference — every public symbol

Installation

pip install datarecord           # core
pip install datarecord[pypsa]    # with the PyPSA tool

A taste

from datarecord import DirectoryRecord, connect

con = connect()
record = DirectoryRecord("s3://bucket/my-record/", con)

record.entity_types["Generator"].collect()  # wide member rows
record.attributes["p_max_pu"].collect()  # long value rows, one per value
record.flags("Generator")  # which axes each attribute uses

Every frame is a narwhals.LazyFrame — a plan, not data. Nothing is read until you .collect().

Records stack, and a WorkingRecord accumulates edits that become one layer at commit:

from datarecord import WorkingRecord, NewChild

w = WorkingRecord(revision.record, con)
w.set("p_nom", 150.0, names=["wind1", "wind2"])
child = w.commit(NewChild())

See Usage for the rest.

Development

This project is managed by pixi:

git clone https://github.com/energy-models/datarecord
cd datarecord

pixi run test    # the test suite
pixi run lint    # ruff, prettier, taplo, typos, zizmor, reuse, mypy

See CONTRIBUTING.md for the workflow and conventions, and AGENTS.md for how AI-assisted contributions must be marked.