dataioc

Declarative data dependency graphs for Python.

Each DataDescriptor names a quantity and defines how it is derived from its direct dependencies. DataIoC composes these local rules into a graph, then resolves and caches only the subgraph required by the result you request. Bind a provider to any quantity to replace that part of the graph without changing downstream calculations.

English | 简体中文 | Documentation | PyPI

Example

The raw readings 10, 20, and 60 represent measurements of 1, 2, and 6.

from dataioc import DataDescriptor, DataIoC


class RawReadings(DataDescriptor):
    pass


class Measurements(DataDescriptor):
    def __build__(self, data):
        return tuple(value / 10 for value in data[RawReadings])


class Statistics(DataDescriptor):
    def __build__(self, data):
        values = data[Measurements]
        return sum(values) / len(values), max(values)


class Report(DataDescriptor):
    def __build__(self, data):
        mean, peak = data[Statistics]
        return f"mean={mean:g}, peak={peak:g}"


data = DataIoC().add(RawReadings, (10, 20, 60))
assert data[Report] == "mean=3, peak=6"
assert data[Measurements] is data[Measurements]

data[Report] is the only request the caller has to make. The container follows the dependencies and caches each value it builds.

To use recorded measurements instead, bind a provider for Measurements:

recorded = DataIoC().add_provider(Measurements, lambda _: (1.0, 2.0, 6.0))
assert recorded[Report] == "mean=3, peak=6"

RawReadings is no longer needed; Statistics and Report stay as they are.

When it helps

If both the inputs and the calculation path are fixed, ordinary function calls are simpler. Use dataioc when the relationships stay stable but a value may come from live measurements, recorded data, a simulation, or an estimate.

dataioc grew out of deinterf. In its direction-cosine example, the same compensation terms work whether direction cosines are derived from magnetic-vector measurements or supplied by an INS estimate. dvmss applies the pattern to simulation: supply the inputs, request Tmi, and let the container resolve the intermediate quantities.

Scope

dataioc resolves data dependencies synchronously in the current process; it is not a workflow scheduler. Use a fresh container for each dataset or provider configuration. See Core concepts for caching, diagnostics, and other runtime limits.

Install

python -m pip install dataioc
python -m pip install "dataioc[numpy]"

Documentation

• Quickstart: build a result from local dependency rules.
• Core concepts: understand descriptors, builders, caching, and failure behavior.
• Providers: bind a quantity to another source or derivation.
• Indexed data: reuse one model across related data groups.
• NumPy: use array subclasses.
• API: look up interfaces.

Contributors

yanang007 wrote the original container. dyuu7 shaped the design, extracted it into dataioc, and maintains the project.

Licensed under the MIT License.