datum-stream

Python bindings for Datum stream blueprints.

Datum mirrors the Source -> Flow -> Sink and GraphDSL vocabulary from the Rust crate while keeping Python execution explicit: building a pipeline creates an immutable blueprint, and work starts only when a runnable graph is materialized with a Runtime.

Install

python -m pip install datum-stream

The package requires Python 3.13 or newer and depends on PyArrow. User Python code always runs through Arrow UDF batches: integer-stream map, filter, and flat_map wrap callables into single-column Arrow batches, while map_batches is the vectorized RecordBatch -> RecordBatch tier.

For scalar integer hot paths, use the named kernels such as map_add, map_multiply, and filter_greater_than. For Arrow batch streams, use typed col() expressions for lowerable work and map_batches for arbitrary Python. Terminal arithmetic is named too: Sink.fold() / Sink.fold_sum() sum values, and Sink.fold_product() multiplies them. Graph partitions select the typed PartitionStrategy.MODULO constant.

The package ships PEP 561 stubs (py.typed) with generic Source, Flow, Sink, RunnableGraph, Inlet, and Outlet types. Public construction is always strict: graph wiring and declared Arrow schemas are validated during builder calls, and empty Arrow inputs require an explicit schema=....

Example

import datum

with datum.Runtime() as runtime:
    graph = (
        datum.Source.range(1, 5)
        .map_add(1)
        .to_mat(datum.Sink.fold_product())
    )
    completion = graph.run(runtime)
    assert completion.wait() == 120

Scope

The current Python surface covers integer linear streams, a focused GraphDSL surface (Broadcast, Balance, Merge, Partition, Zip, Concat, and Interleave), Arrow batch UDFs, and Datum Connect for trusted client/server execution. Connect can run linear plans, supported junction graphs, tuple FlowShape results, direct ZipShape runs, and Merge/Concat/Interleave FanInShape runs. Local Connect defaults to arrow-ipc; remote Arrow payloads should prefer arrow-ipc-zstd when compression is wanted. Those wire-format defaults were chosen by measurement.

For the broader guide, see the Datum Python docs in docs/guides/python.