datum-stream 0.10.4

Python bindings for Datum stream blueprints

datum-stream

Python bindings for Datum stream blueprints.

Datum mirrors the Source -> Flow -> Sink and GraphDSL vocabulary from the Rust crate while keeping Python execution explicit: building a pipeline creates an immutable blueprint, and work starts only when a runnable graph is materialized with a Runtime.

Install

python -m pip install datum-stream

The package requires Python 3.13 or newer and depends on PyArrow. User Python code always runs through Arrow UDF batches: integer-stream map, filter, and flat_map wrap callables into single-column Arrow batches, while map_batches is the vectorized RecordBatch -> RecordBatch tier.

For scalar integer hot paths, use the named kernels such as map_add, map_multiply, and filter_greater_than. For Arrow batch streams, use typed col() expressions for lowerable work and map_batches for arbitrary Python.

The package ships PEP 561 stubs (py.typed) with generic Source, Flow, Sink, RunnableGraph, Inlet, and Outlet types. Public construction is always strict: graph wiring and declared Arrow schemas are validated during builder calls, and empty Arrow inputs require an explicit schema=....

Example

import datum

with datum.Runtime() as runtime:
    graph = (
        datum.Source.range(0, 10)
        .map(lambda x: x * 3)
        .filter(lambda x: x > 10)
        .to_mat(datum.Sink.collect())
    )
    completion = graph.run(runtime)
    print(completion.wait())

Scope

The current Python surface covers integer linear streams, a focused GraphDSL surface (Broadcast, Balance, Merge, Partition, Zip, Concat, and Interleave), Arrow batch UDFs, and Datum Connect for trusted client/server execution. Connect can run linear plans, supported junction graphs, tuple FlowShape results, direct ZipShape runs, and Merge/Concat/Interleave FanInShape runs. Local Connect defaults to arrow-ipc; remote Arrow payloads should prefer arrow-ipc-zstd when compression is wanted. Those wire-format defaults were chosen by measurement.

For the broader guide, see the Datum Python docs in docs/guides/python.