relflow 0.2.3

relflow builds PyTorch/Lightning models from nested records streamed through Apache Arrow. Typed fields represent values; branches combine them into local contexts; decoders learn to predict selected fields from the available context.

Install

Python 3.12 or newer:

uv add relflow

Add relflow[text] for Hugging Face text encoders or relflow[serving] for the HTTP runtime.

Describe One Record

line_items:
  - sku: A12
    quantity: 2
    price: 19.99
  - sku: B07
    quantity: 1
    price: 45.50
returned: false

YAML illustrates one observation. Store these records as nested Arrow structs and lists in Parquet datasets; relflow scans them in batches.

import relflow as rf

model = rf.Model(
    d_model=64,
    n_layers=2,
    n_heads=4,
    batch_size=128,
    line_items=rf.Branch(
        length=32,
        sku=rf.Category,
        quantity=rf.Number,
        price=rf.Number,
    ),
    returned=rf.Boolean(mask=True),
)

Parent keywords name each field. mask=True makes returned a supervised target whose value never enters the encoder. The branch builds line-item context before its reduced representation reaches the order root.

Stream Training And Prediction

The paths below are application-supplied Parquet datasets. Training and validation records include returned; request records omit it.

import lightning.pytorch as lit
import pyarrow.dataset as ds

train = ds.dataset("warehouse/train", format="parquet")
validation = ds.dataset("warehouse/validation", format="parquet")
requests = ds.dataset("warehouse/requests", format="parquet")

model.optimizer = rf.adamw(learning_rate=1e-3)
data = rf.ArrowDataModule(
    model=model, train=train, validate=validation, predict=requests
)
trainer = lit.Trainer(
    max_epochs=30, devices=1, callbacks=[rf.Writer("predictions")]
)
trainer.fit(model=model, datamodule=data)
trainer.predict(model=model, datamodule=data, return_predictions=False)

predictions = ds.dataset("predictions", format="parquet")

ArrowDataModule opens a fresh scan for each pass and prepares model batches as records arrive. rf.Writer writes prediction batches to predictions/rank-0.parquet; return_predictions=False avoids collecting all outputs in memory. Open the output as an Arrow dataset for further batch processing. Decoded values live under its predictions column at addresses such as /returned.

Documentation

• Getting started
• Model structure
• Data types
• Arrow data loading
• Preprocessing
• Training and checkpoints
• Batch inference
• Prediction output

Docs use static examples and Typst model diagrams. Build them with make render; run make check-docs to validate the render in a temporary directory. Package checks use uv run pytest; synthetic learning checks use make proofs. See CONTRIBUTING.md for development conventions.

Community · Apache 2.0 license