SGN

SGN is a lightweight Python library for building streaming data pipelines. Connect sources, transforms, and sinks into clear workflows while SGN handles asynchronous execution under the hood. Zero external dependencies.

Installation

pip install sgn

Example

import functools
from sgn import Pipeline, IterSource, CallableTransform, CollectSink

def scale(frame, factor: float):
    return None if frame.data is None else frame.data * factor

src = IterSource(name="src", source_pad_names=["H1"], iters={"H1": [1, 2, 3]})

transform = CallableTransform.from_callable(
    name="t1",
    sink_pad_names=["H1"],
    callable=functools.partial(scale, factor=10),
    output_pad_name="H1",
)

sink = CollectSink(name="snk", sink_pad_names=["H1"])

p = Pipeline()
p.connect(src, transform)
p.connect(transform, sink)
p.run()

assert list(sink.collects["H1"]) == [10, 20, 30]

Documentation

• Tutorial — New to SGN? Build your first pipeline step by step.
• User Guide — Solve specific problems: connecting elements, grouping, parallelism, and more.
• Background — Understand how SGN works: execution model, core concepts, and design decisions.
• Reference — Elements and auto-generated API documentation.

Related Libraries

• sgn-ts: TimeSeries utilities for SGN
• sgn-ligo: LSC specific utilities for SGN

Instructions for reviewers

• We will use glreview for this project. Please familiarize yourself with it.

Unreviewed modules

• A librarian or designated person will generate issues using glreview for all modules
  • The issue will contain an AI review with Claude
  • An AI generated merge request will be made to address issues. The librarian/designee may take liberties to guide that process so that it is high quality
  • The librarian/designee will seek out a human to be assigned to each issue at this point
• The assigned human will:
  • Verify the pre-review checklist in the git issue (and check boxes if appropriate)
  • Post independent findings as well as notes on the AI review in findings
  • Fill out the overall summary table
  • update the MR (if required) and work with the code authors to approve and merge the MR
  • The issue will be closed when the MR is merged
  • run glreview signoff after the issue is closed

Reviewed modules - updating when code changes

• FIXME. More to say on this later