lightstream

Move Arrow tables between processes, services and storage from Python without adding a gRPC stack or writing transport-specific framing.

lightstream provides one streaming API across files, memory maps, sockets and network transports. Readers return minarrow objects and implement the Arrow PyCapsule protocol, allowing PyArrow, Polars, DuckDB and other Arrow-compatible libraries to consume data straight off the wire without an intermediate conversion.

Installation

pip install lightstream-io

Usage

Everything is read or write.

The URI selects the transport, protocol selects the wire framing, and file extensions select the storage format.

import lightstream as ls

Files

Read Arrow IPC, Parquet, CSV and JSON through the same interface.

# Read an entire dataset.
table = ls.read("quotes.arrow").read_all()

# Stream batches without loading the complete dataset into memory.
for batch in ls.read("large.parquet"):
    process(batch)

# Write any Arrow-compatible object.
with ls.write("output.parquet", compression="zstd") as writer:
    writer.write(table)

Arrow IPC readers use memory mapping and out-of-core routing where appropriate.

Readers yield minarrow.Table batches. read_all() returns a Table when the result is contiguous or a ChunkedTable when multiple chunks must be preserved.

Network transports

The same interface streams raw Arrow IPC over TCP, Unix domain sockets, WebSocket, HTTP, QUIC, WebTransport and standard I/O.

for batch in ls.read("tcp://feed.example.com:9000"):
    process(batch)

Supported URI schemes include:

Transport URI
TCP tcp://host:port
Unix domain socket uds:///path/to/socket
WebSocket ws://host/path
Secure WebSocket wss://host/path
HTTP http://host/path
HTTPS https://host/path
QUIC quic://host:port
WebTransport wt://host/path
Standard input/output stdio://

Accepting connections

Either endpoint can accept the connection.

Set accept=True to bind the endpoint on first use and block until the connecting peer arrives.

writer = ls.write(
    "uds:///tmp/feed.sock",
    accept=True,
)

writer.write(table)
writer.close()

The listener remains available for the lifetime of the process. Connections arriving while a serving loop is active wait in the listener backlog, allowing the accepting endpoint to operate as a persistent server.

TLS

QUIC and WebTransport include TLS at the protocol level. The accepting endpoint presents a PEM certificate and private key, while the connecting endpoint verifies the certificate using PEM roots.

writer = ls.write(
    "quic://0.0.0.0:4433",
    accept=True,
    tls_cert="server.pem",
    tls_key="server-key.pem",
)

for batch in ls.read(
    "quic://feed.example.com:4433",
    tls_ca="roots.pem",
):
    process(batch)

Secure WebSocket and HTTPS transports use their corresponding TLS-enabled URI schemes.

Arrow interoperability

Every reader implements the Arrow PyCapsule stream protocol.

Arrow-compatible libraries can therefore consume a Lightstream reader directly.

import duckdb
import lightstream as ls

reader = ls.read("quotes.arrow")

result = duckdb.sql(
    "SELECT SUM(qty) FROM reader"
)

The same reader can be passed to libraries such as Polars without first converting its batches into Python objects.

Lightstream protocol

The Lightstream protocol multiplexes named Arrow tables and opaque messages over one connection.

Opaque payloads can carry formats such as Protobuf or MessagePack, while table channels retain their Arrow schema and batch representation.

reader = ls.read(
    "uds:///tmp/feed.sock",
    protocol="lightstream",
)

reader.register_table(
    "quotes",
    representative_table,
)
reader.register_message("health")

for frame in reader:
    if frame.is_table():
        on_quotes(frame.table)
    else:
        on_health(frame.payload)

frame.table is a minarrow.Table. Message payloads are returned as bytes.

Both peers must register compatible channels before exchanging frames.

The protocol is transport-independent, so the same table and message definitions can be used over TCP, Unix domain sockets, WebSocket, HTTP, QUIC or WebTransport.

Standard I/O pipelines

The standard I/O transport can stream Arrow IPC or line-oriented text.

Set format="csv" to write CSV records or format="json" to write NDJSON. Text emitted by another process is decoded back into table batches on read.

This allows Unix tools, command-line programs and agents to participate directly in a table pipeline.

python examples/run-example.py sed
python examples/run-example.py jq
python examples/run-example.py claude
python examples/run-example.py sql

The pipeline examples demonstrate:

  • sed rewriting rows in flight
  • jq filtering NDJSON and returning CSV
  • an agent diagnosing invalid batches
  • rolling DuckDB SQL over a live stream

Transport examples

Worked examples are provided under examples/transport/.

Each transport includes:

  • a Python server
  • a Rust server
  • a Python client
  • the same million-row input table

Use the example router to select the transport and backend:

# Python backend over TCP.
python examples/run-example.py tcp

# Rust backend over QUIC.
python examples/run-example.py quic --rust

Available transports are:

tcp
uds
ws
wss
http
https
quic
wt
stdio

TLS examples generate a private root certificate and a server certificate for local execution.

Building from source

Lightstream currently requires the Rust nightly toolchain. The repository selects the required toolchain automatically through rust-toolchain.toml.

pip install maturin
maturin develop
pytest tests/

Licence

Copyright © 2025–2026 Peter Garfield Bower.

Licensed under the Mozilla Public License 2.0. See LICENSE for the standard terms.

See MPL 2.0 FAQ if you are unfamiliar with this open-source license.

Affiliation notice

Lightstream is not affiliated with Apache Arrow or the Apache Software Foundation.

It implements public Arrow formats through Minarrow and interoperates with the Arrow ecosystem through the Arrow PyCapsule protocol.

lightstream is maintained by SpaceCell and forms part of its open-source foundation for high-performance data computing.

Project links

Data verified by PyPI on Jul 27, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for spacecell from gravatar.com spacecell

Credits

Author: Peter Bower

GitHub Statistics

Data verified by PyPI on Jul 27, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MPL-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Tags

arrow streaming minarrow ipc parquet quic

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lightstream_io-0.6.0.tar.gz (584.7 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

lightstream_io-0.6.0-cp39-abi3-macosx_11_0_arm64.whl (3.1 MB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

lightstream_io-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl (3.3 MB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

File details

Details for the file lightstream_io-0.6.0.tar.gz.

File metadata

  • Download URL: lightstream_io-0.6.0.tar.gz
  • Upload date:
  • Size: 584.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for lightstream_io-0.6.0.tar.gz
Algorithm Hash digest
SHA256 f1289a75a342d82e02e1cbf23fcc68ca5310e91bdcd428e2810ef54c0286552e
MD5 f70898b848331211902e8aab439c039d
BLAKE2b-256 b24f2c36a1d25efaa3f7848e373df26836e8ad259d4c22632b2cc12c67fa0920

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.6.0.tar.gz:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 bae7229b84549da2c15eb9668aa037feadd285030c31ca5068c078846d3e7864
MD5 81541ddf7983e18d13118bb85f9e4571
BLAKE2b-256 bd0ba1e6879456f9fd25b82aff87399152de2ef3f2de2ad5b4816f8800570ee8

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 55e7ddbd7577d6e1865e0cf4bea272bf1a32099ed3e9ce58cfc19fca2a6f6f67
MD5 3aa1a8df95d28dcd24147aa0ece1d3b3
BLAKE2b-256 f60bf1e207cfb32fe086d6d4baa45200ecd379e1252c1e620765af38ca1ecfc4

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.6.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.6.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8ebe8a6f2707b603360f85c5bcaf26c61f49f509430c8dcac4f8b12bcd32e4e3
MD5 b28e3b796a42d38e5b91bf2385eef6fa
BLAKE2b-256 c10e4a0d7e7122db2387dd2599165a25b9e93270f6d2e619ce2896ac21c7ac57

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.6.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 1a4222771634f3abfad69da810a9261a67edbee5339d71cf3f86de61cd7cf1c5
MD5 410e81201baba8c7ae0d61a704fe8ffe
BLAKE2b-256 3f1ed07a299f8371b3a1380d136830c8e20d77ee8da2bf094e12b36ec6887983

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page