lightstream

Move Arrow tables between processes, services and storage from Python without adding a gRPC stack or writing transport-specific framing.

lightstream provides one streaming API across files, memory maps, sockets and network transports. Readers return minarrow objects and implement the Arrow PyCapsule protocol, allowing PyArrow, Polars, DuckDB and other Arrow-compatible libraries to consume data straight off the wire without an intermediate conversion.

Installation

pip install lightstream-io

Usage

Everything is read or write.

The URI selects the transport, protocol selects the wire framing, and file extensions select the storage format.

import lightstream as ls

Files

Read Arrow IPC, Parquet, CSV and JSON through the same interface.

# Read an entire dataset.
table = ls.read("quotes.arrow").read_all()

# Stream batches without loading the complete dataset into memory.
for batch in ls.read("large.parquet"):
    process(batch)

# Write any Arrow-compatible object.
with ls.write("output.parquet", compression="zstd") as writer:
    writer.write(table)

Arrow IPC readers use memory mapping and out-of-core routing where appropriate.

Readers yield minarrow.Table batches. read_all() returns a Table when the result is contiguous or a ChunkedTable when multiple chunks must be preserved.

Network transports

The same interface streams raw Arrow IPC over TCP, Unix domain sockets, WebSocket, HTTP, QUIC, WebTransport and standard I/O.

for batch in ls.read("tcp://feed.example.com:9000"):
    process(batch)

Supported URI schemes include:

Transport URI
TCP tcp://host:port
Unix domain socket uds:///path/to/socket
WebSocket ws://host/path
Secure WebSocket wss://host/path
HTTP http://host/path
HTTPS https://host/path
QUIC quic://host:port
WebTransport wt://host/path
Standard input/output stdio://

Accepting connections

Either endpoint can accept the connection.

Set accept=True to bind the endpoint on first use and block until the connecting peer arrives.

writer = ls.write(
    "uds:///tmp/feed.sock",
    accept=True,
)

writer.write(table)
writer.close()

The listener remains available for the lifetime of the process. Connections arriving while a serving loop is active wait in the listener backlog, allowing the accepting endpoint to operate as a persistent server.

TLS

QUIC and WebTransport include TLS at the protocol level. The accepting endpoint presents a PEM certificate and private key, while the connecting endpoint verifies the certificate using PEM roots.

writer = ls.write(
    "quic://0.0.0.0:4433",
    accept=True,
    tls_cert="server.pem",
    tls_key="server-key.pem",
)

for batch in ls.read(
    "quic://feed.example.com:4433",
    tls_ca="roots.pem",
):
    process(batch)

Secure WebSocket and HTTPS transports use their corresponding TLS-enabled URI schemes.

Arrow interoperability

Every reader implements the Arrow PyCapsule stream protocol.

Arrow-compatible libraries can therefore consume a Lightstream reader directly.

import duckdb
import lightstream as ls

reader = ls.read("quotes.arrow")

result = duckdb.sql(
    "SELECT SUM(qty) FROM reader"
)

The same reader can be passed to libraries such as Polars without first converting its batches into Python objects.

Lightstream protocol

The Lightstream protocol multiplexes named Arrow tables and opaque messages over one connection.

Opaque payloads can carry formats such as Protobuf or MessagePack, while table channels retain their Arrow schema and batch representation.

reader = ls.read(
    "uds:///tmp/feed.sock",
    protocol="lightstream",
)

reader.register_table(
    "quotes",
    representative_table,
)
reader.register_message("health")

for frame in reader:
    if frame.is_table():
        on_quotes(frame.table)
    else:
        on_health(frame.payload)

frame.table is a minarrow.Table. Message payloads are returned as bytes.

Both peers must register compatible channels before exchanging frames.

The protocol is transport-independent, so the same table and message definitions can be used over TCP, Unix domain sockets, WebSocket, HTTP, QUIC or WebTransport.

Standard I/O pipelines

The standard I/O transport can stream Arrow IPC or line-oriented text.

Set format="csv" to write CSV records or format="json" to write NDJSON. Text emitted by another process is decoded back into table batches on read.

This allows Unix tools, command-line programs and agents to participate directly in a table pipeline.

python examples/run-example.py sed
python examples/run-example.py jq
python examples/run-example.py claude
python examples/run-example.py sql
python examples/run-example.py debug

The pipeline examples demonstrate:

  • sed rewriting rows in flight
  • jq filtering NDJSON and returning CSV
  • an agent diagnosing invalid batches
  • rolling DuckDB SQL over a live stream
  • tail -f-style table inspection and pretty-printing

Transport examples

Worked examples are provided under examples/transport/.

Each transport includes:

  • a Python server
  • a Rust server
  • a Python client
  • the same million-row input table

Use the example router to select the transport and backend:

# Python backend over TCP.
python examples/run-example.py tcp

# Rust backend over QUIC.
python examples/run-example.py quic --rust

Available transports are:

tcp
uds
ws
wss
http
https
quic
wt
stdio

TLS examples generate a private root certificate and a server certificate for local execution.

Building from source

Lightstream currently requires the Rust nightly toolchain. The repository selects the required toolchain automatically through rust-toolchain.toml.

pip install maturin
maturin develop
pytest tests/

Licence

Copyright © 2025–2026 Peter Garfield Bower.

Licensed under the Mozilla Public License 2.0. See LICENSE for the standard terms.

See MPL 2.0 FAQ if you are unfamiliar with this open-source license.

Affiliation notice

Lightstream is not affiliated with Apache Arrow or the Apache Software Foundation.

It implements public Arrow formats through Minarrow and interoperates with the Arrow ecosystem through the Arrow PyCapsule protocol.

lightstream is maintained by SpaceCell and forms part of its open-source foundation for high-performance data computing.

Project links

Data verified by PyPI on Jul 23, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.6.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for spacecell from gravatar.com spacecell

Credits

Author: Peter Bower

GitHub Statistics

Data verified by PyPI on Jul 23, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MPL-2.0
View SPDX License List

Requires

Python >=3.9

Provides Extra

dev

Tags

arrow streaming minarrow ipc parquet quic

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lightstream_io-0.5.0.tar.gz (596.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

lightstream_io-0.5.0-cp39-abi3-macosx_11_0_arm64.whl (3.1 MB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

lightstream_io-0.5.0-cp39-abi3-macosx_10_12_x86_64.whl (3.3 MB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

File details

Details for the file lightstream_io-0.5.0.tar.gz.

File metadata

  • Download URL: lightstream_io-0.5.0.tar.gz
  • Upload date:
  • Size: 596.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for lightstream_io-0.5.0.tar.gz
Algorithm Hash digest
SHA256 cb3614b08ea3b7314288867c25b2b7ef03dc099105a041f1360ab9a67ff0bda1
MD5 ac35ebb9e35104605919f330a85ca260
BLAKE2b-256 a526f198e0537ccc37d58cf5ca0a14040f9bbd34f98fb8a5d9bd96a688c6e8f7

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.5.0.tar.gz:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4e9fc31432892e68810384f57bcdd469a91175ff9b02860dd1662000d3c32488
MD5 f950c7e807284a43f9bb8ffc7cd8ac19
BLAKE2b-256 f3d396560c0ee4730bb9215c5036027b76fe00c1417d6b535247fb79788324f4

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 023c10c21265c58851f5438bd4b751a062d6cd465d2576f7746d5ee9a18b89d9
MD5 a0c349528d2a46c74b6bb4c2a7724ede
BLAKE2b-256 ce69a482d1e29049c95f070d67165e75753bbfde94c3863831c63ef30fea9a38

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.5.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.5.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.5.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 419a401cfbc2486a227572b841c2532eb35eb07340c50d8a1542ee299ebcc775
MD5 8a10c8ea63d807417dae9883bc6502d3
BLAKE2b-256 7b7f1ef9c27dbfde4c3f66c457ffd82f9a0c95e51614a3f85b070540aa26acf8

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.5.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file lightstream_io-0.5.0-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for lightstream_io-0.5.0-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 0c8fbeeceb164030373da0ff0a10edfdcebf73a50fcafdff53e1caabd50e9577
MD5 3e4db2da7d779c012b217b36ea1e3c90
BLAKE2b-256 7ce763810a73f7a7b4afa946cce8fa88879c9667f0d10754f502e2338a69b4d1

See more details on using hashes here.

Provenance

The following attestation bundles were made for lightstream_io-0.5.0-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on SpaceCell/lightstream

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page