Polaris SDKs

The official Rust, Python, and TypeScript SDKs for the Polaris API. Rust and Python share one Rust engine; TypeScript is an independent Node.js and browser package. All three distributions are named polaris-data, with Python importing as polaris_data.

Documentation can be found at https://polaris.supply/docs

Install

Install the Python SDK from PyPI:

pip install polaris-data

If you use uv, install it into a project with:

uv add polaris-data

Or install it into the active environment with:

uv pip install polaris-data

Install optional notebook and Arrow support with:

pip install "polaris-data[dataframe]"  # Pandas + PyArrow
pip install "polaris-data[arrow]"      # PyArrow batches only

Install the Rust SDK from crates.io:

cargo add polaris-data

Install the TypeScript SDK from npm:

npm install polaris-data

Python wheels always include the Rust core. CPython 3.9+ is supported through PyO3's stable ABI; there is no pure-Python runtime fallback.

Quickstart

from polaris_data import PolarisClient

with PolarisClient(api_key="polaris_key_your_key") as client:
    row_count = sum(
        1
        for _ in client.replay(
            source="binance",
            market="BTC-USDT",
            from_="2024-01-01T00:00:00Z",
            to="2024-01-01T01:00:00Z",
        )
    )
    print(f"Replayed {row_count} rows")

If api_key is omitted, the client reads POLARIS_API_KEY from the environment.

The equivalent async Rust workflow is:

use futures_util::StreamExt;
use polaris_data::{PolarisClient, ReplayQuery};

#[tokio::main]
async fn main() -> Result<(), polaris_data::PolarisError> {
    let client = PolarisClient::builder().build()?;
    let mut rows = client
        .replay(ReplayQuery {
            source: "binance".into(),
            market: "BTC-USDT".into(),
            from: Some("2024-01-01T00:00:00Z".into()),
            to: Some("2024-01-01T01:00:00Z".into()),
            allow_gaps: false,
            materialize_orderbooks: true,
        })
        .await?;

    while let Some(row) = rows.next().await {
        println!("{:?}", row?);
    }
    Ok(())
}

For synchronous Rust applications use polaris_data::blocking::PolarisClient. It owns a Tokio runtime and returns PolarisError::BlockingInAsyncRuntime when called from an active Tokio runtime, instead of panicking.

Realtime streams

stream(...) opens an unbounded WebSocket feed of the same standardized event shape returned by replay(...). A stream covers one source and up to 1,000 markets, reconnects automatically after transport failures, and closes when its iterator is dropped or explicitly closed.

from polaris_data import PolarisClient

with PolarisClient(api_key="polaris_key_your_key") as client:
    with client.stream(source="binance", markets=["BTC-USDT", "ETH-USDT"]) as events:
        for event in events:
            print(event)

The equivalent async Rust workflow is:

use futures_util::StreamExt;
use polaris_data::{PolarisClient, StreamQuery};

#[tokio::main]
async fn main() -> Result<(), polaris_data::PolarisError> {
    let client = PolarisClient::builder().build()?;
    let mut events = client.stream(StreamQuery {
        source: "binance".into(),
        markets: vec!["BTC-USDT".into(), "ETH-USDT".into()],
        include_buffer: false,
        materialize_orderbooks: true,
    }).await?;

    while let Some(event) = events.next().await {
        println!("{:?}", event?);
    }
    Ok(())
}

Orderbooks are materialized by default. A standardized orderbook event replaces the complete book; each orderbook_delta updates only its listed prices, and a zero quantity deletes that price. Materialized output is relabeled orderbook and uses sorted {price, quantity} levels. Set materialize_orderbooks=False (Python), materialize_orderbooks: false (Rust), or materializeOrderbooks: false (TypeScript) to receive raw deltas.

Reconnection is best-effort: the current live protocol has no resume cursor, so a reconnect can introduce a gap or duplicate event. The SDK clears reconstructed books on reconnect and suppresses later deltas until a new snapshot arrives. Protocol and authentication errors are terminal and are not retried.

Use l2_updates() in Python and Rust or l2Updates() in TypeScript to read the initial snapshots and sparse deltas without reconstructing every intermediate book. Reusable OrderbookBuilder exports in all three SDKs let applications materialize those updates when needed:

from polaris_data import OrderbookBuilder

books = OrderbookBuilder()
books.update(snapshot)
books.update(delta)  # False until a snapshot initializes this book
complete = books.snapshot("lighter", "BTC-USD")
books.clear_book("lighter", "BTC-USD")

update() mutates book state without constructing a full result. Call snapshot() only when you need sorted levels. The existing apply() method remains available as a compatibility shortcut that performs both operations.

PolarisClient API

PolarisClient is the main sync client for the SDK:

PolarisClient(
    api_key=None,
    base_url="https://api.polaris.supply",
    timeout=30.0,
    dataset_root=None,
    stream_url=None,
)

Use it to inspect available data, query historical market data, and open realtime streams.

Discovery

Access patterns

Standardized Data Schemas

Historical row methods are single-pass iterators. Iterate them directly for bounded memory, or call list(...) when you intentionally want an eager result. Setup and coverage errors occur when the method is called; decode errors can occur later while iterating. If you stop early, call the generator's close() method to promptly release its native reader. bbo(interval="1s") emits the last quote from each non-empty, UTC-aligned interval.

Standardized replay automatically prefetches and decompresses one subsequent snapshot file on a bounded background worker while preserving file and row order. The legacy parallel argument remains accepted for compatibility; raw replay still uses its historical 24-hour chunking behavior.

The columnar methods above accept output="batches" for a bounded iterator of pyarrow.RecordBatch objects or output="dataframe" for an eager Pandas DataFrame. Typed-series output flattens fields, uses UTC millisecond timestamps, and dictionary-encodes source, market, and side. Venue-specific trade and point fields appear as sorted extra.<name> columns; discovering those fields requires one schema pass before batches are emitted.

Exact events() and standardized replay() batches use UTC microsecond timestamps and preserve storage order without timestamp sorting. Every row has stable replay_ordinal, source_file_ordinal, and source_row_ordinal fields, plus nullable receive timestamp, sequence, and sequence scope fields. Common trade, order-book, and point payloads have typed columns. With raw order-book updates, event_json preserves the complete source event, including unknown event types and venue-specific fields; with materialization enabled it contains the resulting complete-book event. Use materialize_orderbooks=False for execution replay so batches contain the initial snapshot and every ordered delta.

For parameter details, response shapes, and end-to-end examples, see the Python SDK docs.

Benchmarks

Streaming and memory

Run the opt-in end-to-end benchmark after building the Python extension:

uv run python benchmarks/streaming_memory.py

It generates a 3,000-level local book with one million deltas, consumes raw standardized events, direct BBO, raw L2 updates, and lazy application-managed books in isolated processes, and reports end-to-end wall time, rows per second, and peak RSS. The command fails when peak RSS from 100,000 to one million deltas grows by more than the larger of 20% or 64 MiB, or when long-run throughput falls below 75% of short-run throughput.

Optionally set machine-specific throughput floors:

uv run python benchmarks/streaming_memory.py \
  --min-rps events=50000 --min-rps bbo=100000 \
  --min-rps l2_updates=500000 --min-rps l2_builder=250000

Compare iterator, native RecordBatch, and native DataFrame paths over the 788,383-trade fixture with:

uv run python benchmarks/columnar.py

Compare exact dictionary replay with microsecond Arrow batches across trade, point, shallow-book, and deep-book fixtures using explicitly page-cache-warm samples and recorded hardware metadata with:

uv run python benchmarks/exact_events.py --events 788383 --runs 3

Evaluate first-use Arrow IPC construction, disk amplification, and repeated memory-mapped scans without enabling a persistent SDK cache with:

uv run python benchmarks/replay_cache_prototype.py --events 788383

The standardized IPC cache is deliberately not enabled by the SDK yet. The prototype reports cache-build cost, compressed-source versus IPC disk size, and warm scan throughput so a persistent format can be chosen against real venue datasets. Adoption also requires versioned cache keys, atomic multi-process construction, corruption recovery, invalidation, and an eviction policy.

The benchmark runs each mode in a separate process and reports elapsed time, throughput, peak and incremental RSS, row count, and a price checksum. It does not enforce machine-specific performance thresholds. Its list-json-normalize mode is a worst-case convenience pattern: it first materializes every nested row dictionary with list(client.trades(...)), then calls pandas.json_normalize(...). The resulting peak RSS includes the Python row objects and Pandas conversion temporaries, not just the final DataFrame.

Full-book materialization remains available as an explicitly scaled benchmark:

uv run python benchmarks/streaming_memory.py --modes l2 \
  --short-deltas 100 --long-deltas 1000

Absolute throughput floors are intentionally opt-in because results vary by hardware and build profile.

Local event replay

Use the focused local replay benchmark to compare direct zstd+orjson decoding with events() and replay():

uv run python benchmarks/local_replay.py \
  --fixture trade --events 788383

The benchmark warms the filesystem cache, runs each path in an isolated process, and reports iterator construction time, time to first event, steady-state throughput, and peak RSS growth. Pass --enforce-targets to require the SDK to deliver the first event in under 10 ms, process at least 500,000 events/s, and stay within 2× of direct zstd+orjson.

Reference results

The streaming and memory results below came from a local development build. The raw modes used the default million-update scale; materialized L2 used the explicit 1,000-update scale shown above. The local replay results came from one Apple Silicon macOS release run using the synthetic 788,383-event UNI-sized trade fixture. The exact-event results are three-run medians from page-cache-warm 200,000-event trade, point, shallow-book, and deep-book fixtures on Apple Silicon macOS.

In these runs, events() was 1.94× the direct decoder time while exceeding the 500,000 events/s target by more than 2×. Exact Arrow batches were 1.35–1.41× faster than the per-event Python dictionary iterator across the four fixture shapes. Construction and first-event latency were reported separately; together they remained under 0.8 ms for events().

Local dataset storage

Standardized snapshots are stored under the shared Polaris app-data root so the Python SDK and CLI can reuse the same files. Legacy materialized day files are also recognized when present.

Default roots:

• macOS: ~/Library/Application Support/polaris
• Linux: $XDG_DATA_HOME/polaris or ~/.local/share/polaris
• Windows: %APPDATA%\\polaris

Within that root, the SDK uses the same layout as the CLI:

<root>/
  data/
  daily/
  tmp/
  cache/
  locks/

Standardized snapshot downloads are stored under:

<root>/data/<tier>/<source>/<market>/<YYYY-MM-DD>/<opaque-key>.jsonl.zst

When the snapshot service provides authoritative bounds, the SDK stores them in an atomic <opaque-key>.jsonl.zst.coverage.json sidecar. Explicitly bounded replays whose local files cover the requested interval do not perform a remote coverage lookup. Older caches without sidecars remain readable using estimated filename coverage and emit a warning until exact metadata is available.

The opaque key is the flat upstream snapshot identifier, for example:

standard-aster-ASTERUSDT-2026-06-01-00

which is stored on disk as:

<root>/data/standard/aster/ASTERUSDT/2026-06-01/standard-aster-ASTERUSDT-2026-06-01-00.jsonl.zst

Compatible materialized day files, when present, are stored under:

<root>/daily/<source>/<market>/<YYYY-MM-DD>.jsonl.zst

Pass dataset_root=... to PolarisClient(...) to override the root explicitly. POLARIS_ROOT overrides the shared root globally. POLARIS_DATASET_DOWNLOAD_DIR is still accepted as a deprecated compatibility override.

Snapshot-first replay

For standardized historical data, replay(...), events(...), trades(...), vwap(...), volatility(...), bbo(...), depth_metrics(...), l2_snapshots(...), l2_updates(...), volume(...), and default/tradingview ohlcv(...) now prefer /snapshots plus daily bulk /download?source=...&market=...&date=...&mode=json manifests, and reuse local snapshot files when they already exist:

from polaris_data import PolarisClient

with PolarisClient(api_key="polaris_key_your_key") as client:
    for row in client.replay(
        source="binance",
        market="BTC-USDT",
        from_="2024-01-01T00:00:00Z",
        to="2024-01-01T01:00:00Z",
    ):
        print(row)

If the requested standardized range cannot be satisfied from available standardized snapshots, replay(...), events(...), trades(...), vwap(...), volatility(...), bbo(...), depth_metrics(...), l2_snapshots(...), l2_updates(...), volume(...), and ohlcv(...) raise by default instead of falling back. Pass allow_gaps=True on standardized methods to return only covered data and receive a warning with the missing intervals.

Error handling

from polaris_data import PolarisClient, RateLimitedError, UnauthorizedError

client = PolarisClient()

try:
    client.replay(
        source="binance",
        market="BTC-USDT",
        from_="2024-01-01T00:00:00Z",
        to="2024-01-01T01:00:00Z",
    )
except UnauthorizedError:
    print("API key is required")
except RateLimitedError as err:
    print(f"Rate limited. Reset at: {err.reset_at}")

Tests

uv run pytest
cargo test --workspace
cd typescript && npm ci && npm run typecheck && npm test

Build and inspect the native Python wheel with:

uv run --with maturin maturin build --release

Python, Rust, and TypeScript are versioned independently. Python releases use python-vX.Y.Z tags and publish polaris-data to PyPI; Rust releases use rust-vX.Y.Z tags and publish polaris-data to crates.io; TypeScript releases use typescript-vX.Y.Z tags and publish polaris-data to npm.