lighter-data

Pull Lighter (zkLighter) market data into Parquet: funding history, candles, historical L2 book + trades — plus a live WebSocket recorder for the data no archive sells yet. Everything is free: public REST / WebSocket, no API key, no account.

Lighter is a young zk CLOB venue with 0%/0% maker/taker fees, perps and a small set of USDC spot markets. There is no official historical-data tooling: symbols are numeric market indices, history endpoints silently cap a single response (~750 funding rows, 500 candles), and the funding rate arrives as an unsigned percent-per-hour magnitude with the sign in a separate field. This package deals with all of that and writes tidy, manifest-checked Parquet.

Install

pip install lighter-data
# or, from a checkout:
pip install -e .

Python ≥3.10. Dependencies: pyarrow, websocket-client (the recorder); all HTTP is stdlib.

Quickstart

# full hourly funding history, per coin, paginated to the floor date
lighter-data funding --coins ETH SOL --root data

# hourly candles for the perp and the (young) USDC spot market
lighter-data candles --coins ETH --kinds perp spot --root data

# historical L2 + trades: Tardis.dev free first-of-month samples, no key
lighter-data tardis --coins XPL,EIGEN,JTO --dates 2026-05-01,2026-06-01 --root data

# live recorder: sampled top-20 book every 500ms + every trade
lighter-data record --coins WLD,ONDO,LINK --root data

--coins (and tardis --dates) accept space- and comma-separated lists interchangeably on every subcommand.

Datasets

Candle/funding partitions get a manifest.json sidecar (row count, bounds, cadence, duplicate timestamps, largest gaps), so a half-month hole from a broken resume shows up at download time, not as a silent bias in a backtest.

The funding sign, pinned

A Lighter funding row is {timestamp, value, rate, direction} where rate is a non-negative percent-per-hour magnitude and the sign lives in direction ("long" = longs pay shorts). The written rate column is the signed hourly fraction:

rate = rate_pct / 100 * (+1 if direction == "long" else -1)

verified live against value / index_price, and on the same scale as Hyperliquid's fundingRate — so cross-venue carry math needs no adapter. (Sister package: hyperliquid-data; the candle and funding schemas are identical across the two.)

Caps and pagination

A single wide request against Lighter's history endpoints does not error — it silently returns the most-recent slice (~750 funding rows, 500 candles), which looks like the venue's whole history. The pullers paginate fixed overlapping windows from a floor date (default 2025-01-01, before the first mainnet perp listing) and collapse the seam duplicates, so the written series reaches the true listing date.

The recorder

lighter-data record maintains each subscribed book from snapshot + deltas and samples the top-20 every 500 ms (~20x smaller than storing every delta, and the shape a fill-sim wants), records every trade with both counterparty account ids (Lighter's public feed exposes them — enough to build maker/taker cohorts offline), flushes Parquet parts every 10 min, and survives disconnects with backoff. It is a long-running service: run it under docker-compose/systemd with restart, not from a terminal you plan to close.

data/l2book/lighter/wld_perp/date=20260506/part-123005-000100.parquet
data/trades/lighter/wld_perp/date=20260506/part-123005-000100.parquet

Library use

from pathlib import Path

from lighter_data import resolve_perp_ids, fetch_all_funding, funding_rows, write_funding

mid = resolve_perp_ids()["ETH"]
pairs = fetch_all_funding(mid, start_s=1735689600, end_s=1754377200)
write_funding(funding_rows("ETH", pairs), root=Path("data"), instrument="ETH-PERP")

The supported surface is lighter_data.__all__ (pinned by tests/test_public_api.py); schemas are typed in lighter_data.types and pinned as pyarrow schemas in lighter_data.parquet.

License

MIT.