hyperliquid-data

Pull Hyperliquid history into Parquet: candles, funding, L2 book, trades, fills, liquidations. Price the S3 egress bill before you run the pull.

Both Hyperliquid S3 buckets are requester-pays. fills alone is 0.8–1.0 GiB per day of all-coin data, so a casual one-year pull is a three-figure AWS bill you find out about at the end of the month. hl-data cost lists the exact prefixes a pull would touch and prints the number first.

$ hl-data cost --dataset fills --start 2026-06-01 --end 2026-06-30
dataset      fills
range        30 day(s), 3 sampled
per day      0.820 GiB  (24 objects)
TOTAL        24.61 GiB  (720 objects)
EGRESS COST  ~$2.21  (@ $0.09/GB, requester-pays)

samples:
  2026-06-01  0.960 GiB  24 obj
  2026-06-15  0.856 GiB  24 obj
  2026-06-30  0.644 GiB  24 obj

note: extrapolated from 3 sampled day(s) (min 0.644 / max 0.960 GiB). Actual volume varies with market activity.

The estimator and the pullers read the same prefix map (prefixes.py), so they cannot disagree about which objects a date lives under. That is not a hypothetical worry: the fill stream changed format in mid-2025, and a hardcoded prefix under-prices any range that crosses the handoff.

Why this exists

Install

pip install hyperliquid-data
# or, from a checkout:
pip install -e .

Python ≥3.10, and nothing outside the wheel. Archives are decompressed in-process by the lz4 package, streaming: no decompressed copy ever touches disk, and there is no external binary to install.

Quickstart

# free: public REST, no AWS account
hl-data candles all --coin BTC --interval 1h --start 2024-01-01 \
    --raw-dir raw --root data
hl-data funding pull --coins BTC ETH HYPE --root data

# requester-pays: size it first, then pull
hl-data cost   --dataset l2book --start 2026-06-01 --end 2026-06-07 --coins BTC
hl-data l2book pull --coins BTC --start 2026-06-01 --end 2026-06-07 --root data

Datasets

funding, l2book and liquidations take a subcommand (pull, plus probe or list where those make sense); candles takes candles, funding, ingest or all. trades and fills take flags directly. hl-data <command> --help spells out either one. Everywhere --coins appears it accepts both spellings: --coins BTC ETH and --coins BTC,ETH.

Every REST partition (candles, funding) gets a sibling manifest.json: row count, first and last timestamp, duplicates, out-of-order rows, and the gaps against the expected grid. A backtest can assert its coverage instead of trusting the file. The S3 datasets write Parquet without one.

Two things worth knowing before you pick a coin list:

• Coin names are case-sensitive, exactly as Hyperliquid spells them: kPEPE, kBONK, not KPEPE. This is checked, not assumed. Upper-casing them used to make l2book a silent no-op and trades/fills download a full day for nothing.
• fills and liquidations are all-coin streams. A coin filter narrows what is written, never what is downloaded — the bill is the same for one coin as for two hundred. trades fetches each day once and fans out to every requested coin in that pass, so adding coins there is free too.

Credentials

candles and funding need none. They read the public REST endpoint. Only the four S3 datasets need AWS credentials, and only ever for reading.

Nothing is passed to boto3 explicitly (there is no aws_access_key_id= anywhere in this package), so a key never travels through a function argument, a log line or a traceback. Resolution is boto3's own, in this order:

1. AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ AWS_SESSION_TOKEN)
2. AWS_PROFILE → ~/.aws/credentials
3. ~/.aws/config, including SSO and assume-role profiles
4. Container credentials / EC2 instance profile

Where to keep them

Best: nowhere this tool can see. Use a named profile or SSO and let the AWS CLI own the secret:

aws configure --profile hl          # or: aws configure sso --profile hl
AWS_PROFILE=hl hl-data fills --start 2026-06-01 --root data

Or a dotenv, if you want per-project isolation. $HL_DATA_ENV if set, else ./.env in the current working directory:

# .env  — never commit this file
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1

HL_DATA_ENV=~/secrets/hl.env hl-data l2book pull --coins BTC --start 2026-06-01

The file is loaded with setdefault, so a shell-exported value always beats the file: a stale .env cannot silently override the profile you meant to use. A missing file is not an error. There is no baked-in path, and nothing is written back.

Add .env to your .gitignore before you write one. A committed key is a key you have to rotate, and this one can be charged against.

Minimum IAM policy

Read-only on the two buckets is enough. Requester-pays needs no special permission of its own. It needs your policy to allow the call, and the header this package always sends (RequestPayer: requester) is what moves the bill to you:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:ListBucket"],
    "Resource": [
      "arn:aws:s3:::hyperliquid-archive",
      "arn:aws:s3:::hyperliquid-archive/*",
      "arn:aws:s3:::hl-mainnet-node-data",
      "arn:aws:s3:::hl-mainnet-node-data/*"
    ]
  }]
}

No write, no delete, no other bucket. If the key leaks the exposure is a bill, not your data, which is a good reason not to reuse a general-purpose key here.

hl-data l2book probe and hl-data liquidations probe print the AWS account the credentials resolve to, so you can check which account is about to be charged before you pull.

Typed API

The package ships py.typed, so the public surface type-checks in a consumer's project. Every written schema has a TypedDict mirroring its Arrow columns:

from hyperliquid_data import FillRow, estimate, write_funding

est = estimate("fills", "2026-06-01", "2026-06-30")
est.total_bytes / 1024**3     # -> float, Estimate is a dataclass

def pnl_by_wallet(rows: list[FillRow]) -> dict[str, float]:
    ...                        # row["side"] narrows to Literal["B", "A"]

Called as a library, estimate() uses whatever AWS configuration the process already has. The dotenv above is loaded by the CLI, not on import; call hyperliquid_data.load_env() yourself if you want it.

BookLevel, CandleRow, FundingRow, TradeRow, FillRow, L2SnapshotRow, LiquidationBucketRow, plus the Dataset and Side literals, are all re-exported from the package root. A test asserts the TypedDict keys stay in lockstep with the Arrow schemas, so the types cannot silently drift from the files.

What counts as public

Whatever hyperliquid_data.__all__ lists — the schemas above, estimate, the parquet writers and partition paths, the date and naming helpers, and the universe/funding reference calls (hl_universe, funding_first_ts, binance_onboard_map, …). That set is pinned by a test, so it cannot be dropped or renamed by accident.

Anything else — importing a submodule and taking a name off it, especially a _-prefixed one — is internal and can move in a patch release. If you need something that is not exported, open an issue rather than reaching in; that coupling is exactly what this list exists to prevent.

Gotchas this bakes in

Each of these cost a wrong pull to discover.

• node_trades is dead. The prefix exists and returns empty files. Trades live in node_fills_by_block/hourly/<YYYYMMDD>/<H>.lz4, one JSONL line per block envelope {"events": [[address, fill], …]}, all coins in one stream.
• The fill stream changed format on 2025-07-27. node_fills/hourly holds 2025-05-25 up to that date and matches the public API shape; node_fills_by_block/hourly holds it onward and is the current one. Pullers and estimator both resolve the prefix per date. Assuming a single prefix under-reports a range that crosses the handoff by a third, and a range entirely before it does not resolve at all.
• Every trade appears twice, taker (crossed=true) and maker (crossed=false), sharing a tid. trades keeps the taker row, since it carries the aggressor side (side="B" lifts asks, "A" hits bids); fills keeps both, plus the wallet.
• liquidations writes one aggregated file per coin, not one per date, so a re-run has to derive its coverage from the timestamps already on disk. Otherwise extending a six-month dataset by a week re-buys the six months.
• fundingHistory is not months-capped. With startTime=0 it paginates forward 500 rows at a time from the coin's true listing date (BTC → 2023-05-12, HL mainnet launch). Funding needs no S3 backfill.
• Duplicate funding settlements from a resumed pull would be double-counted by any downstream resample().sum(), so the writer collapses them, last wins.
• L2 snapshots are event-driven (~550 ms cadence, ~1.8/s on an active alt), not a fixed grid. time is node wall-clock at ns precision; the nested data.time is exchange time in ms.
• Parquet timestamps are timestamp[us, UTC] everywhere, so mixing sources never silently compares ms against ns.

Status

Alpha. The public API is pinned (see CHANGELOG), but breaking changes are still possible before 1.0.

Not included: HIP-3 equity-perp carry pulls. They need a Yahoo hedge leg, which is out of scope for a Hyperliquid data library.

Development

pip install -e ".[dev]"
pytest
pytest --cov=hyperliquid_data --cov-report=term-missing  # 88%, CI floor 80%
ruff check src tests
mypy

Tests are offline. S3 and the REST endpoint are stubbed, so the suite never spends egress. Coverage is enforced at 80% in CI, and the paths that decide what a pull costs and what the data says sit well above that, which is the point of the number.

License

MIT.