nekojiru-binance 0.1.0

Binance public data sync CLI and Python API powered by Rust

binance-cli

A Rust CLI for downloading Binance public klines, mark-price-klines, index-price-klines, premium-index-klines, book-ticker, book-depth, liquidation-snapshot, metrics, trades, and agg-trades datasets, validating Binance .CHECKSUM files, converting the extracted CSV payload into csv or parquet, and writing the result through OpenDAL to local storage, S3, or Aliyun OSS.

Features

• Rust-first project structure with isolated modules for CLI, download, formatting, storage, and pipeline orchestration
• Binance klines, mark-price-klines, index-price-klines, premium-index-klines, book-ticker, book-depth, liquidation-snapshot, metrics, trades, and agg-trades downloads from https://data.binance.vision, with daily/monthly source files chosen automatically per dataset capability
• SHA-256 validation using the companion .CHECKSUM object
• Output format support for csv and parquet with parquet as the default
• OpenDAL-backed sinks for local filesystem, S3, and Aliyun OSS

Usage

cargo run -- sync klines \
  --market spot \
  --symbol BTCUSDT \
  --interval 1m \
  --start 2024-01-01 \
  --end 2024-03-31 \
  --dest /tmp/binance-data

List supported datasets:

cargo run -- list datasets

List available symbols for a market and dataset:

cargo run -- list symbols --market spot --dataset trades

Download spot trades to CSV:

cargo run -- sync trades \
  --market spot \
  --symbol BTCUSDT \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --format csv \
  --dest /tmp/binance-data

Download futures mark price klines to Parquet:

cargo run -- sync mark-price-klines \
  --market um \
  --symbol BTCUSDT \
  --interval 1m \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --dest /tmp/binance-data

Download futures index price klines to Parquet:

cargo run -- sync index-price-klines \
  --market um \
  --symbol BTCUSDT \
  --interval 1m \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --dest /tmp/binance-data

Download futures premium index klines to Parquet:

cargo run -- sync premium-index-klines \
  --market um \
  --symbol BTCUSDT \
  --interval 1m \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --dest /tmp/binance-data

Download futures book ticker snapshots to Parquet:

cargo run -- sync book-ticker \
  --market um \
  --symbol BTCUSDT \
  --start 2024-03-30 \
  --end 2024-03-30 \
  --dest /tmp/binance-data

Download futures metrics to Parquet:

cargo run -- sync metrics \
  --market um \
  --symbol BTCUSDT \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --dest /tmp/binance-data

Download COIN-M liquidation snapshots to Parquet:

cargo run -- sync liquidation-snapshot \
  --market cm \
  --symbol BTCUSD_PERP \
  --start 2024-03-30 \
  --end 2024-03-30 \
  --dest /tmp/binance-data

Download futures book depth snapshots to Parquet:

cargo run -- sync book-depth \
  --market um \
  --symbol BTCUSDT \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --dest /tmp/binance-data

Download spot aggregated trades to Parquet:

cargo run -- sync agg-trades \
  --market spot \
  --symbol BTCUSDT \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --dest /tmp/binance-data

Write to S3:

cargo run -- sync klines \
  --market spot \
  --symbol BTCUSDT \
  --interval 1h \
  --start 2024-01-01 \
  --end 2024-01-31 \
  --format parquet \
  --dest s3://my-market-data/binance/raw \
  --storage-opt region=us-east-1 \
  --storage-opt endpoint=https://s3.amazonaws.com

Write to Aliyun OSS:

cargo run -- sync klines \
  --market spot \
  --symbol BTCUSDT \
  --interval 1h \
  --start 2024-01-01 \
  --end 2024-01-31 \
  --format parquet \
  --dest oss://my-market-data/binance/raw \
  --storage-opt endpoint=https://oss-cn-hangzhou.aliyuncs.com \
  --storage-opt addressing_style=path

Download one day to local disk:

cargo run -- sync klines \
  --market spot \
  --symbol BTCUSDT \
  --interval 1m \
  --start 2026-04-01 \
  --end 2026-04-01 \
  --dest /tmp/binance-data

Output Layout

Each sync writes one file per partition under a dataset-specific prefix. Paths are shown wrapped for readability:

binance/klines/market={market}/symbol={symbol}/interval={interval}/
  year={YYYY}/month={MM}/part.{csv|parquet}
binance/mark-price-klines/market={market}/symbol={symbol}/
  interval={interval}/year={YYYY}/month={MM}/
  part.{csv|parquet}
binance/index-price-klines/market={market}/symbol={symbol}/
  interval={interval}/year={YYYY}/month={MM}/
  part.{csv|parquet}
binance/premium-index-klines/market={market}/symbol={symbol}/
  interval={interval}/year={YYYY}/month={MM}/
  part.{csv|parquet}
binance/metrics/market={market}/symbol={symbol}/year={YYYY}/
  month={MM}/part.{csv|parquet}
binance/book-depth/market={market}/symbol={symbol}/year={YYYY}/
  month={MM}/part.{csv|parquet}
binance/liquidation-snapshot/market={market}/symbol={symbol}/
  year={YYYY}/month={MM}/part.{csv|parquet}
binance/trades/market={market}/symbol={symbol}/year={YYYY}/
  month={MM}/day={DD}/part.{csv|parquet}
binance/agg-trades/market={market}/symbol={symbol}/year={YYYY}/
  month={MM}/day={DD}/part.{csv|parquet}
binance/book-ticker/market={market}/symbol={symbol}/year={YYYY}/
  month={MM}/day={DD}/part.{csv|parquet}

Notes

• --start and --end always use YYYY-MM-DD.
• The CLI chooses Binance monthly files for full months when a dataset publishes monthly archives; otherwise it stays on daily archives.
• Output partitioning is dataset-specific:
  • Monthly: klines, mark-price-klines, index-price-klines, premium-index-klines, metrics, book-depth, liquidation-snapshot.
  • Daily: trades, agg-trades, book-ticker.
• mark-price-klines currently supports futures markets only: um and cm.
• index-price-klines currently supports futures markets only: um and cm.
• premium-index-klines currently supports futures markets only: um and cm.
• book-ticker currently supports futures markets only: um and cm.
• book-depth currently supports futures markets only: um and cm.
• liquidation-snapshot currently supports COIN-M futures only: cm.
• metrics currently supports futures markets only: um and cm.
• As of April 11, 2026, observed futures book-ticker public objects for BTCUSDT were available through 2024-03-30; requests beyond published coverage will return 404.
• As of April 11, 2026, observed COIN-M liquidation-snapshot public objects for BTCUSD_PERP were available through 2024-10-14; requests beyond published coverage will return 404.
• --dest accepts an absolute local path, an s3://bucket/prefix URI, or an oss://bucket/prefix URI.
• --storage-opt key=value is the preferred way to pass backend-specific settings such as region, endpoint, or addressing_style.
• s3:// destinations still need a region, supplied by --storage-opt region=..., AWS_REGION, or AWS_DEFAULT_REGION.
• oss:// destinations need an endpoint, typically via --storage-opt endpoint=https://oss-cn-hangzhou.aliyuncs.com.
• Credentials are usually better supplied via environment variables, such as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for S3 and ALIBABA_CLOUD_ACCESS_KEY_ID / ALIBABA_CLOUD_ACCESS_KEY_SECRET for OSS.
• Use --help for static option values and list symbols --market ... --dataset ... for dynamic symbol discovery.
• Decimal-like Binance fields are preserved as strings to avoid precision loss during transformation.

Python Package (nekojiru-binance)

Install from PyPI:

pip install nekojiru-binance

Run CLI via Python package entrypoint:

nekojiru-binance sync klines \
  --market spot \
  --symbol BTCUSDT \
  --interval 1m \
  --start 2024-01-01 \
  --end 2024-01-02 \
  --dest /tmp/binance-data

Python API examples:

import nekojiru_binance as nb

print(nb.list_datasets())
print(nb.list_symbols(market="spot", dataset="trades"))

summary = nb.sync(
    dataset="klines",
    market="spot",
    symbol="BTCUSDT",
    interval="1m",
    start="2024-01-01",
    end="2024-01-02",
    dest="/tmp/binance-data",
    format="parquet",
)
print(summary)  # {'files_written': ..., 'rows_written': ...}

Python Build & Publish

Local build and develop install:

python -m pip install -U maturin
maturin develop --features python

Build wheel and sdist:

maturin build --release --features python --out dist
maturin sdist --out dist

Sanity check in a clean virtual environment:

python -m venv .venv
source .venv/bin/activate
pip install dist/*.whl
nekojiru-binance --help
python -c "import nekojiru_binance as nb; print(nb.list_datasets()[:3])"