HdW Crypto Data

Utilities for acquiring Binance and Coinbase Exchange candles, verifying local archives, building provider-aware total datasets, and loading crypto time series for research and analysis.

The core package is intentionally kept lightweight. The Archive Manager is an optional package module; the showcase app and technical-analysis chart helpers live in examples/. Heavy dependencies are imported only when those tools are used.

Repository: https://github.com/hansdeweme/HdWCryptoData

Version 0.7.0 adds Coinbase acquisition, archives, totals and Showcase integration. Version 0.6.1 fixes mixed timestamp units when merging archive and REST candles. Version 0.6.0 adds the lightweight, in-memory BinanceKlineClient. Interval-aware total datasets and strict research loading are available since 0.5.0. The local Crypto Archive Manager and shared SQLite archive index remain available. See the changelog for release details. The Showcase App opens standalone HTML charts and logs chart paths and calculation errors to the terminal.

What It Does

• downloads historical Binance Vision kline CSV files
• verifies downloaded archives with Binance checksum files
• stores monthly and daily spot kline data in the Binance Vision folder layout
• merges historical files with recent live Binance candle data
• writes a self-documenting total dataset name, for example BONKUSDT-spot-1h-total-2026-12-22T00-00-00Z--2026-12-22T23-00-00Z.csv
• loads exact UTC observations for research without filling missing candles
• retains cleaned pandas DataFrame loading with timezone handling and optional gap filling for visualization

Core Modules

• binance_vision_dumper.py provides BinanceVisionDumper for downloading historical Binance Vision data.
• binance_vision_client.py contains safe Binance Vision HTTP, retry, URL validation, and checksum helpers.
• binance_rest_client.py provides BinanceRestClient for recent Binance REST API candles.
• total_dataset_builder.py provides TotalDatasetBuilder for building total CSV datasets.
• total_dataset_loader.py provides TotalDatasetLoader for loading and normalizing total CSV files.
• symbols.py provides symbol normalization helpers.

Public imports are available from the package root:

from hdw_crypto_data import BinanceVisionDumper, TotalDatasetBuilder, TotalDatasetLoader

Coinbase Exchange (0.7.0)

Use explicit product IDs such as HNT-USD; Coinbase archive/research workflows support 1h and 15m. The client pages in windows of at most 300 candles, paces requests and uses bounded transport/server/rate-limit retries.

from hdw_crypto_data import CoinbaseClient, CoinbaseArchiveProvisioner
from hdw_crypto_data import TotalDatasetBuilder, TotalDatasetLoader, resolve_market

datastore = "./datastore"
cutoff = "2026-10-02T00:00:00Z"
identity = resolve_market("HNT-USD", provider="coinbase", interval="1h")
with CoinbaseClient() as client:
    candles = client.get_candles(identity.product, interval="1h",
        start="2026-10-01T00:00:00Z", end=cutoff, as_of=cutoff)
    archives = CoinbaseArchiveProvisioner(datastore, client=client)
    partitions = archives.provision(identity.product, interval="1h",
        start="2026-10-01T00:00:00Z", end=cutoff, as_of=cutoff)
    result = TotalDatasetBuilder(identity.product, {"datastore_root": datastore},
        provider="coinbase", interval="1h", as_of=cutoff,
        recent_source=client, current_dir="./totals").build()

loader = TotalDatasetLoader(identity.product, {}, provider="coinbase", interval="1h")
frame = loader.load_research_dataframe(result.filepath)

Provisioning is a separate explicit operation. It stores completed UTC months and days under <datastore>/coinbase/spot/{monthly,daily}/klines/<product>/<interval>/, separate from Binance archives. Current UTC days are excluded. Repeating a request resumes unverified windows; repair=True explicitly refreshes selected partitions. Each CSV has a required .csv.metadata.json sidecar with coverage, provenance and SHA-256. Keep the sidecar when copying archives or total CSVs. inspect_partition(path) reports verification and coverage without creating an index.

Coinbase total filenames begin with coinbase__. Construction verifies archives, merges a bounded recent handoff (at most 500 interval buckets from the latest archived observation to the cutoff), and rejects conflicting OHLCV overlaps. Provision newer archives first if that bound is exceeded. Loading preserves the 12-column CSV contract, with unavailable auxiliary measurements left missing. Gaps are not filled or interpolated; candle cutoffs exclude unfinished candles.

Coinbase may omit buckets without trades. Missing observations are not proof of no trading and may prevent research calculations requiring complete coverage. Explicit historical cutoffs select candle times; acquisition timestamps do not prove that those observations were available historically. API interval support differs by provider: Binance research supports 15m, 1h, 1d, while the lightweight Binance client supports additional exchange intervals.

The Showcase's Coinbase crypto source provides separate archive provisioning and total-dataset actions. Run python -m examples.showcase_pyqt_app after installing examples/requirements.txt. A bounded live integration check is python -m hdw_crypto_data.coinbase_smoke --totals; it uses disposable storage.

Existing Binance public exports and defaults remain available. Binance load_research_dataframe() still defaults to 1h, even when settings select a different frequency; pass interval= explicitly to override it. Coinbase's omitted research interval follows the loader's configured interval. Legacy Binance loader construction defers research-frequency validation. Binance overlaps now use deterministic recent > daily > monthly priority, retain the first source at that priority and report conflicting OHLCV as warnings.

Downstream Weekly Pattern applications using these Coinbase APIs must require hdw-crypto-data>=0.7.0; the older >=0.5.0 floor does not provide them.

Strict Research Data (0.5.0)

Use an explicit interval (15m, 1h or 1d) when building or locating total research datasets. The legacy builder default remains 1h.

from hdw_crypto_data import TotalDatasetBuilder, TotalDatasetLoader

# Explicit update: build a total dataset and omit the open live candle.
result = TotalDatasetBuilder("BONK", settings, interval="15m").build(
    open_candle="exclude")

# Exact observations: no sorting, deduplication, interpolation or side files.
loader = TotalDatasetLoader("BONK", settings, current_dir=total_directory)
frame = loader.load_research_dataframe(
    file_path=result.filepath, interval="15m", preferred_tz="UTC")

# Read-only: discover an existing hourly total without building/updating it.
hourly = loader.load_research_dataframe(interval="1h", preferred_tz="UTC")

For Binance, the strict loader preserves internal gaps, duplicates, row order and malformed values; callers must validate observations before analyzing them. It parses second, millisecond and microsecond timestamps in their original units. Use load_total_dataframe() for the existing optional cleaning/visualization path.

Recent candles in memory

BinanceKlineClient provides recent Spot data without archive downloads, temporary CSV files, logging, GUI imports or technical indicators:

from hdw_crypto_data import BinanceKlineClient

with BinanceKlineClient() as client:
    df = client.get_klines("PEPEUSDT", interval="5m", limit=500)

The result has a sorted, unique UTC DatetimeIndex named dt and exactly six numeric columns: open, high, low, close, volume (float64), and number_of_trades (int64). Candle timestamps identify the candle's open time. Identical duplicate timestamps collapse; conflicting duplicates, invalid numeric values and inconsistent OHLC bounds raise BinanceKlineResponseError. Missing candles are not filled or interpolated. Short/new listings can return fewer rows (including an empty, correctly typed frame); callers enforce their own minimum lookback requirements.

get_klines(symbol, *, interval="1h", limit=500, closed_only=True) accepts full Binance pair symbols and limits from 1 through 1000. Symbols are stripped and uppercased; intervals are case-sensitive (1m and 1M are different). Spot intervals include 1s, 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 8h, 12h, 1d, 3d, 1w and 1M. The archive/research APIs retain their existing interval support and behavior.

Completed candles are the default: Binance's inclusive close time must be strictly earlier than the UTC millisecond cutoff frozen before the call's first request. The request's endTime also bounds the snapshot. The client requests an extra candle and, when necessary at the 1000-row API ceiling, pages backward to preserve the requested completed history. This conservative rule does not admit a partial snapshot merely because the candle closes during retries or response processing. It relies on an accurate system clock. Use closed_only=False to explicitly include a possibly unfinished current candle. Returned df.attrs records symbol, interval, closed_only and retrieval_cutoff_ms.

Constructor options are timeout=(5, 15) (connect/read seconds), max_attempts=3, min_request_interval=0.25, backoff_factor=0.5, max_retry_wait=30.0 and an optional caller-owned requests-compatible session. One instance serializes calls, reuses its HTTP session, and spaces request starts. This is per-instance pacing; applications sharing an IP still coordinate their overall API usage. Transport failures and HTTP 5xx responses use bounded exponential backoff. HTTP 429 respects the full Retry-After cooldown; a long cooldown or HTTP 418 raises BinanceRateLimitError immediately, carrying retry_after (seconds) and status_code when supplied by the response. The instance remembers the cooldown across calls and never retries before it expires. Other HTTP failures raise BinanceKlineError without retrying; malformed data or JSON raises BinanceKlineResponseError. All three errors are public root exports. The client closes its own session via close() or the context manager and does not close an injected session. Closed clients cannot issue further requests.

Offline checks: py -3.13 -m unittest discover -s tests -p test_binance_kline_client.py -v. API references: Spot klines and rate limits.

Installation

The core package supports Python 3.11 or newer. Python 3.13 is recommended for the optional chart dependencies. From the repository root:

py -3.13 -m venv .venv
.\.venv\Scripts\Activate.ps1
py -m pip install hdw-crypto-data
# Required when running the showcase and chart examples:
py -m pip install -r examples/requirements.txt

For development, replace py -m pip install hdw-crypto-data with py -m pip install -e .; also install examples/requirements.txt to run the examples. The examples require a source checkout because they are not included in the installed wheel.

The examples requirements install hdw-ta (imported as hdw_ta) for the Keltner calculations, plus PyQt6, Plotly, pandas-ta, ta, SciPy, progressbar2, and openpyxl for the GUI, charts, and Excel export. NumPy and pandas are installed by the core package.

Folder Layout

Typical repository structure:

HdWCryptoData/
  hdw_crypto_data/
    __init__.py
    version.py
    binance_vision_client.py
    binance_vision_dumper.py
    binance_rest_client.py
    total_dataset_builder.py
    total_dataset_loader.py
    symbols.py
    archive_paths.py
    archive_index.py
    archive_manager.py
    archive_manager_gui.py
    stylesheet.py
  examples/
    requirements.txt
    binance_dump.py
    settings.json
    showcase_pyqt_app.py
    showcase_sources.py
    showcase_ui.py
    stylesheet.py            # imports the shared package theme
    ta_charts.py
  tests/
    smoke_test.py
    test_*.py
  pyproject.toml
  requirements.txt
  readme.md
  changelog.md
  archive_manager.md
  archive_index.md

Typical Binance Vision data layout under full_spot:

spot/
  monthly/klines/BONKUSDT/1h/*.csv
  daily/klines/BONKUSDT/1h/*.csv

Optional Tools

• examples/binance_dump.py is a command-line runner for BinanceVisionDumper.
• python -m examples.showcase_pyqt_app launches the PyQt6 showcase from the repository root.
• examples.ta_charts contains the optional analysis and Plotly helpers.
• python -m hdw_crypto_data.archive_manager_gui launches the Archive Manager.

Install examples/requirements.txt for the showcase and chart helpers. For the Archive Manager alone, install the gui extra with py -m pip install "hdw-crypto-data[gui]". The showcase and chart helpers require a source checkout; they are not included in the installed wheel.

Settings

The download script and showcase use examples/settings.json by default. The showcase resolves relative paths against the repository root, independently of the working directory. Local icon folders default to examples/stock_icons and examples/crypto_icons; either folder may be absent. The Archive Manager checks the working directory for its own settings file. The showcase requires a source checkout.

Portable showcase settings:

{
  "spot": ".",
  "full_spot": "./spot",
  "stock_icons": "examples/stock_icons",
  "crypto_icons": "examples/crypto_icons",
  "quote_currency": "USDT",
  "preferred_time_zone": "Europe/Amsterdam"
}

full_spot should point to the directory containing the spot data tree. The dumper handles both base/spot/... and base/... layouts where possible. open_candle controls recent REST candles: exclude omits unfinished live candles, while include keeps them for dashboards or live inspection.

Basic Usage

Download Binance Vision data:

from hdw_crypto_data import BinanceVisionDumper

dumper = BinanceVisionDumper(
    path_dir_where_to_dump=r"D:\Coding\forecast\spot",
    asset_class="spot",
    data_type="klines",
    data_frequency="1h",
)
dumper.dump_data(tickers=["BONKUSDT"])
dumper.delete_outdated_daily_results()

Missing Binance archives are not always failures. New listings, inactive symbols, and dates before a market existed can legitimately return 404. The dumper reports not_found separately from network errors, rate limits, checksum failures, and invalid archives. If operational failures occur, BinanceVisionDumpError.failures contains the per-file ArchiveDownloadResult values with date, status, and optional error.

Build a total dataset:

from hdw_crypto_data import BinanceRestClient, BinanceVisionDumper, TotalDatasetBuilder

vision_dumper = BinanceVisionDumper(path_dir_where_to_dump=r"D:\Coding\forecast\spot")
rest_client = BinanceRestClient()

builder = TotalDatasetBuilder(
    "BONK",
    "settings.json",
    force_merge=False,
    historical_source=vision_dumper,
    recent_source=rest_client,
)
result = builder.build()
print(result.filepath, result.rows)

The builder defaults to open_candle="exclude" from settings. You can override it per run:

result = builder.build(open_candle="include")

force_merge=False requires recent historical Binance Vision files before merging. Use force_merge=True to merge anyway when historical files are older or incomplete, for example during manual recovery or experiments.

Total CSV filenames include the market pair, literal spot, candle frequency, and UTC start/end candle datetimes:

<MARKET>-spot-<FREQUENCY>-total-<START_DATETIME>--<END_DATETIME>.csv

Load a total dataset:

from hdw_crypto_data import TotalDatasetLoader

loader = TotalDatasetLoader("BONK", "settings.json")
df = loader.load_total_dataframe(mode="ta", preferred_tz="CET")
print(df.tail())

By default the loader cleans the data without synthesizing missing candles:

df = loader.load_total_dataframe(clean=True, fill_gaps=False)

Set fill_gaps=True only when you want a continuous hourly index. In that mode, missing hourly candles are created and numeric fields are filled by interpolation/backfill/forward-fill. The returned DataFrame includes an is_imputed column so synthetic rows can be filtered or audited:

df = loader.load_total_dataframe(fill_gaps=True)
synthetic_rows = df[df["is_imputed"]]

number_of_trades is rounded back to integer values after filling so interpolated trade counts are not fractional.

Run the small download script from the project root:

py -3 examples/binance_dump.py BONK
py -3 examples/binance_dump.py BONK --start 2026-08-01 --end 2026-08-21

The CLI exits normally on success and prints * * * KLAAR * * *. Argument errors are handled by argparse. Download failures raise BinanceVisionDumpError, so the process exits non-zero and prints the exception traceback unless you catch it from your own wrapper.

Local archive manager

Launch the PyQt6 inventory and recoverable cleanup utility from the project environment:

python -m hdw_crypto_data.archive_manager_gui

Choose spot or its parent, scan, inspect symbol/interval groups, and select individual files. A shared SQLite index under .hdw_archive beside spot supplies cached inventory immediately; background verification inspects only new or changed CSV contents. Writers update the index after successful commits. Refresh / Verify, Deep rescan selected, and Rebuild index offer incremental and forced repair. Cleanup requires a preview and explicit confirmation, then moves files to .hdw_archive/quarantine with a JSON manifest (existing spot-quarantine folders are reused). Restore is available through a tested core API and refuses overwrites. A separate Quarantined assets… overview offers a second cleanup level: permanently delete one whole quarantined asset across all intervals and operations, after preview and explicit confirmation. Active spot files remain untouched. See archive manager documentation for usage, restore examples, safety behavior and limitations. The existing gui extra includes PyQt6. See shared index documentation for schema, ownership, concurrency, cache repair and benchmarks. The filesystem remains authoritative.

Testing

Run the unit tests from the repository root:

$env:NUMBA_DISABLE_JIT='1'
py -m unittest discover -s tests -p "test*.py"

NUMBA_DISABLE_JIT=1 avoids optional pandas-ta/numba cache issues when importing chart tests. tests/smoke_test.py is a manual end-to-end script and is not part of unit-test discovery.

Notes

The package downloads public market data from Binance endpoints. Network failures, missing Binance archives, checksum mismatches, rate limits, and invalid archives are reported explicitly by the dumper.

Original design notes: https://code2trade.dev/c