marketdata

Daily bars — equities, ETFs and futures — as a producer/consumer split over a file-based store, with adjustment derived on read wherever it can be.

Sibling of cotdata and built on the same design ideas, but deliberately a separate package. cotdata's registry requires a cftc_code and equities have no COT report. See docs/design.md for the full reasoning and for everything about each vendor's behaviour that was verified rather than assumed.

Futures arrived with ADR-0007, which makes cotdata CFTC positioning only and moves every bar here.

Install

uv venv --python 3.11 && uv pip install -e ".[yahoo,dev]" "setuptools<81"

From an index, the distribution is crucible-marketdata and the import stays marketdata:

uv pip install crucible-marketdata     # then: import marketdata

The two differ because marketdata is taken on PyPI by an unrelated, abandoned project (marketData 0.2.0, last released 2020-04-19 — PyPI normalises both to the same name). Same split as python-dateutil → import dateutil. A dependency on this package must name crucible-marketdata, since that is what pip resolves; depending on marketdata would fetch a stranger's 2020 module.

For the databento futures producer (any OS, and the only paid-API path here), add the databento extra:

uv pip install -e ".[yahoo,databento,dev]" "setuptools<81"

On the Windows futures producer, add the norgate extra — nothing else pulls norgatedata, and without it --domain futures stops before it fetches:

uv pip install -e ".[yahoo,norgate,dev]" "setuptools<81"

Installing it elsewhere does not help. It drives a locally installed Norgate Data Updater rather than an API, and NDU is Windows-only, so every other machine reads a synced store instead of producing one.

Use

export MARKETDATA_STORE=~/code/marketdata_store
marketdata-update --bars                    # every registry symbol this box can produce
marketdata-update --bars --symbols SPY TLT  # scoped
marketdata-update --bars --domain equities  # skip futures (no Norgate on this box)
marketdata-update --metadata                # futures contract specs (Windows + Norgate)
marketdata-update --check                   # read-only summary, no network

marketdata-update --bars --domain futures --require-final   # only once Norgate has settled

marketdata-update --ingest-databento        # databento Stage 1 (PAID), any OS
marketdata-update --build-databento         # databento Stage 2 (FREE, offline)

marketdata-update --pin snap.json           # capture the store's state
marketdata-update --verify-pin snap.json    # prove it has not moved, exit 1 on drift

Futures without Norgate: the databento producer

A box that cannot run Norgate — a Linux dashboard server, most obviously — can produce futures bars from Databento GLBX.MDP3 instead. One provider owns a symbol end to end (ADR-0006); a series is never blended.

export MARKETDATA_STORE=~/code/marketdata_store
export DATABENTO_API_KEY=db-...             # Stage 1 only
# optional: MARKETDATA_DATABENTO_RAW=/path  (default: _raw/databento under the store)

marketdata-update --ingest-databento --windowed-n1-stats   # Stage 1: PAID
marketdata-update --build-databento                        # Stage 2: FREE

Two stages, and only one costs money. Stage 1 pulls raw .n.0/.n.1 ohlcv-1d and statistics into an append-only raw store, resuming from each table's last fetched date, so a re-run or a mid-pull failure pulls only what is missing. Stage 2 reads that local store with no API and no network, so the back-adjustment can be iterated for free. The raw store is producer-internal — keep _raw/ out of any sync to consumers.

--windowed-n1-stats fetches the second contract's statistics only around roll dates. Its settlement is read only at rolls, so this drops the largest avoidable download with no accuracy loss; worth it on a cold-start backfill.

Not a Norgate replacement. History starts at the GLBX floor (2010-06-06) against Norgate's decades, and eight registry markets are not on CME Globex at all — the ICE softs, lumber and the dollar index carry databento: null and are skipped rather than paid for. Local research stays on Norgate.

If the resume ledger and the disk disagree, --reconcile-databento repairs both directions from local files: it records tables an interrupted run left unrecorded, and prunes entries whose parquet is missing. The second is the one that matters — such an entry still carries a current last_date, so a restart skips it as "already current" and leaves a permanent hole in a paid dataset with no error anywhere.

Both vendors land in the same store under different paths (bars/futures/norgate/ beside bars/futures/databento/), which is what makes scripts/validate_databento_vs_norgate.py a single-store comparison.

Scheduling it: docs/LINUX_SCHEDULING.md — the wrapper, the crontab line, why the cold-start backfill is not the nightly job, and what to do when the resume ledger and the disk disagree.

Pinning a store for a study

A study that quotes numbers is only reproducible if the data behind them is identifiable, and --bars rewrites every symbol's updated_at while Yahoo restates adjusted history whenever a dividend lands. So the same command against the same path can produce different figures on different days.

--pin captures row counts, date spans, source and updated_at per symbol. --verify-pin compares them and exits non-zero naming every field that moved. Commit the snapshot next to the study that depends on it.

A snapshot is evidence, not configuration. If verification fails, the honest response is to say which figures are now unreproducible, not to re-pin and move on.

from marketdata import get_bars

px = get_bars("TLT", "total", start="2010-01-01")

Two domains, two adjustment axes

The domain a symbol belongs to decides which tiers it has, and get_bars resolves it from the registry rather than taking it as an argument. Asking for a futures tier on an equity — or the reverse — raises a message naming the right ones.

Equities derive everything because corporate actions arrive as dated events alongside the bars. Futures cannot: Norgate's back-adjustment is roll splicing it performed itself, and the stitched calendar spread at each roll appears in no other series, so backadj and unadj are two separate stored facts.

The three equity adjustment tiers

The store holds one frame per equity symbol exactly as the vendor serves it, plus the dated action columns. Yahoo's Adj Close is not stored: it is restated every time a new dividend lands, so a backtest pinned to it is not reproducible. Raw bars plus dated actions are immutable facts, and marketdata.adjust rebuilds any tier from them deterministically.

This is not a cosmetic distinction. TLT over its full history returns +2.1% on the price series and +132.3% on total return.

The three futures adjustment tiers

propadj is not an optional refinement. Additive adjustment accumulates roll gaps downward, and across the cotdata store 52.3% of ZS's back-adjusted closes and 41.2% of DC's are non-positive — a percent return or an R-multiple is meaningless on those. Ratio adjustment preserves percentage returns. It does not make the series strictly positive: it scales by a positive factor, so it keeps the underlying's sign, and CL prints −24.11 on 2020-04-20 because WTI really settled at −37.63. That is one bar out of the whole store.

Because propadj needs both stored tiers, the futures producer writes both or neither, and a read that finds only one raises instead of returning empty. A half-stored symbol is worth being loud about: additive back-adjusted percent volatility comes out ~200x too high for soybeans and 0.47x for gold, and 0.47x passes every implausibility screen a spot check would apply.

Store layout

$MARKETDATA_STORE/
  bars/<domain>/<source>/<symbol>.parquet          # equities — one stored frame
  bars/<domain>/<source>/<symbol>_<tier>.parquet   # futures  — one per stored tier
  metadata/contract_specs.parquet                  # futures point value, tick size, margin
  manifest.json
  _raw/databento/                                  # PRODUCER-INTERNAL, do not sync

The leading underscore marks _raw/ as not a consumer domain. It holds databento's append-only bronze store — the paid Stage-1 landing area that Stage 2 rebuilds from — and it is both the largest thing in the store and useless to a reader. Exclude it from any sync.

The vendor is part of the path, not just the manifest. Yahoo and Norgate overlap almost completely on equities and ETFs and do not store the same columns, so a single bars/<symbol>.parquet would let whichever producer ran last silently win. The domain sits above it because futures and equities have entirely different adjustment axes, so their frames are not interchangeable even when symbol strings collide. Separate directories also make a vendor A/B comparison possible:

get_bars("SPY", "total", source="yfinance")
get_bars("SPY", "total", source="norgate")

Omit source= and the registry resolves one for this deployment. A symbol missing under the resolved vendor but present under another raises rather than returning empty, because silently substituting a vendor is what ADR-0006 forbids.

Environment

The store root may share a parent folder with cotdata's, but the two must not share a manifest.json. Both producers do a read-modify-write on it.

On the Windows futures producer

That box now runs three scheduled producers across two packages, so it needs both store variables set at once, pointing at different roots. This is new with the futures domain: until ADR-0007 moved bars here, COTDATA_STORE alone was the whole story.

setx COTDATA_STORE    C:\Users\YourUsername\cotdata_store
setx MARKETDATA_STORE C:\Users\YourUsername\marketdata_store

setx persists; plain set lasts only for the current Command Prompt, which is the usual reason a scheduled task cannot find a store an interactive shell could. Open a NEW prompt afterwards — setx does not affect the one you typed it in — and verify:

echo %COTDATA_STORE%
echo %MARKETDATA_STORE%
marketdata-update --check

--check reads the manifest and no network, so it is the cheap confirmation that the variable points where you think. An unset variable is refused by name rather than defaulted, because a silent default would write a second store somewhere nobody looks.

Do not point them at one root. Sharing a parent folder is fine and makes the pair easy to sync; sharing a root is not, because both packages keep a manifest.json at their root and each does a read-modify-write on it, so the two producers would eventually drop each other's entries.

Python, virtualenv and Task Scheduler setup are identical to cotdata's and are not duplicated here — see cotdata's Windows setup guide. The only marketdata-specific pieces are the norgate extra in Install above, the two variables here, and the finals gate below.

Waiting for Norgate's Finals

Schedule the nightly futures run with --require-final:

marketdata-update --bars --domain futures --require-final

Norgate's Final futures prices land in the evening, but the Norgate Data Updater still has to pull them on its next poll. --require-final fetches only once Norgate holds a newer settled session than the store already does (checked across ES, CL and ZC, all of which must have advanced). Until then it prints which reference is lagging and exits non-zero.

Give the task's trigger a repetition — fire at 20:55, then repeat every 15 minutes for 5 hours. That turns "fire at 9pm" into "run the moment the Finals land": each repeat is one short date comparison that defers immediately until they do, and on a weekend or holiday the window simply closes, harmlessly.

schtasks /Create /TN "marketdata bars" /TR "<DIR>\run-prices.cmd" /SC DAILY /ST 20:55 /RI 15 /DU 0005:00

[!WARNING] Not "if the task fails, restart every N minutes." This page used to recommend that, and it was wrong in production. That setting covers the scheduler failing to launch the action — it does not fire on a non-zero exit code from your script. A run whose action returns 1 is recorded as event 102, "Task Scheduler successfully finished", and no restart is scheduled.

Measured on the reference box, which had RestartCount 20 / RestartInterval PT15M set on the bars task: from 2026-08-12 to 08-15 the action returned exit 1 (a --require-final defer) and the task was launched exactly once each night — four consecutive nights, no retry, no bars captured. Events 111 and 322–324, the restart and queue events, never appeared at all.

The failure stayed invisible because the gate self-heals: a missed night is captured by the next run that finds Norgate ahead of the store, so the data was never permanently wrong, only a day late, and --check a week later looked fine.

A repetition fires on schedule regardless of what the previous run returned, which is exactly what a poll needs. schtasks sets it with /RI (interval, minutes) and /DU (duration, HHHH:MM); /Change converts an existing task in place. In the GUI it is on the Triggers tab — "Repeat task every…" — not the Settings tab, which is where restart-on-failure lives. Full detail, including why the duration is picked from when the source could plausibly arrive, is in cotdata's scheduling guide.

The gate is opt-in and futures-only. Without the flag the run is unconditional as before; with --domain equities it is refused rather than ignored, because yfinance has no settled-versus-interim distinction to gate on. --final-cutoff is accepted and ignored, so a scheduler carrying cotdata's flag does not break: the gate compares data, not clocks, and docs/design.md records why (cotdata's fixed cutoff deferred every attempt on 2026-07-27, when Norgate published at 8:49pm against a 20:55 threshold).

If the bars task is currently chained behind cotdata's run-prices.cmd with an ERRORLEVEL guard, cotdata's gate has been protecting this one. That still works; the flag makes the task correct on its own, so the chain becomes a convenience rather than the only thing standing between the store and an unsettled bar.

Scheduling the equities half

Equities get their own task, not a step appended to the futures wrapper. Three reasons, any one of which is sufficient:

1. The futures wrapper exits early by design. run-prices.cmd carries the --require-final exit code straight out, so once the futures half has captured, every later repeat exits at line one. Anything chained behind it is unreachable on those repeats — equities would get exactly one attempt per night, with the repetition trigger above providing no retry for it at all.
2. The two halves fail differently. --bars --domain equities reports ok only when failed == 0, so one flaky Yahoo symbol fails the whole run. Chained behind the futures fetch, that transient would abort the replica syncs and strand the futures bars written that night on the producer. One vendor's hiccup should not hold the other vendor's good data hostage.
3. Yahoo needs no finals gate. The session's daily bar is available shortly after the 16:00 ET close, so this runs at 17:30 ET and is finished — retries included — before the 20:55 futures task starts. Both wrappers end by mirroring the same replicas, and two of those running concurrently is a race nobody wants to debug.

schtasks /Create /TN "marketdata equities" /TR "<DIR>\run-equities.cmd" /SC WEEKLY /D MON,TUE,WED,THU,FRI /ST 17:30

No --require-final — the gate is futures-only and is refused here rather than ignored (update.py exits 2). yfinance publishes no settled-versus-interim distinction, so there is nothing to gate on. The protection comes from cadence instead: the provider fetches period="max" and write_bars replaces the whole parquet, so every run restates the full history and a provisional bar captured today is overwritten tomorrow. That self-healing is why this task must be daily rather than weekly or monthly — the store keeps no per-bar record of whether a value was provisional, so on a monthly cadence a bad capture would sit there unmarked for a month.

No --metadata either: that fetches futures contract specs from Norgate, and the futures task already runs it nightly.

Put the retry inside the wrapper, not on the task — see the warning above for why Task Scheduler's restart-on-failure cannot do it. A repetition trigger is also the wrong shape here: with no --require-final gate to defer cheaply, every repeat after a success would re-fetch every symbol and re-run both replica syncs. A short retry loop around the fetch alone is what you want; there is a worked template at cotdata's docs/examples/windows/run-equities.cmd.

Tests

.venv/bin/python -m pytest tests/ -q -m "not network"

.venv/bin/python -m pytest tests/ -q

CI runs the first form on every push and PR across Python 3.10 to 3.14. The network tests are not part of that gate. They run weekly in a separate vendor-pin workflow, because their result depends on a third party rather than on this code: a change in Yahoo's adjustment convention is worth hearing about within a week, but is never a reason to block an unrelated PR.

tests/test_pin.py is the load-bearing one. It reconstructs Yahoo's own Adj Close from Close + Dividends across five symbols and asserts the match to 1e-4. That test is what earns the right to drop the restated column from the store. It also asserts the pin set contains split-heavy names, because TLT and SPY have never split and would pass even with a double-applied-split bug.

Survivorship

Every registry symbol is currently listed. yfinance cannot serve delisted securities or point-in-time index membership, so this is not a point-in-time universe and must never be treated as one. Fine for liquid ETFs, a hard ceiling for a broad equity study. Norgate US Stocks fixes it, but only at Platinum and above.

Not built yet

The RealTest CSV exporter, pending verification of RealTest's current data-import layout. The intent is a pure projection: parquet stays authoritative, CSV is regenerated, and the export records which store state fed it.