nse-warehouse

NSE bhavcopy as a partitioned Parquet warehouse with a DuckDB query layer, corporate-action adjustment, data-quality checks, and a point-in-time-correct feature pipeline.

No data is committed. NSE publishes the bhavcopy; redistributing a reconstructed copy of it is not something this repo does. What ships is the pipeline and a documented build process — run it and you get your own warehouse.

Query timings from the committed bench/results.json (timings only, no prices). Regenerate the chart: python bench/make_readme_chart.py; the numbers themselves come from bench/run_bench.py.

Two things the measurements contradicted

Both were premises going in. Both turned out to be wrong, and the numbers are below rather than quietly dropped.

1. Partitioning does not help at this scale

The single file wins on every query shape and is 13% smaller. More files means less cross-file compression and more metadata to open, and with six year-partitions over a mostly-contiguous span there is nearly nothing to prune.

Partitioning is still the default, but for a different and honest reason: incremental rebuilds. Adding a year is a new directory rather than a rewrite. It is not a query-speed optimisation here.

2. NSE equity data is not big enough to OOM pandas

The spec asked for "a query that would OOM in pandas running fine here". It does not exist for this dataset, so no such demo is shipped.

Measured peak RSS for the same group-by:

That is a 10.7× memory ratio, which is the real and defensible claim. But extrapolated linearly, pandas would only exhaust 16 GB at ~36 M rows — roughly 60 years of NSE cash equities at current breadth. A full 20-year backfill is ~12 M rows and fits in pandas comfortably.

So the honest version of the scale claim is "10× less memory and it streams", not "pandas cannot do this". Manufacturing an OOM by loading the data badly would have been easy and dishonest.

What was built and measured

Dataset actually built (see Coverage below — this is a sample, not a full backfill):

Query timings

Median of 5 runs, warm cache, DuckDB 1.5.5, local NVMe.

Compression, measured rather than defaulted

zstd gets within 2% of gzip's size at a third of the write cost and a faster scan. snappy is 46% larger for 2.6 ms. zstd is the default because of this table, not because it is fashionable.

Corporate actions — the part most people get silently wrong

NSE's corporate-actions endpoint returns 403 to automated clients, so actions are inferred from the price series.

The obvious approach fails, and it is worth knowing why. One might hope PrvsClsgPric is already adjusted, making prev_close[t] != close[t-1] an exact flag. It is not — checked against Reliance's 1:1 bonus, ex-date 2024-10-28:

2024-10-25   close = 2655.70   prev_close = 2679.60
2024-10-28   close = 1334.35   prev_close = 2655.70   <- raw prior close

prev_close carries no signal. The only usable one is the discontinuity: 1334.35 / 2655.70 = 0.5024, which snaps to 1/2.

It works. On the real warehouse, RELIANCE is detected correctly, and the effect on returns is the whole point:

Every momentum and volatility feature spanning that date is wrong without this.

A bug the real data found

The first run detected 509 actions. Inspecting them showed the warehouse's sampled date ranges (2005, 2012, 2020, 2024, 2025-26) created seams: the "previous" close for 2012-03-01 was from 2005-01-31, producing ratios like 80× that snapped to plausible-looking splits.

Detection is now gap-aware — pairs more than 7 calendar days apart are skipped — which drops it to 230. This is not just a sampling artefact fix: a genuinely suspended stock resuming months later at a different price would have been misread as a split in a full backfill too.

Known limits, stated not implied

• Dividends are not handled. Ordinary dividend drops are indistinguishable from noise this way. These are capital-adjusted, not total-return prices.
• Small bonuses are missed. A 5:4 bonus (ratio 0.8) sits on the edge of the plausible-trading band and is not reliably caught.
• False positives remain. Rights entitlements (-RE suffix) are volatile instruments that trip the detector; SUMEET-RE produced hits on two consecutive days, which is a near-certain tell that neither is real.
• Detected actions land in a corporate_actions table so they can be reviewed and overridden by an external feed later without rebuilding the raw layer.

Data quality findings on the real data

python -m nse_warehouse.cli check. These are findings, not failures.

The breadth "outliers" are not errors: NSE listed ~800 names in 2005 and ~2,700 now. The check is measuring twenty years of market growth, which is the right answer to a question phrased as an anomaly check.

missing_trading_days reports 5,136 — almost entirely the gaps between the sampled ranges, not missing data. With a contiguous backfill this number becomes meaningful; here it mostly measures the sampling plan.

Coverage — what was actually downloaded

Not a full 20-year backfill. 497 trading days were fetched, chosen to exercise every format era plus a contiguous recent span:

Zero download failures. The full backfill is one command (cli fetch --start 1996-01-01 --end 2026-08-05) and roughly 5,000 files; at the built-in 1.5 s politeness interval that is ~2 hours of wall clock. It was not run here, and no numbers in this README pretend it was.

The three format eras

Mapped by fetching real files, not by reading documentation.

The eras overlap: both formats serve 2024-01 through 2024-11. The fetcher prefers UDiFF there because it carries strictly more information. The legacy path 404s from 2024-12.

Era 1's missing ISIN is not cosmetic — ISIN is the only stable identity across a rename, so symbol-change tracking is impossible on pre-2012 data. The quality check reports that limitation rather than working around it.

Feature pipeline

Point-in-time correct, enforced rather than trusted. Every window ends at t-1 (ROWS BETWEEN n PRECEDING AND 1 PRECEDING), and assert_no_lookahead() fails the build if any feature value reflects its own bar.

Built on adjusted prices: an unadjusted 1:1 bonus looks like a −50% return and poisons every momentum and volatility feature spanning it.

training_set: 1,137,908 rows, 16 columns
no-lookahead assertion passed

Features: 5/21/63/252-day momentum, 21/63-day realised volatility, 21-day median turnover, Amihud illiquidity, and cross-sectional percentile ranks computed strictly within a single date. The 5-day forward return lives in a separate targets view so it can never be mistaken for a feature.

Install

pip install nse-warehouse

Python 3.10+. Runtime dependencies are duckdb, pyarrow and httpx — nothing else. pandas is not required: the warehouse is queried through DuckDB, and pandas appears only in bench/ as the thing being measured against.

Build it yourself

Installing gives you the nse-warehouse command. Every subcommand is also reachable as python -m nse_warehouse.cli <cmd> if you prefer.

nse-warehouse fetch --start 2025-01-01 --end 2026-08-05
nse-warehouse build          # -> partitioned Parquet
nse-warehouse adjust         # -> corporate_actions + bars_adjusted
nse-warehouse check          # -> quality report
nse-warehouse features       # -> features / targets / training_set
nse-warehouse stats

Working from a clone instead, with the benchmark and test extras:

python -m venv .venv && .venv/bin/pip install -e ".[dev,bench]"
python -m bench.run_bench                  # -> bench/results.json

The fetcher is resumable and idempotent: files already on disk are never re-requested, holidays are remembered in a manifest so a resume does not re-probe them, and downloads are write-then-rename so a kill leaves no half-file. Interrupt it freely.

Politeness is not configurable below 0.5 s per request. This is a public archive.

Licence and data position

Code: MIT.

NSE bhavcopy data is NSE's. This repo ships the fetcher and the build process, never the data. If you redistribute a built warehouse, that is between you and NSE's terms.

Not done

• Full 1996→2026 backfill (pipeline ready, ~2 hours, not run).
• Wiring into nifty-forecaster as a consumer — the feature views exist and are tested, but the integration is not written.
• An authoritative corporate-actions feed to replace the heuristic.