market-data-normalizer (mdnorm)

Normalize heterogeneous market-data feeds — CSV tick dumps, exchange WebSocket JSON, and FIX — into a single, exchange-agnostic event schema, so downstream research and execution code never has to care where a tick came from.

Zero runtime dependencies. Pure Python (3.10+). Decimal prices, integer nanosecond timestamps.

Why

Every venue spells the same thing differently: BTCUSDT vs XBT/USD, millisecond epochs vs FIX UTCTimestamp, is_buyer_maker booleans vs side codes. Research notebooks and backtesters end up littered with per-venue parsing branches. mdnorm pushes that mess to the edge and hands the rest of your stack one clean type.

Install

pip install market-data-normalizer

The distribution is named market-data-normalizer; the import name is mdnorm:

import mdnorm

Pure Python, no runtime dependencies, Python 3.10+.

Quick start

from mdnorm import from_csv_row, from_ws_json, from_fix

# CSV row (ISO-8601 timestamp)
from_csv_row(
    {"symbol": "btc/usd", "ts": "2026-01-02T00:00:00Z",
     "price": "42000.5", "size": "0.25", "side": "buy"},
    venue="coinbase",
)

# Exchange WebSocket trade message
from_ws_json({"s": "BTCUSDT", "p": "42000.5", "q": "0.25",
              "T": 1767312000000, "m": False}, venue="binance")

# FIX execution report (SOH-delimited in the wild; "|" here for readability)
from_fix("55=BTC/USD|31=42000.5|32=0.25|54=1|60=20260102-00:00:00",
         venue="lmax", sep="|")

All three calls above produce the same MarketEvent.

Quotes (bid/ask)

from mdnorm import from_ws_quote

q = from_ws_quote(
    {"s": "BTCUSDT", "b": "41999.5", "B": "1.2",
     "a": "42000.5", "A": "0.8", "T": 1767312000000},
    venue="binance",
)
q.mid_price   # Decimal("42000.0")
q.spread      # Decimal("1.0")

from_csv_quote does the same for CSV rows with bid/ask columns.

OHLCV bars

from mdnorm import time_bars

bars = time_bars(events, interval_ns=60_000_000_000)  # 1-minute bars
bars[0].open, bars[0].high, bars[0].low, bars[0].close, bars[0].volume, bars[0].vwap

time_bars reduces a stream of trade events into fixed-interval OHLCV Bars (with VWAP and trade count), sorting out-of-order input and skipping quotes.

resample_bars(bars, interval_ns) downsamples bars to a coarser interval (e.g. 1-minute → 5-minute) with correct OHLC aggregation and volume-weighted VWAP.

fill_gaps(bars) returns a gapless series, inserting flat zero-volume bars (OHLC = previous close) for any interval with no trades — a continuous grid for backtests and feature pipelines.

Event-driven bars

Time bars are not the only clock. Sample by activity instead:

from decimal import Decimal
from mdnorm import count_bars, volume_bars, dollar_bars

count_bars(events, every=500)                       # tick bars
volume_bars(events, min_volume=Decimal("100"))      # volume bars
dollar_bars(events, min_notional=Decimal("1e6"))    # dollar bars

Trading sessions

Filter a feed down to the hours that matter, with daylight saving handled for you:

from mdnorm import US_EQUITY_RTH, filter_session, group_by_session_date

rth = filter_session(events, US_EQUITY_RTH)        # 09:30-16:00 New York
by_day = group_by_session_date(events, US_EQUITY_RTH)

Overnight windows (a session that opens at 18:00 and closes at 17:00 the next day) are supported, and session_date keeps a whole night in one bucket. From the command line:

$ mdnorm bars trades.csv --interval 5m --session 09:30-16:00 --tz America/New_York -o rth.csv

Holidays, half-days, and the year that is not 252 sessions

A session is a recurring window. A calendar is that window plus the exceptions to it, and the exceptions are the part that quietly breaks things.

from mdnorm import read_calendar_csv, US_EQUITY_RTH

cal = read_calendar_csv("us_2026.csv", US_EQUITY_RTH)
cal.is_trading_day(date(2026, 7, 3))                     # False, from the file
cal.close_time(date(2026, 11, 27))                       # 13:00, a half-day
cal.trading_minutes_between(jan, dec)                    # not sessions x 390

A missing holiday looks exactly like missing data. A pipeline that does not know a date is a holiday sees a day-long gap and reports an outage, or fills it, or drops the instrument for poor coverage. All three are wrong in the same way: the data was never supposed to be there.

A half-day is not half a problem. An early close shortens the session and changes nothing else, so bars keep being cut against a 6.5-hour assumption, a volatility annualised on session length is overstated for that day, and a staleness check fires on every instrument at once an hour before it should.

A calendar cannot answer outside the range it was given. A file listing 2026 says nothing about 2027, and a calendar that treats an unknown weekday as open turns a missing file into a confident wrong answer. Every query outside covers raises instead — noisy exactly once, and then correct.

252 is a convention, not a count. How many sessions a year holds depends on where the weekends and holidays fell; how many minutes it holds depends on how many of those sessions closed early. Both are computable, and both rescale every annualised figure in a report while leaving its shape untouched.

$ mdnorm calendar us_2026.csv --session 09:30-16:00 --tz America/New_York
trading days         251
early closes         2
trading minutes      97530
note: early closes cost 360 minute(s) against a flat 390-minute session.
for `mdnorm features`: --sessions-per-year 251 --session-length 23400s
note: 2026 has 251 sessions here, not the conventional 252. Annualising a
volatility on 252 overstates it by 0.20%.

A price is a number and a currency

The moment a study spans venues that quote in different currencies, every figure in it depends on a second series nobody was watching.

from mdnorm import CurrencyPair, FxRates, convert_series, decompose_return

rates = FxRates({CurrencyPair("EUR", "USD"): eurusd})
usd, dropped = convert_series(prices, rates, base="EUR", to="USD",
                              max_age_ns=MINUTE)

There is no default conversion time. Converting at the observation's own timestamp, at a daily fix, or at the end of the study are three different questions, and only the first was available to someone standing at that moment. The last is the one that gets used by accident, because one rate is easier to obtain than a series — and it restates the whole history using a number that did not exist until the end of it. Nothing here accepts a scalar rate.

Staleness is the ordinary failure. FX stops over weekends while other venues keep trading, so an as-of join with no age limit converts a Sunday print with Friday's close. max_age_ns is required, and every conversion carries the age of the rate it used.

Direction is not guessable from a name. Vendors disagree about which way round to publish a pair, and a rate applied upside-down is either wrong by a factor of thousands or — near parity — wrong by a few per cent and entirely plausible. Pairs carry an explicit base and quote, inversion is recorded in the result, and it can be refused outright.

A cross is not free, and no path is searched for. Going through a vehicle currency multiplies two quotes and inherits both spreads and both staleness windows. State the vehicle with via= or the conversion is refused: a library that finds its own way through the currency graph is choosing which spreads you pay, invisibly.

A converted return is not a converted price. `(1 + total) = (1 + asset)(1

• fx)holds exactly; the familiar shorthand adds the two and drops the product.decompose_return` returns both and the difference between them.

$ mdnorm fx prices.csv rates.csv --from EUR --to USD --max-age 1m -o usd.csv
note: the rate moved +9.09% across this span, so a single-rate conversion
would have restated the whole series by a number that did not exist until the
end of it.

Prices live on a grid, and the grid is data

A venue does not accept any price. It accepts multiples of a tick, and the tick depends on the price band, the instrument and the year.

from mdnorm import TickTable, TickBand, grid_report, spread_in_ticks

table = TickTable([TickBand(D("0"), D("0.0001")),
                   TickBand(D("1"), D("0.01"))])
grid_report(prices, table).looks_raw      # could the venue have quoted these?
spread_in_ticks(bid, ask, table)          # 1.0 is the floor, not a tight market

A price off the grid is telling you something. Raw prints sit on the grid by construction — the venue would not have accepted them otherwise. So a series that does not is a mid, a VWAP, an average of venues, a back-adjusted history, or an error, and those are indistinguishable by eye. grid_report is one pass over the data and answers a question most pipelines never ask.

Back-adjustment takes a series off the grid permanently, and that is correct. An adjusted history is a returns object, not a price object. It stops being correct when someone rounds it back on to make it look tidy. The grid is the cheapest way to tell the two apart after the fact.

There is no default tick size. The familiar penny is wrong below a dollar on most venues, wrong for sub-penny programmes, wrong for crypto by orders of magnitude, and wrong for the same instrument before the last regime change — so tick tables are point-in-time data, and TickSchedule refuses to answer before the first one it was given.

Ties are not an edge case here. On a continuous scale an exact half is a curiosity; on a tick grid a mid between adjacent ticks is a half-tick every single time. Rounding has no default and no tie shortcut.

Round against yourself, or say that you did not. executable rounds a buy down and a sell up, so the grid never makes an order more aggressive than the strategy asked for. Rounding to the nearest tick does the opposite about half the time, which lifts the fill rate in any backtest that fills limit orders at their limit. The clearest case is the mid: a market quoted one tick wide has a mid exactly half a tick from both sides, so it is not a price the venue could ever accept, and filling there understates cost by half the spread on every trade.

$ mdnorm ticks prices.csv --table ticks.csv
off the grid         2
note: 2 price(s) could not have been quoted on this grid, so this series is
not raw prints.

Corporate actions and contract rolls

A raw price series is not continuous. A 4-for-1 split divides the printed price by four overnight, a cash dividend drops it by the amount paid, and a futures roll steps it by the spread between the two contracts. None of them are market moves, but all of them look like returns:

from decimal import Decimal
from mdnorm import adjust_bars, split, dividend, roll, iso_to_ns

actions = [
    split(iso_to_ns("2026-06-06T00:00:00Z"), Decimal("4")),
    dividend(iso_to_ns("2026-05-09T00:00:00Z"), Decimal("0.25")),
]
clean = adjust_bars(bars, actions)

Back-adjustment leaves the most recent segment at the prices that actually printed and restates everything before each event, so the joins are seamless:

raw closes    500   502   498   504  │  126  125.5   127  126.5
raw returns       +0.4% -0.8% +1.2%  │ -75.0% -0.4% +1.2% -0.4%
                                     ^ the split, not a crash

adj closes    125  125.5 124.5  126  │  126  125.5   127  126.5
adj returns       +0.4% -0.8% +1.2%  │  +0.0% -0.4% +1.2% -0.4%

Splits scale volume as well as price. Dividends take their reference price from the last print before the ex-date unless you pass one. Rolls support both conventions — AdjustMethod.RATIO (default, preserves returns) and AdjustMethod.DIFFERENCE (preserves price differences, the usual choice for futures). Factors are composed as exact rationals, so a 1-for-2 followed by a 1-for-3 restates 600 to exactly 100 rather than 99.999...96.

Actions can come from a file, and the CLI wires it up:

$ mdnorm bars trades.csv --interval 1d --actions actions.csv -o adjusted.csv
$ mdnorm bars tape.jsonl --infer-sides --every-imbalance 500 -o imbalance.csv
$ mdnorm book deltas.csv --symbol BTC-USD -o quotes.jsonl
$ mdnorm nbbo quotes.jsonl --max-age 2s -o top.jsonl
$ mdnorm tca fills.csv --market tape.jsonl --decision-price 100

ts,kind,value,ref_price
2026-06-06T00:00:00Z,split,4,
2026-05-09T00:00:00Z,dividend,0.25,190.50
2026-03-14T00:00:00Z,roll,5312.50,5290.25

Who crossed the spread

Most trade tapes give you a price and a size but not the aggressor. That one missing field is what separates a price series from an order-flow series, and signed volume, order imbalance and imbalance bars are all defined in terms of it. mdnorm.micro infers it, using the three rules the literature settled on:

from mdnorm import SideRule, infer_sides, trade_imbalance, mean_effective_spread

classified = infer_sides(events)                       # Lee-Ready by default
print(trade_imbalance(classified))                     # -1 selling .. +1 buying
print(mean_effective_spread(classified))               # 2 * |price - mid|

SideRule.TICK compares each trade with the previous different price and needs trades only. SideRule.QUOTE compares the trade with the prevailing mid. SideRule.LEE_READY — the default — uses the quote rule and falls back to the tick rule at the mid. A side reported by the venue always wins; inference only fills gaps, and trades it cannot resolve stay None rather than being guessed at. Published accuracy of these rules is roughly 75-85% on liquid names, so treat an inferred side as an estimate.

roll_spread estimates the effective spread from trade prices alone, via the serial covariance that bid-ask bounce induces. It needs no quotes, which makes it a useful cross-check on the rest — and it returns None rather than zero when the covariance comes out non-negative and the estimator is undefined.

Imbalance bars

Once trades carry a side, the sampling clock can follow order flow instead of time or volume:

from mdnorm import Pipeline

bars = Pipeline().infer_sides().imbalance_bars(Decimal("500")).run(events)

A bar runs until buyers have outbought sellers, or the reverse, by the threshold. Balanced two-sided periods produce one long bar; a sustained one-sided push produces several short ones. by="tick" measures the imbalance in trade count rather than size. From the command line:

$ mdnorm bars tape.jsonl --infer-sides --every-imbalance 500 -o imbalance.csv

Rebuilding the order book

Exchanges do not send you a book. They send a snapshot and then a stream of deltas, and the book only exists if you apply every one of them, in order:

from mdnorm import BookDelta, OrderBook, Side, replay_book

book = OrderBook("BTC-USD", "binance")
book.apply_snapshot(ts, bids=[(D("100"), D("2"))], asks=[(D("101"), D("3"))], seq=10)

quotes = list(replay_book(book, deltas))     # one quote per change in the top
print(book.best_bid, book.spread, book.imbalance(levels=5))

Two failure modes make a reconstructed book silently untrue, and this implementation refuses to hide either.

A sequence gap means a message was missed, and no later update repairs the damage — the book is simply wrong from then on, in a way that looks completely normal. OrderBook raises SequenceGapError the moment a number is skipped, naming how many updates went missing, because the correct response is to resynchronise from a snapshot rather than carry on. Duplicated or replayed messages are rejected the same way. Feeds without sequence numbers work fine; pass strict_sequence=False to opt out entirely.

A crossed book — best bid at or above best ask — is not a market state but a symptom: a dropped delete, a stale snapshot, two venues merged by mistake. It is exposed as is_crossed, and the spread goes negative rather than being quietly clamped to zero.

to_quote() turns the top of the book into an ordinary MarketEvent, so a reconstructed book feeds straight into session filtering, trade classification and effective spreads with nothing in between. From the command line:

$ mdnorm book deltas.csv --symbol BTC-USD --venue binance -o quotes.jsonl

One instrument, several venues

When something trades in more than one place, "the price" is a question. The consolidated top of book is the answer, and it is where three problems live that a maximum over venues will not warn you about:

from mdnorm import consolidate

top = consolidate(quotes, max_age_ns=2_000_000_000)   # 2s staleness cutoff

A venue that goes quiet keeps voting. When a feed disconnects, its last quote stays in the consolidation forever — and a stale price is very often the best price, so the dead venue ends up setting the top of book. This is the failure that produces a consolidated feed which looks excellent and is fiction. max_age_ns retires a venue that has not spoken recently; stale_venues() names them.

A consolidated book can appear crossed. A bid on one venue above the offer on another looks like free money and is almost always clock skew between two feeds timestamped by different machines. is_crossed reports it and crossed_updates counts it, because the useful response is to check the clocks rather than to trade the spread.

Ties need a rule. Equal best prices are broken by size, then by venue name, so the same input always produces the same output.

Which venue actually sets the price is a measurement in its own right, and leadership counts it. The pieces compose: an order book becomes a quote, quotes from several venues consolidate into one, and the result feeds trade classification and effective spreads unchanged.

$ mdnorm nbbo quotes.jsonl --symbol BTC-USD --max-age 2s -o top.jsonl

Did I execute well?

Once the tape is clean the next question is about you rather than the market, and every standard benchmark has a way of quietly flattering the person running it:

from mdnorm import Fill, Side, evaluate

report = evaluate(my_fills, market_trades, decision_price=D("100"))
print(report.slippage_vs_vwap_bps, report.participation_rate)

Your own trades are in the benchmark. A VWAP over the public tape includes the prints you just made, so you end up partly benchmarking yourself against yourself — and the bigger your share of volume, the more the benchmark bends toward your own average price. evaluate removes your fills from the tape before computing anything; exclude_fills does it on its own if you want the benchmark separately. In the library's own test suite, leaving them in turns a 100 VWAP into 109 and a losing execution into a winning one.

Participation decides whether the number means anything. Beating VWAP by two basis points on 0.1% of volume is a result; the same number on 30% of volume mostly measures your own impact. The summary always reports the two together, and the CLI says so out loud above 10%.

Sign conventions are stated, not assumed. Positive basis points always mean better than the benchmark — paying below it on a buy, selling above it on a sell. Mixed-side fills are refused rather than netted, because one number covering both directions has no meaning.

By default the window runs from your first fill to your last. That is right for a worked order and wrong for a single fill — the only print in the window is then your own — so start_ns and end_ns let you score against an interval you chose instead.

TWAP skips intervals that never traded instead of carrying the last price forward, for the same reason nothing else here invents data.

$ mdnorm tca fills.csv --market tape.jsonl --decision-price 100

Several instruments, one time grid

Research wants a matrix — one row per timestamp, one column per instrument — and building it from independent tick streams is where look-ahead bias gets in, because every mistake here makes the backtest better rather than raising:

from mdnorm import Field, align

rows = align({"BTC": btc_events, "ETH": eth_events},
             interval_ns=60_000_000_000,       # a one-minute grid
             max_age_ns=5 * 60_000_000_000)    # nothing older than 5 minutes
rows[0].values      # {"BTC": Decimal("60000"), "ETH": Decimal("3000")}
rows[0].ages_ns     # how old each value was at that grid point
rows[0].complete    # False if any column had nothing to show

The join only looks backwards. A value is visible at a grid point only if it was observed at or before it. "Nearest observation" is the expensive default in this area: on a one-minute grid it lets a print from 09:30:20 be read at 09:30:00, and twenty seconds of hindsight is enough to make a mediocre signal look tradeable.

A bar labelled 09:30 is not knowable at 09:30. It contains everything that traded until 09:31, so joining bars on their label imports an interval of the future. AsOfSeries.from_bars timestamps each bar at its end, and align_bars therefore gives you the last closed bar per column — one interval further back than the naive join, and the version you could have traded.

Forward-filling has no natural end. A halted or delisted stream otherwise contributes its last price forever, and a frozen price correlates with nothing, which reads as diversification. With max_age_ns a quiet column becomes None; the age is still reported, so row.stale (had data, too old) and row.missing (never had data) stay distinguishable.

A feed you get late was not available on time. AsOfSeries.delayed(250ms) shifts observation times forward by the delivery delay, so alignment reflects when you could have acted rather than when the source stamped it. If you do not know what your delay is, the next section measures it.

Nothing interpolates or smooths. align_on takes timestamps you supply, for one row per print of a reference instrument, per signal, or per fill.

$ mdnorm align BTC=btc.csv ETH=eth.jsonl --interval 1m --max-age 5m -o matrix.csv

When the venue says it happened, and when you found out

AsOfSeries.delayed has always taken a delay and said, in its own docstring, that a delay of zero is a claim about your infrastructure rather than a default. It never offered a way to find out what yours is. This is that half:

from mdnorm import Arrival, delay_report, as_received, as_stamped, view_gap

report = delay_report(arrivals)     # what the transport actually costs
report.median_ns, report.p95_ns     # the typical case, and the one to size for
report.negative                     # rows received before they happened
report.out_of_order                 # messages that overtook the one before

knowable = as_received(arrivals)    # the series you could have acted on
optimistic = as_stamped(arrivals)   # the series research usually builds
view_gap(arrivals, grid).share      # how often those two disagree

A venue timestamp is not an arrival. Keying research on it claims the information reached you instantly, and the error only ever points one way: every signal looks actionable slightly earlier than it was, every cross-venue lead is inflated by the difference in transport, and a fill is priced at a quote that had not reached the machine placing the order. Nothing fails. The result is simply better than it should be.

There is no default delay. A file with no receipt column gets no invented one. State what you decided to believe with assume_delay_ns= and the report comes back with assumed=True — a report that hides which of the two it used is worse than no report.

A negative delay is a fact, not an outlier. Receipt before the venue stamp means the clocks disagree, which is usually the more interesting finding. It is counted separately and never clamped, because clamping turns a clock problem into a latency figure that looks fine.

The mean latency is the least useful summary there is. A transport distribution has a tail and the mean mostly measures it. The report gives the median and the p95 by nearest rank, so every figure it prints is a delay that actually happened, and tail_ratio says whether the typical case and the bad case are the same problem.

view_gap asks both views what they knew at each grid point and reports where they differ, with largest_gain_ns — the most unearned foresight found. That number is the one to compare against the horizon your signal acts on: a quarter of a second is nothing to a daily rebalance and everything to a queue position.

$ mdnorm arrival feed.csv --interval 1s
observations         184203
median delay         412us
p95 delay            3.9ms
p95 / median         9.47x
received before sent 0
out of order         37
for `AsOfSeries.delayed`: by_ns=412000 for the typical case, 3900000 for the
case worth sizing against.

Choosing where the sample starts is a trial

A backtest is reported as one number over one window. The window was chosen too — by when the data happened to begin, by which vendor file was to hand, or by somebody sliding the start forward until the curve looked right. The last of those is a search, and nobody counts it:

from mdnorm import trimmed_starts, sweep, sharpe_ratio

rep = sweep(returns, trimmed_starts(len(returns), step=21, count=24),
            sharpe_ratio, kind=WindowKind.TRIMMED_START)
rep.full          # -0.0329 — a losing strategy over everything
rep.highest       # +0.0524 — profitable, starting eleven months in
rep.changes_sign  # True
rep.trials        # 25 — the number to hand a deflated Sharpe

A window count is a trial count. Twenty-four start dates are twenty-four alternatives that could have been reported. Quoting the best of them without deflating for that count is the same error as quoting the best of twenty-four strategies, and it is harder to see because only one strategy was ever written down. SensitivityReport.trials exists so the number reaches deflated_sharpe_ratio instead of somebody's memory of how the window was picked.

The spread is the finding, not the best value in it. A metric that is positive on seven windows out of twenty-four and negative on the other seventeen has not been measured badly — it has been measured, and the answer is that the headline figure is mostly a function of where the window opens.

Shorter windows are noisier, and that is part of what you see. A metric over a third of the data carries roughly √3 times the standard error, so some of the spread is sampling noise rather than instability. Nothing here separates the two; that would need a model of the return process and this library does not have one. Every window carries its observation count so the shrinkage is visible, and a wide spread is consistent with instability rather than proof of it.

$ mdnorm windows pnl.csv --metric sharpe --trim-start 21 --count 24 --deflate
observations         1000
windows              24 (trimmed_start)
  shortest           496
  longest            979
full sample          -0.0329
lowest               -0.0351
median               -0.0168
highest              0.0524
spread               0.0875
positive             7/24 (29.2%)
note: the metric is positive on some windows and negative on others. That is
not a matter of degree.
trials               25
best window deflated
  as one trial       0.9510
  as 25 trials       0.4730

Read the last two lines together. The best window deflates to 0.95 if you pretend it was the only thing you ever looked at, and to 0.47 once the twenty-five windows are counted — and the strategy loses money over the full sample either way.

The sensitivity is usually to a handful of observations. If dropping the first four months changes the answer, find out what was in those four months before concluding anything about regimes. mdnorm.extremes counts how few observations a result rests on, and a moved start date is often just a dropped outlier wearing a different hat.

A fat tail and a fat finger look identical

One is the risk you are paid to carry, the other is a typo, and nothing in the number distinguishes them. So this measures what removing them would cost and removes nothing:

from mdnorm import flag_extremes, clip_effect, concentration

flag_extremes(returns, sigma=5, robust=True)     # 30 found
flag_extremes(returns, sigma=5)                  # 0 found
concentration(returns, share=Decimal("0.5"))     # 3 observations

The outliers hide inside the ruler used to find them. A z-score divides by a standard deviation computed from the same sample, and every extreme observation inflates it. Thirty contaminated points in a thousand raised the scale by 1.42x, which pushed all thirty from six sigma down to four and a bit — so at a five-sigma cut the ordinary score found none of them and the robust score found all thirty. That is masking, and it starts as soon as contamination is more than about one per cent.

So the scale is computed two ways and you choose. robust=True is the median and the median absolute deviation scaled by 1.4826, which the extremes cannot move. Right for detection, wrong for description — a robust scale deliberately ignores the tail you may be trying to measure. Both are reported and neither is assumed.

Clipping always lowers the measured volatility; what it does to the Sharpe depends on which side the tail was on. Symmetric extremes leave the mean alone and the ratio rises, which is the case people have in mind. A one-sided tail takes the profit with it and the ratio falls — 0.88x on the run below. Both are distortions of the same size, so clip_effect reports the shift without asserting a sign and hands back no data.

Concentration is the question behind all of it. Three observations out of a thousand make half the total here. That strategy is a bet on three days, and whether those three were real is the only question that matters.

$ mdnorm extremes pnl.csv --sigma 5 --robust --tail 10 \
    --concentration 0.5 --clip 3
observations         1000
ordinary centre +0.00034184  scale 0.01422197
robust   centre +0.00024111  scale 0.01004799
note: the ordinary scale is 1.42x the robust one, which means the extremes
are inflating the ruler they would be measured against.
at 5 sigma
  ordinary score     0
  robust score       30
largest 10 by size
  their total        0.12
  whole total        0.3418381129
  without them       0.2218381129
  share of the total 35.10%
concentration        3 observation(s) make 50% of the total
clipping at 3 sigma
  would touch        32
  volatility         0.01422197 -> 0.01100780
  understated to     0.7740x
  Sharpe             0.8838x (lower)
note: the clip lowered the Sharpe rather than raising it, which means the
tail was one-sided and the profit went with it.

Nothing was clipped. winsorise exists and is separate, so trimming a sample is a visible act rather than a side effect of measuring it — and the sigma you picked is a parameter chosen after seeing the data, which is what mdnorm.provenance is for.

A result you cannot reproduce is not a result

Every other module here refuses to guess a constant. metrics will not invent the number of trials a search ran, coverage will not pick a gap threshold, halts will not infer a pause, independence will not choose a truncation lag. Each refusal hands the caller a decision — and until now nothing wrote down what they decided:

from mdnorm import manifest, verify, read_manifest, write_manifest

m = manifest(command="sharpe", inputs=["pnl.csv"],
             parameters={"trials": 500, "risk_free": "0.04"})
write_manifest(m, "run.json")          # fingerprint ed731c772bdd

v = verify(read_manifest("run.json"), parameters={"trials": 50})
v.reproducible                          # False
v.drifts                                # pnl.csv: digest, trials: 500 -> 50

The parameters are the part that goes missing. An input that changes is usually noticed, because somebody had to change it. A trial count that was 500 in the run and 50 in the write-up is noticed by nobody, and it is the whole difference between a deflated Sharpe that survives and one that does not. A manifest records the arguments beside the data because they are the same kind of fact.

A fingerprint that includes the clock answers no question. Two runs of the same pipeline over the same inputs with the same arguments must fingerprint identically, so created_ns and the free-text note are excluded and only what would change the numbers is covered. Outputs are excluded too, deliberately: the same fingerprint with different results is the finding.

Floats are refused. 0.1 is not a value, it is a rendering of one, and the rendering differs by platform. Pass a str or a Decimal and the manifest records what you meant; pass a float and it raises rather than promising to reproduce something it can only approximate.

An edited manifest is refused. read_manifest re-derives the fingerprint from the contents and raises if the file disagrees with itself. A manifest somebody has corrected by hand is worse than no manifest, because it carries the authority of a record while stating something that never happened.

Nothing here judges. verify reports what moved and stops. Whether a changed input is a correction or a corruption is not a question a library can answer, and a tool that decided would be trusted for a judgment it is not entitled to make.

$ mdnorm provenance run.json --verify --parameter trials=50
command              sharpe
recorded with        market-data-normalizer 1.35.0
fingerprint          ed731c772bdd
inputs               1
result               2 difference(s)
  pnl.csv: digest 'b264dd52615d' -> 'db1608907c45'
  trials: parameter '500' -> '50'
note: nothing here says which side is right. A changed input may be a
correction or a corruption, and that is not a question this library can answer.

Exit status is non-zero when a run does not reproduce, so this drops into a scheduled check without any parsing.

The absence of a row is not the absence of an event

A feed that stops delivering and a market that stops trading produce the same thing: nothing. Every calculation downstream reads the silence as information:

from mdnorm import coverage_report, panel_coverage

rep = coverage_report(events, min_gap_ns=5 * MINUTE, calendar=cal,
                      halts=halts, start_ns=first, end_ns=last)
rep.explained_share        # 97.27% of the silence was the venue being shut
rep.unexplained_ns         # 5h54m nobody has accounted for
rep.longest_unexplained_ns # 3h10m — a feed that stopped mid-session

A gap has to be explained before it is a gap. Most silences are ordinary, and a raw gap count is mostly weekends. explain_gaps subtracts closed sessions using a TradingCalendar and pauses using the halts windows, then reports the residual — which is the only part that says anything about the feed. The three components are times rather than one label, because a Friday outage running into a weekend is part missing data and part closed venue.

Give the bounds, or a feed that stopped is invisible. Without start_ns and end_ns a gap only exists between two observations, so a symbol that goes quiet and never returns produces nothing at all — its last print has nothing after it to be distant from. That is the case worth catching, because instruments stop printing when something has happened to them. With the bounds, a named symbol that never appears is one gap the width of the period.

Names go missing when they are in trouble. A cross-sectional rank or z-score over "the instruments that printed" is computed on a universe whose width moves, and the ones that drop out are not a random sample. panel_coverage counts the width at every point instead of averaging it away.

No default threshold, and the calendar is recorded. Five minutes without a print is remarkable on a liquid future and unremarkable on a corporate bond, so min_gap_ns is required — the same objection halts makes to inferring a pause. A report built without a calendar carries calendar=False, because counting every night as missing data is right for a venue that never closes and badly wrong for one that does.

$ mdnorm coverage feed.csv --min-gap 5m --calendar us_2026.csv \
    --session 09:30-16:00 --tz America/New_York --panel 1d \
    --since 1773149400000000000 --until 1773432000000000000
events               5481
symbols              3
covered span         307h30m
gaps over 5m         15
  total silence      216h24m
  venue was shut     210h
  halted             30m
  unexplained        5h54m
longest unexplained  3h10m
explained            97.27% of the silence

A price you could not have traded at

When an instrument is halted the tape goes quiet, and a backtest reading that tape sees nothing unusual. The last print stands, the features keep updating off it, and every fill placed in that silence is a fill that could not have happened:

from mdnorm import halt_report, reopen_gaps, unfillable

halt_report(events, halts).halted_share      # 25.6% of the covered span
reopen_gaps(events, halts)[0].move_bps       # -1,834.8 across one pause
unfillable(decisions, halts).value_share     # 91.6% of the money

Halts are concentrated in exactly the wrong place. An instrument is not paused on a quiet afternoon. It is paused on the day of the earnings leak, the guidance cut, the tender offer — the days carrying the largest moves in the sample. The fills a backtest invents during a halt are not a random slice of its trades; they come from the fattest part of the tail, and they land on the right side of it, because the strategy is reading a price that has not yet absorbed the news.

The count understates it and the value does not. One decision in two on this input fell inside the pause, and those decisions carried 91.6 per cent of the money. A value_share above a count_share is the signature of the whole problem, which is why unfillable reports both and sums in absolute terms, so a short and a long of the same size cannot cancel into a reassuring zero.

The reopening move belongs to nobody. A name halted at 41.15 and reopened at 33.60 fell eighteen per cent with no tradable print in between. Whoever held it took the loss; whoever "entered" during the pause entered at the stale price and was marked at the new one, which is not a trade. reopen_gaps prices every one of those.

Nothing is inferred. There is no rule here that a long enough quiet stretch is a halt — on an illiquid name that rule fires constantly, and the resulting statistic is a property of the threshold rather than of the market. Either the windows are supplied, or the report says it has none. Symbols with no halt record at all are named rather than assumed clean.

$ mdnorm halts trades.csv --halts halts.csv --decisions fills.csv
events               30
halts                1 across 1 symbol(s)
halted time          10m
longest halt         10m
share of covered     25.64%
reopening moves      (no tradable price existed across these)
  AAA 10m  41.15 -> 33.6  -1834.8 bps
decisions            4
  unfillable         2 (50.00%)
  by value           91.60%
note: the value share exceeds the count share, which means the decisions taken
while halted were the large ones.

Prints stamped inside a halt window are counted and not interpreted: usually a late report of a pre-halt execution, occasionally a cross that is allowed to print, sometimes a vendor with a broken clock.

A price that stopped moving is not a price that stopped being risky

align has warned since it was written that a frozen price is uncorrelated with everything and therefore reads as diversification. It gave no way to find out how much of that you have:

from mdnorm import staleness_report, smoothing_bias

rep = staleness_report(marks, min_run=3)
rep.unchanged_share, rep.longest_run    # 75%, 31 observations

bias = smoothing_bias(returns)
bias.weight_current                     # 0.721 — the rest arrives tomorrow
bias.volatility_understated             # 0.773x
bias.sharpe_inflation                   # 1.294x

Repeated values are a fact; staleness is an interpretation. An illiquid instrument genuinely does not trade for an hour, and a vendor carrying yesterday's mark forward produces identical rows. Nothing here decides which you have — runs and staleness_report count, and what makes a flat stretch suspicious is the instrument and the sampling interval, which is why there is no default min_run.

Smoothing is where the money is. A reported series that partly reflects the previous period's move is a moving average of the true one, and a moving average has lower variance than what it averages. Lower measured volatility with the same mean is a higher Sharpe, a lower beta and a smaller correlation with everything else — four numbers moving the flattering way from one cause, with nothing raising an objection.

The adjustment is a model and says so. smoothing_bias assumes the reported return is a two-period weighted average, infers the weights from the first-order autocorrelation and reports the implied understatement. The result carries modelled=True, so it can never be confused with the run counts, which are arithmetic. A negative autocorrelation is bid-ask bounce rather than staleness and returns weights of one and zero rather than pretending.

Where the model cannot fit, it refuses. A two-period average cannot produce a first-order autocorrelation above one half, so a larger value is evidence of something else — a trend, a longer memory, an overlapping sampling window. fits comes back False and no figure is offered.

$ mdnorm staleness returns.csv --returns
observations         2999
unchanged            0 (0.00%)
autocorrelation      +0.3368
weight on today      0.721
variance reported    0.5975 of the truth (modelled)
volatility           0.7730x understated
Sharpe               1.2937x overstated

Nothing is unsmoothed in place and no repeated value is dropped. Both would be corrections applied to data whose cause has not been established.

How many observations you actually have

A five-day forward return sampled every day gives you a thousand rows and about two hundred pieces of information. Every t-statistic, Sharpe, confidence interval and p-value computed on the thousand is overstated by roughly the square root of five, and nothing in the arithmetic complains:

from mdnorm import label_spans, effective_sample_size, deflate_t_stat

sample = effective_sample_size(label_spans(1_000, horizon=5))
sample.nominal, sample.effective     # 1000, 200.8
sample.inflation                     # 2.232x on every t-statistic
deflate_t_stat(Decimal("2.1"), sample)   # 0.941

forward_returns produces exactly those overlapping labels and purged_splits already removes the training rows whose windows reach into a test block. This is the other half of the same problem: purging stops the overlap leaking across a split, and nothing stops it inflating the sample within one.

Overlap is arithmetic, not an estimate. Given each label's window you can count how many are live at every point. A label sharing its window with four others is worth a fifth of an observation; summing that gives the effective count exactly, with no model and no assumption. uniqueness exposes the per-label figure and concurrency the step function underneath it.

Autocorrelation is an estimate, and it says so. For a return series with no explicit windows there is only the sample autocorrelation, which is itself noisy, so effective_sample_size_series marks its answer estimated. The sum stops at the first non-positive autocorrelation — past that the terms are noise whose signs cancel arbitrarily and can produce an effective sample larger than the real one, which is the one direction this module exists to rule out. On an AR(1) it lands on the closed form (1-φ)/(1+φ), which the test suite checks.

No default lag, no default horizon, no silent correction. How far the dependence reaches is a property of your data. deflate_t_stat returns the adjusted figure and the report keeps both counts, because a statistic that has quietly been divided by something is harder to argue with than one that shows its working.

$ mdnorm independence --count 1000 --horizon 5 --t-stat 2.1
nominal sample       1000
effective sample     200.80  (exact)
ratio                20.1%
t-statistic inflated 2.232x
t-statistic adjusted 0.941
note: that crosses the conventional two-sigma line in the wrong direction.
The overlap did it, not the strategy.

The crosses are not points on the tape

The opening and closing auctions are single prints, at a single price, aggregating orders that never met each other in a book. Treating them as ordinary trades is wrong in a direction that flatters:

from mdnorm import auction_windows, auction_report, vwap_gap

windows = auction_windows(days, calendar)     # from the calendar, not a constant
auction_report(trades, windows).volume_share  # what went through the crosses
vwap_gap(trades, windows).difference_bps      # what that does to the benchmark

An auction print has no aggressor. Nobody crossed a spread; a clearing price was computed. The tick rule and the quote rule will still return a side, because those functions always return a side, and it is an artefact of where the last continuous print happened to sit.

Execution benchmarks are where it costs money. A VWAP with the closing cross in it is dominated by one print. A strategy that never touches the auction, scored against that benchmark, is being measured against a price it could not have obtained; a strategy that only trades the auction beats it by construction. vwap_gap reports both numbers and the distance between them.

Auctions are not inferred. No condition-code guessing, no rule that a print ten times the median must be a cross — on a busy day that rule reclassifies ordinary blocks and the resulting statistic describes the threshold rather than the market. Windows come from a TradingCalendar, so a half-day's cross lands where the venue actually closed rather than three hours later. The window extents default to zero: wide enough for a print stamped at the bell and nothing else, because "thirty seconds, everyone uses that" is a constant that differs by venue and by decade.

Nothing is deleted. split_auctions hands back both halves, and largest_print_share is measured even with no windows at all — a file where one print is a tenth of the day has a cross in it whether or not anything has been told where.

$ mdnorm auctions trades.csv --calendar us_2026.csv --session 09:30-16:00 \
      --tz America/New_York --ts-unit ns
trades               3730
  in an auction      20
volume in auctions   67.86%
notional in auctions 67.89%
VWAP with auctions   100.103008
VWAP without         100.008100
benchmark difference +9.49 bps

Stored in nanoseconds is not measured in nanoseconds

Every timestamp here is an integer nanosecond. That is a storage decision. A vendor that stamps to the millisecond and hands you nanoseconds has multiplied by a million, and the extra six digits are zeros wearing the clothes of precision:

from mdnorm import detect_resolution, classification_risk

res = detect_resolution(ts)
res.granularity_ns, res.overstated_digits   # 1_000_000, 6
res.tied_share                              # 76% of rows share a timestamp

risk = classification_risk(events)
risk.same_tick, risk.changed                # exposure, and what it moves

Resolution is detectable, and detection is divisibility. A millisecond feed leaves every timestamp a multiple of a million. The module walks the decimal ladder — nanosecond up to a second — and reports the coarsest unit that divides everything. Only decimal units, because those are the units a clock is read in; a divisor of 2,000,000 would be arithmetic rather than a statement about the venue.

Divisibility needs enough observations to mean anything. Twenty distinct timestamps all being multiples of ten is a one-in-10²⁰ coincidence on a real nanosecond feed, so twenty is plenty — three is not. Below the threshold the answer is undetermined, which is not the same answer as one nanosecond. The threshold counts distinct values, since a thousand copies of one round number is one reading of the clock.

A tie is not an ordering. Rows sharing a timestamp are in the order the writer used — an unstable sort, a queue that interleaved, a buffer flushed however it was held. Reading sequence out of them is reading the writer.

Where it costs something is trade classification. The quote rule matches a trade against the newest quote at or before it. When that quote carries the same timestamp as the trade, which came first is not in the data. classification_risk counts those trades, then re-runs the rule against the last quote that is provably earlier and reports how many sides actually move.

$ mdnorm resolution trades.jsonl
distinct timestamps  3000
resolution           1ms
padding digits       6
tied timestamps      3704 (76.34%)
largest tie          2
trades               1852
  same-tick quote    1852 (100.00%)
  side would change  815 (44.01%)

Nothing here re-sorts, jitters, or invents a finer timestamp. The resolution you have is the resolution you have.

The shape of the day, fitted without the rest of the year

Volume, spread and volatility all follow a curve inside a session — heavy at the open, thin at midday, heavy into the close. Any statistic computed across a day without removing that curve is mostly measuring the time of day:

from mdnorm import Sample, US_EQUITY_RTH, session_profile, deseasonalise

shape = session_profile(volumes, US_EQUITY_RTH, bucket_ns=5 * 60 * 10**9)
shape.factor_at(0)                      # 2.09x — the first five minutes
shape.sessions, shape.excluded          # what the curve is actually built on

adjusted = deseasonalise(volumes, US_EQUITY_RTH,
                         bucket_ns=5 * 60 * 10**9, min_sessions=20)

Removing the shape is the easy half; not reading the future while you do it is the hard half. The usual recipe fits one profile over the whole sample and divides every day by it, including the first. That profile contains days that had not happened yet, so an unusually heavy open in January is judged against a curve that already knows December. expanding_profiles gives each session a profile built only from the sessions before it, and deseasonalise uses it. full_sample_deseasonalise is the other one, kept deliberately: it is the better estimate for describing a market and the wrong input to anything that trades, and shipping both is what lets profile_leak measure the difference instead of arguing about it.

No default bucket size. Five minutes over a 6½-hour session is 78 buckets; the same five minutes on a venue that never closes is 288. Finer buckets describe the curve better and put less evidence in each, and where that trade sits is a property of your data.

A thin bucket reports nothing rather than the average. Filling it with the overall mean makes the adjusted series look well-behaved in exactly the places where nothing is known about it, so min_observations sets the bar and the bucket comes back empty below it. A point with no factor leaves the output; a point silently divided by one would be a point claiming to have been adjusted.

A short session is not a quiet one. Bucketing by offset from the open puts a half-day's closing surge into a bucket that is mid-afternoon on every other day. Given a TradingCalendar, early closes are left out and counted.

$ mdnorm seasonality volume.csv --session 09:30-16:00 --tz America/New_York --bucket 5m
sessions used        60
buckets              78 of 5m
heaviest bucket      +6h25m into the session (2.09x)
lightest bucket      +3h55m into the session (0.47x)
comparable samples   3120 of 4680
factor differs by    >0.01: 2533 (81.19%)
median gap           3.59%
largest gap          30.46%

Features that cannot see the future

With the matrix built, the next step is turning prices into returns, z-scores, volatility and correlations. This is the second place look-ahead gets in, and it gets in just as quietly:

from mdnorm import ReturnMethod, column, returns, rolling_zscore, realized_volatility

px  = column(rows, "BTC")
r   = returns(px, method=ReturnMethod.LOG)
z   = rolling_zscore(px, window=60)          # trailing, never full-sample
vol = realized_volatility(r, window=60)      # per period until you annualise it
qty = rolling_sum(volumes, window=60)        # O(n), not O(n x window)

A full-sample z-score is look-ahead. Subtracting the mean and dividing by the standard deviation of the whole series gives every observation knowledge of the distribution it sits in — including the part that had not happened yet. It is one line of code and it is everywhere. Every statistic here is trailing: the value at i comes from values[i-window+1 : i+1] and nothing else. There is a test that pins this as a property — change the tail of the input and every earlier output must be byte-identical — and a second test that shows the full-sample form failing it.

A partial window is not a result. Until the window fills you get None, not a twenty-period statistic computed from three observations. A gap inside a window propagates for the same reason: stepping over the hole would compute a twenty-period number from nineteen and label it twenty.

A frozen series has no z-score and no correlation. Zero dispersion makes both undefined, so both return None rather than 0. Reading that zero as a correlation is how a dead feed becomes an apparent diversifier.

There is no default annualisation factor. √252 is right for daily bars on a 252-day calendar and wrong for almost everything else. realized_volatility returns per-period volatility unless you pass a factor, and periods_per_year makes you state the calendar rather than assume one — the same minute bars are 525,600 periods a year on a continuous venue and 98,280 on a cash equity session.

The trailing sum is slid, and only where sliding is exact. rolling_sum and rolling_mean no longer resum the window at every index, which took them from O(n × window) to O(n) — window 250 now costs the same as window 60, 21× faster than before. The reason libraries avoid this is drift: a running Decimal total rounds differently from a fresh sum. That rounding is observable, so every update runs with the Inexact flag cleared and is thrown away the moment it would round, falling back to a full sum of the window. On ordinary data every output is unchanged; where one does change it is because the forward recomputation rounded an intermediate partial and the slid total did not, and the test suite checks against rational arithmetic that the slid answer is the exact one. The variance pass inside rolling_std and rolling_zscore is deliberately not slid — the identity that would allow it is a different sequence of roundings, and that is a trade we decline. BENCHMARKS.md has the numbers and the argument.

$ mdnorm features matrix.csv --returns log --zscore 60 --vol 60 \
      --interval 1m --sessions-per-year 365 --session-length 24h -o feats.csv

Labels, and a split that does not leak

A label is the one series in a research dataset that is allowed to look forward — it is the thing you are predicting. That makes it the series which quietly contaminates every split it touches:

from mdnorm import forward_returns, purged_splits

y = forward_returns(prices, horizon=5)
for split in purged_splits(len(prices), n_splits=5, horizon=5, embargo=60):
    train, test = split.train, split.test

A label with a horizon makes neighbouring rows overlap. If the label at row i spans the next five bars, rows i through i+5 all describe the same stretch of future. Put row i in train and row i+3 in test and the model has already seen most of the answer. Shuffling does not help — the rows genuinely are different rows, they merely share an outcome. purged_splits drops the training samples whose label window reaches into each test block, and reports how many it dropped.

A gap after the test block is not enough, because features have memory. A rolling statistic computed just after a test period is built partly from observations inside it. The embargo removes the training rows immediately following each block; set it to at least your longest feature window. It defaults to 0 because the right value is a property of your features, not of this function.

forward_returns looks forward on purpose. It is the only function in the library that does, which is why it lives in mdnorm.labels rather than mdnorm.features. Its output belongs on the left-hand side of a model; feeding it back in as an input is not a subtle mistake.

The purging and embargo scheme follows López de Prado, Advances in Financial Machine Learning (2018), ch. 7.

$ mdnorm labels feats.csv --column BTC --horizon 5 --splits 5 --embargo 60 -o ml.csv

The instruments that existed then

Two of the biases this library guards against are about time. The third is about membership: a universe assembled today did not exist in the past.

from mdnorm import Universe, Listing, cross_section, cross_sectional_rank

pit = Universe([Listing("AAA", listed_ns=...), Listing("BBB", listed_ns=..., delisted_ns=...)])
ranks = cross_section(rows, cross_sectional_rank, universe=pit)

Survivorship bias produces no strange values anywhere. Take the names listed and liquid now, pull their history, rank them against each other over ten years, and every instrument in the study is one that survived. Unlike a look-ahead bug there is nothing odd to spot — the numbers are all real, the sample is just wrong.

Excluding a name is not the same as it having no data. A symbol that has not listed yet, or delisted last month, belongs outside the cross-section rather than inside it as a blank — because a blank gets treated as missing at random, and the instruments that disappear from a market are the opposite of random. mask_to_universe returns the number of cells it removed; over a long window a count of zero usually means the listings file is present-day membership.

The size of the cross-section changes, and that is correct. Percentile ranks are computed against the members present at that moment, so the denominator moves as instruments list and delist.

Ties share an average rank, missing names are ranked neither last nor middle, and a flat cross-section has no z-score rather than a row of zeros.

$ mdnorm universe matrix.csv --listings listings.csv --pct-rank -o pit.csv
$ mdnorm revisions gdp.csv -o published.csv

Two feeds that disagree

quality inspects one feed and reports what looks wrong inside it. A second feed asks the question desks actually use to decide whether a vendor can be trusted.

from mdnorm import reconcile, suggest_shift

report, mismatches = reconcile(primary, secondary,
                               relative_tolerance=D("0.0001"))
report.agreement            # of the shared timestamps, how many matched
report.coverage_difference  # timestamps only one of them had

The two kinds of disagreement are not one number. A timestamp one feed has and the other does not is a coverage difference — a dropped message, a filtered print, a venue one side does not carry. A timestamp both have with different values is a content difference, and at least one of them is wrong about something checkable. agreement is computed over shared timestamps only, so a feed that simply carries less does not look like a feed that lies.

There is no default tolerance. Two feeds of the same instrument differ in the last digits for reasons that are not errors, and a constant deciding how much is acceptable is a judgement about your data rather than a property of it. With none given, values must match exactly — the strictest reading, and one that states its own assumption.

Zero overlap almost never means total disagreement. It usually means a clock offset: one feed stamps at the venue, the other on receipt, exact matching finds nothing in common, and the naive conclusion is that the feeds are unrelated. suggest_shift looks for the constant offset that lines them up and reports how much of the sample it would explain. It does not apply it — a clock difference is a fact about two systems that somebody should confirm.

$ mdnorm reconcile primary.csv vendor.csv --relative 0.0001 -o breaks.csv

Who was in the index, and when they were told

universe applies a membership record. Producing one from the files a vendor actually ships is a separate job, and it is where survivorship gets in.

from mdnorm import MembershipHistory, Basis, survivorship_gap

history = MembershipHistory.from_changes(changes)
history.members_at(t, basis=Basis.EFFECTIVE)   # who was in the index
history.report()                                # what the file cannot say
survivorship_gap(history, t)                    # what a today-list would cost

Two dates, and they answer different questions. An addition is announced on one day and takes effect on another. Who was in the index is the effective date; when could anyone have known is the announcement. Rank on one and trade the other and the file will never object, because both columns are correct. Basis has no default, so the question has to be named.

A snapshot cannot express a deletion. Names that leave do not appear as departures, they simply stop being listed, and the last file that showed them is not the day they left. from_snapshots therefore dates each inferred change at the later snapshot — never claiming membership earlier than the file supports — and records the window it really fell inside. On a monthly file that window is a month, which is longer than many holding periods.

A today-list is the classic error, and it is measurable. survivorship_gap returns both directions: the names a today-list drops (they left, usually not for good reasons) and the names it holds too early (they had not joined yet). The two do not cancel — one removes losers and the other adds winners — which is why the effect is large and one-directional.

The report names the tell. If nothing ever left, the file is a list of today's members wearing a history's clothes, and mdnorm membership says so out loud rather than computing quietly on it.

$ mdnorm membership index_changes.csv --at 1770000000000000000

Values that get corrected later

Every observation so far has had one timestamp: when it happened. A lot of real data has two — the period it describes, and the moment it became knowable — and then it gets revised.

from mdnorm import Revision, RevisionSeries

series = RevisionSeries([
    Revision(event_ts_ns=q1, known_ts_ns=april, value=D("2.1")),
    Revision(event_ts_ns=q1, known_ts_ns=may,   value=D("1.6")),   # revised down
])
series.as_of(event_ts_ns=q1, known_ts_ns=april_20)   # 2.1 — what you knew
series.final(event_ts_ns=q1)                          # 1.6 — what is true now

Using the corrected value is look-ahead, and no timestamp check will catch it. The row is dated correctly. The value is a real number that was genuinely published. Nothing marks it as unavailable until three weeks later. Every guard in mdnorm.align passes and the study is still wrong.

Two honest questions, two different objects. What was the newest published number at time t is a feature — known_series() is keyed by publication time and joins like any other stream. What did the whole table look like at time t is a vintage — vintage_at(t) is keyed by event time and reproduces the sheet as it appeared that day. Reading a vintage at the wrong moment gives a value nobody had; there is a test that shows exactly that.

Measure it rather than assuming. revision_summary() reports how many events were ever revised and how far first releases sat from final values. If that number is large, every backtest built on final data has been reading answers.

$ mdnorm revisions gdp.csv -o published.csv

A daily number on an intraday grid

A daily close, a settlement price, an overnight risk figure — slow series meet fast grids constantly, and they almost always arrive labelled with the period they describe rather than the moment they became knowable.

from mdnorm import PeriodSeries, leak_report, US_EQUITY_RTH, grid

series = PeriodSeries.from_sessions(daily_closes, US_EQUITY_RTH)
feature = series.knowable_series()        # keyed at the close — safe to join
report  = leak_report(series, grid(...))  # what the label join would have cost

A daily bar labelled Tuesday is not knowable on Tuesday morning. It is knowable once Tuesday's session closes — Tuesday evening, and later still if the number has to be published. Join it by its label and every minute of Tuesday sees a value that summarises, among other things, the rest of Tuesday.

The session decides the close, not the file. Daily bars are frequently stamped midnight to midnight regardless of when the market was open. from_sessions and from_daily_bars take a Session, so a 16:00 New York close is 21:00 UTC in January and 20:00 in July without the caller thinking about it.

Publication lag is a separate claim. A settlement price exists at the close; it reaches you when the vendor sends it. publication_lag_ns is where that goes and it defaults to zero, because a lag of zero is a statement about your feed that only you can make.

Measure the leak instead of arguing about it. leak_report counts the grid points where the label join shows a value that did not yet exist, and how far ahead the worst one was read. On back-to-back periods the answer is every point: the moment one value becomes readable the label has already moved to the next. Whether that ruins a study depends on the signal, which is exactly why the number is worth having.

$ mdnorm mixfreq daily.csv --interval 60000000000 --lag 900000000000 -o joined.csv

How much of the result is the search

Everything above is about getting the data right. The last step is a correct dataset that still produces a misleading number, because the number was chosen. A Sharpe ratio from one strategy is an estimate; the same ratio kept after trying two hundred parameter sets is a maximum, and the maximum of two hundred draws from noise is not small.

from mdnorm import sharpe_report

rep = sharpe_report(daily_returns, periods_per_year=D(252),
                    trials=500, trial_sharpe_variance=D("0.004"))

rep.sharpe_annualised   # 0.56  — the figure that goes in the deck
rep.probabilistic       # 0.92  — probability the true ratio is above zero
rep.deflated            # 0.008 — after accounting for 500 attempts
rep.demonstrated        # False — the sample is shorter than it needs to be
rep.warnings            # what the headline number does not say

Ratios are per period until you state the calendar. sharpe_ratio divides mean by standard deviation and stops; annualise_sharpe needs a factor, for the same reason realized_volatility does. Being wrong by a constant is the hardest kind of wrong to notice, because the shape of the series is unchanged.

A short track record is not evidence. min_track_record_length says how many periods a ratio needs before it is distinguishable from the benchmark. A strategy whose minimum is nine years and whose backtest is eighteen months has not been demonstrated, however good the ratio looks.

Selection is measurable. expected_max_sharpe(trials, variance) is the best ratio a search of that size produces from strategies that are all worthless. deflated_sharpe_ratio measures your result against that instead of against zero, following Bailey and López de Prado (2014). Pass the whole search, not the survivors.

Nothing here returns a flattering placeholder. A series that never moved has no Sharpe, a sample with no losing period has no measurable downside, a curve that never fell has no drawdown — all None, not zero and not infinity. Each of them is a statement about the sample being short.

$ mdnorm metrics pnl.csv --column ret --interval 1d \
    --sessions-per-year 252 --session-length 6h \
    --trials 500 --trial-variance 0.004

What the trade costs

mdnorm.execution measures what your fills actually cost. This is the other question: what a backtest should charge itself for a trade it never made. It is the crudest way a result flatters you and it survives every other check, because nothing in the data is wrong — the strategy is simply being priced at a level nobody trades at.

from mdnorm import CostModel, Fees, ImpactModel, Liquidity, estimate, capacity

model = CostModel(fees=Fees(taker_bps=D(1)),
                  impact=ImpactModel(coefficient=D("0.5")))   # no default
liq = Liquidity(adv=D(1_000_000), volatility=D("0.02"), spread_bps=D(4))

b = estimate(model, notional=D(500_000), quantity=D(20_000), liquidity=liq)
b.commission_bps   # 1.0
b.spread_bps       # 2.0   — half of the quoted spread, because you crossed
b.impact_bps       # 14.1  — 2% of daily volume, square-root law
b.total_bps        # 17.1

capacity(D(20), model=model, liquidity=liq)   # 28,900 a day at a 20 bps edge

Zero cost is not a default, it is a claim. A backtest that charges nothing has asserted that it trades at the midpoint, in unlimited size, for free. Written down that way nobody would sign it.

A cost that does not depend on size is not a cost model. A flat five basis points says a strategy can trade a thousand dollars and a billion on identical terms, so every capacity question has the same answer. estimate says so in its warnings every time an impact model is absent.

There is no default impact coefficient. The square-root law is well supported; the constant in front of it is not universal, and a plausible wrong one rescales every cost in the report while changing nothing about its shape. Calibrate it against your own fills — that is what evaluate is for.

The useful output is not the cost. breakeven_participation is the fraction of daily volume at which the edge is exactly consumed, and capacity is the same figure as a quantity. A two-basis-point edge that breaks even at 0.3% of volume is a different object from the same edge breaking even at 30%, and no Sharpe ratio distinguishes them.

$ mdnorm costs pnl.csv --column ret --turnover-column turnover \
    --cost-bps 5 --edge-bps 20 --adv 1000000 --volatility 0.02 \
    --spread-bps 4 --fee-bps 1 --impact-coefficient 0.5

A ticker is not an identifier

canonical_symbol makes BTCUSDT and XBT/USD agree on a spelling. This is the other problem: the same spelling, at two different times, meaning two different things. Exchanges reuse ticker strings — a company delists and its symbol is reassigned, a venue renames a pair and the old name reappears elsewhere.

from mdnorm import SymbolAssignment, SymbolMap, key_by_instrument, series_segments

smap = SymbolMap([
    SymbolAssignment("ABC", "US0000000001", start_ns=t0, end_ns=t1),
    SymbolAssignment("ABC", "US0000000002", start_ns=t2),   # reused later
])

smap.reused_symbols()                 # [("ABC", 2)] — the finding
smap.instrument_at("ABC", t_mid)      # None: in the gap it named nothing
rows, counts = key_by_instrument(rows, smap)
counts["reassigned"]                  # rows the string would have mis-joined
segments, unresolved = series_segments("ABC", timestamps, smap)

The bias is a join, not a bad value. Every price in a spliced series genuinely traded, at its own timestamp, under the ticker it carries. What is wrong is the assumption that the column header names one thing — made once, silently, when the matrix is built.

Reuse looks like a merger, and mergers look profitable. A delisting is usually a fall and a new listing starts at a normal price, so splicing one onto the other inserts a jump. Half the time it is upward, and an upward jump in a name you were holding is indistinguishable from a takeover premium. The series does not look broken; it looks lucky.

A gap is not filled with the next owner. Between the delisting and the reassignment the ticker named nothing, and instrument_at returns None there rather than the instrument that took the letters afterwards. That substitution is the splice.

Overlaps are refused. One ticker bound to two instruments at the same moment is a broken reference file, and picking one of them quietly is how the error reaches a study. SymbolMap raises instead.

No reuse in a long history is a finding, not a pass. A file with one open-ended binding per ticker cannot express reuse at all, so a zero means the file rather than the market — the same shape of diagnostic as a purge that removes nothing.

$ mdnorm instruments symbol_map.csv trades.csv --segments ABC -o keyed.csv

Data quality

from mdnorm.quality import find_issues, clean

find_issues(events)          # list of QualityIssue (outlier / gap / out_of_order / non_positive)
cleaned, issues = clean(events)  # drop bad ticks & invalid rows, keep a report

clean removes price outliers and non-positive price/size records and returns the surviving events plus everything it flagged.

Serialization

from mdnorm import to_records

to_records(events)                 # list of flat dicts (Decimals as strings)
to_records(bars, as_float=True)    # numeric output for DataFrames

to_records (and event_to_dict / bar_to_dict) flatten events and bars into plain, JSON-serialisable dicts — drop straight into pandas.DataFrame, a csv.DictWriter, or json.dumps.

Consolidating streams

from mdnorm import merge_streams, dedupe

timeline = dedupe(merge_streams(binance_events, coinbase_events))

merge_streams interleaves multiple venue feeds into one timestamp-ordered timeline; dedupe drops exact duplicate events left behind by reconnects and replays.

CSV files

from mdnorm import read_csv_trades, write_records_csv

events = read_csv_trades("trades.csv", venue="coinbase")   # file -> events
write_records_csv(bars, "bars.csv", as_float=True)          # events/bars -> file

read_csv_trades parses a whole CSV of trades into normalized events; write_records_csv writes events or bars back out. Standard library only.

NDJSON / JSON Lines

from mdnorm import write_jsonl, read_jsonl_events

write_jsonl(events, "events.jsonl")          # one JSON object per line
events2 = read_jsonl_events("events.jsonl")  # lossless round-trip

# large files: stream lazily, .gz handled transparently
for e in iter_jsonl_events("dump.jsonl.gz"):
    ...

Pipelines

Declare a processing chain once, reuse it everywhere:

from decimal import Decimal
from mdnorm import Pipeline

pipe = (
    Pipeline()
    .dedupe()
    .clean(max_return=Decimal("0.1"))
    .time_bars(60_000_000_000)   # 1-minute bars
    .fill_gaps()
)
bars = pipe.run(events)
print(pipe.last_issues)          # quality report from clean()

Command line

The common conversions ship as a zero-dependency CLI:

$ mdnorm bars trades.csv --venue binance --interval 1m -o bars.csv
$ mdnorm quality trades.csv --max-gap 5m
$ mdnorm convert trades.csv -o trades.jsonl
$ mdnorm bars trades.csv --interval 1d --actions actions.csv -o adjusted.csv
$ mdnorm align BTC=btc.csv ETH=eth.csv --interval 1m --max-age 5m -o matrix.csv
$ mdnorm features matrix.csv --returns log --zscore 60 --vol 60 -o feats.csv
$ mdnorm labels feats.csv --column BTC --horizon 5 --splits 5 -o ml.csv
$ mdnorm universe matrix.csv --listings listings.csv --pct-rank -o pit.csv
$ mdnorm revisions gdp.csv -o published.csv
$ mdnorm metrics pnl.csv --column ret --trials 500 --trial-variance 0.004
$ mdnorm costs pnl.csv --cost-bps 5 --edge-bps 20 --adv 1e6 --volatility 0.02
$ mdnorm instruments symbol_map.csv trades.csv -o keyed.csv
$ mdnorm calendar us_2026.csv --session 09:30-16:00 --tz America/New_York
$ mdnorm fx prices.csv rates.csv --from EUR --to USD --max-age 1m -o usd.csv
$ mdnorm ticks prices.csv --table ticks.csv
$ mdnorm arrival feed.csv --interval 1s
$ mdnorm seasonality volume.csv --session 09:30-16:00 --bucket 5m
$ mdnorm resolution trades.jsonl
$ mdnorm auctions trades.csv --calendar us_2026.csv --session 09:30-16:00
$ mdnorm independence --count 1000 --horizon 5 --t-stat 2.1
$ mdnorm staleness marks.csv --min-run 3
$ mdnorm halts trades.csv --halts halts.csv --decisions fills.csv
$ mdnorm coverage feed.csv --min-gap 5m --calendar us_2026.csv
$ mdnorm provenance run.json --verify --parameter trials=500
$ mdnorm extremes pnl.csv --sigma 5 --robust --tail 10
$ mdnorm windows pnl.csv --metric sharpe --trim-start 21 --deflate

Also available as python -m mdnorm.

The unified schema

@dataclass(frozen=True, slots=True)
class MarketEvent:
    symbol: str          # canonical "BASE-QUOTE", e.g. "BTC-USD"
    venue: str           # source venue
    event_type: EventType  # TRADE | QUOTE
    ts_ns: int           # nanoseconds since Unix epoch (UTC)
    price: Decimal | None
    size:  Decimal | None
    side:  Side | None     # BUY | SELL
    # ... plus bid/ask fields for quotes

Benchmarks

Throughput for the hot paths, measured by a script in this repository rather than asserted: BENCHMARKS.md. The headline finding is that exact Decimal arithmetic costs 3.1× a float loop, not the order of magnitude usually assumed — so the cost of this library is mostly Python and the algorithm, not the exactness.

$ python bench/benchmark.py

Roadmap

What exists, what has been asked for, and what we have decided against is in ROADMAP.md — including the native Rust port two people have now asked for, with an honest account of what it would and would not solve.

Design notes

• Money is Decimal. Prices and sizes never touch binary floats, so 42000.10 stays 42000.10.
• Time is integer nanoseconds, UTC. One comparable integer regardless of whether the source gave seconds, milliseconds, or a FIX timestamp string.
• Symbols are canonicalized to BASE-QUOTE, with venue aliases resolved (XBT → BTC) and quote currencies detected longest-match-first so USDT wins over USD.
• Normalizers are pure functions — one raw record in, one MarketEvent out — which keeps them trivial to unit-test and compose into any streaming or batch pipeline.

Architecture

raw feed ──► normalizer ─────────────► MarketEvent ──► your pipeline
 (CSV /      (from_csv_row /            (unified,       (research,
  WS JSON /   from_ws_json /             immutable)      backtest,
  FIX)        from_fix)                                  execution)
                    │
                    ├── symbols.canonical_symbol()   BTCUSDT → BTC-USDT
                    ├── SymbolMap.instrument_at()    which instrument the ticker named then
                    ├── timeutil.*_to_ns()           any time → ns UTC
                    ├── adjust.adjust_events()       splits/divs/rolls
                    ├── TradingCalendar.is_open()    the holidays and half-days
                    ├── FxRates.convert()            a price in another currency
                    ├── grid_report()                is this a print or a derived number
                    ├── micro.infer_sides()          who crossed the spread
                    ├── book.OrderBook()             deltas → live book → quotes
                    ├── consolidate()                many venues → one best bid/offer
                    ├── evaluate()                   your fills vs the market
                    ├── align()                      N instruments → one time grid
                    ├── returns() / rolling_*()      features, trailing windows only
                    ├── purged_splits()              folds whose labels do not overlap
                    ├── Universe.members_at()        who was actually listed then
                    ├── RevisionSeries.as_of()       which version you had then
                    ├── sharpe_report()              and how much of it is the search
                    └── capacity()                   the size at which the edge runs out

Examples

Three runnable scripts in examples/, standard library only:

Tests

pip install pytest
pytest -q

The suite includes a cross-venue equivalence test proving CSV, WebSocket and FIX representations of one trade collapse to an identical event, and a causality property applied across the feature layer: change the tail of an input, and every output before the change must be byte-identical.

CI also runs mypy, and it is clean. The package ships a PEP 561 py.typed marker, so your type checker will use its annotations rather than ignore them.

That marker was missing for most of this project's life while the packaging metadata claimed otherwise. Removing the false claim and then earning it back took one release each; what the second one mostly consisted of was writing down invariants the code already enforced — a trade cannot exist without a price, a window past the gap guard holds no None. See CONTRIBUTING.md for where cast is and is not acceptable.

License

MIT © HarvestGroup360 (AMII LTD). See LICENSE.

Maintained by HarvestGroup360 as part of our open quantitative-infrastructure tooling.