market-data-normalizer (mdnorm)
Normalize heterogeneous market-data feeds — CSV tick dumps, exchange WebSocket JSON, and FIX — into a single, exchange-agnostic event schema, so downstream research and execution code never has to care where a tick came from.
Zero runtime dependencies. Pure Python (3.10+). Decimal prices, integer
nanosecond timestamps.
Why
Every venue spells the same thing differently: BTCUSDT vs XBT/USD,
millisecond epochs vs FIX UTCTimestamp, is_buyer_maker booleans vs side
codes. Research notebooks and backtesters end up littered with per-venue
parsing branches. mdnorm pushes that mess to the edge and hands the rest of
your stack one clean type.
Install
pip install market-data-normalizer
The distribution is named market-data-normalizer; the import name is
mdnorm:
import mdnorm
Pure Python, no runtime dependencies, Python 3.10+.
Quick start
from mdnorm import from_csv_row, from_ws_json, from_fix
# CSV row (ISO-8601 timestamp)
from_csv_row(
{"symbol": "btc/usd", "ts": "2026-01-02T00:00:00Z",
"price": "42000.5", "size": "0.25", "side": "buy"},
venue="coinbase",
)
# Exchange WebSocket trade message
from_ws_json({"s": "BTCUSDT", "p": "42000.5", "q": "0.25",
"T": 1767312000000, "m": False}, venue="binance")
# FIX execution report (SOH-delimited in the wild; "|" here for readability)
from_fix("55=BTC/USD|31=42000.5|32=0.25|54=1|60=20260102-00:00:00",
venue="lmax", sep="|")
All three calls above produce the same MarketEvent.
Quotes (bid/ask)
from mdnorm import from_ws_quote
q = from_ws_quote(
{"s": "BTCUSDT", "b": "41999.5", "B": "1.2",
"a": "42000.5", "A": "0.8", "T": 1767312000000},
venue="binance",
)
q.mid_price # Decimal("42000.0")
q.spread # Decimal("1.0")
from_csv_quote does the same for CSV rows with bid/ask columns.
OHLCV bars
from mdnorm import time_bars
bars = time_bars(events, interval_ns=60_000_000_000) # 1-minute bars
bars[0].open, bars[0].high, bars[0].low, bars[0].close, bars[0].volume, bars[0].vwap
time_bars reduces a stream of trade events into fixed-interval OHLCV Bars
(with VWAP and trade count), sorting out-of-order input and skipping quotes.
resample_bars(bars, interval_ns) downsamples bars to a coarser interval
(e.g. 1-minute → 5-minute) with correct OHLC aggregation and volume-weighted
VWAP.
fill_gaps(bars) returns a gapless series, inserting flat zero-volume bars
(OHLC = previous close) for any interval with no trades — a continuous grid for
backtests and feature pipelines.
Event-driven bars
Time bars are not the only clock. Sample by activity instead:
from decimal import Decimal
from mdnorm import count_bars, volume_bars, dollar_bars
count_bars(events, every=500) # tick bars
volume_bars(events, min_volume=Decimal("100")) # volume bars
dollar_bars(events, min_notional=Decimal("1e6")) # dollar bars
Trading sessions
Filter a feed down to the hours that matter, with daylight saving handled for you:
from mdnorm import US_EQUITY_RTH, filter_session, group_by_session_date
rth = filter_session(events, US_EQUITY_RTH) # 09:30-16:00 New York
by_day = group_by_session_date(events, US_EQUITY_RTH)
Overnight windows (a session that opens at 18:00 and closes at 17:00 the
next day) are supported, and session_date keeps a whole night in one
bucket. From the command line:
$ mdnorm bars trades.csv --interval 5m --session 09:30-16:00 --tz America/New_York -o rth.csv
Holidays, half-days, and the year that is not 252 sessions
A session is a recurring window. A calendar is that window plus the exceptions to it, and the exceptions are the part that quietly breaks things.
from mdnorm import read_calendar_csv, US_EQUITY_RTH
cal = read_calendar_csv("us_2026.csv", US_EQUITY_RTH)
cal.is_trading_day(date(2026, 7, 3)) # False, from the file
cal.close_time(date(2026, 11, 27)) # 13:00, a half-day
cal.trading_minutes_between(jan, dec) # not sessions x 390
A missing holiday looks exactly like missing data. A pipeline that does not know a date is a holiday sees a day-long gap and reports an outage, or fills it, or drops the instrument for poor coverage. All three are wrong in the same way: the data was never supposed to be there.
A half-day is not half a problem. An early close shortens the session and changes nothing else, so bars keep being cut against a 6.5-hour assumption, a volatility annualised on session length is overstated for that day, and a staleness check fires on every instrument at once an hour before it should.
A calendar cannot answer outside the range it was given. A file listing
2026 says nothing about 2027, and a calendar that treats an unknown weekday as
open turns a missing file into a confident wrong answer. Every query outside
covers raises instead — noisy exactly once, and then correct.
252 is a convention, not a count. How many sessions a year holds depends on where the weekends and holidays fell; how many minutes it holds depends on how many of those sessions closed early. Both are computable, and both rescale every annualised figure in a report while leaving its shape untouched.
$ mdnorm calendar us_2026.csv --session 09:30-16:00 --tz America/New_York
trading days 251
early closes 2
trading minutes 97530
note: early closes cost 360 minute(s) against a flat 390-minute session.
for `mdnorm features`: --sessions-per-year 251 --session-length 23400s
note: 2026 has 251 sessions here, not the conventional 252. Annualising a
volatility on 252 overstates it by 0.20%.
A price is a number and a currency
The moment a study spans venues that quote in different currencies, every figure in it depends on a second series nobody was watching.
from mdnorm import CurrencyPair, FxRates, convert_series, decompose_return
rates = FxRates({CurrencyPair("EUR", "USD"): eurusd})
usd, dropped = convert_series(prices, rates, base="EUR", to="USD",
max_age_ns=MINUTE)
There is no default conversion time. Converting at the observation's own timestamp, at a daily fix, or at the end of the study are three different questions, and only the first was available to someone standing at that moment. The last is the one that gets used by accident, because one rate is easier to obtain than a series — and it restates the whole history using a number that did not exist until the end of it. Nothing here accepts a scalar rate.
Staleness is the ordinary failure. FX stops over weekends while other
venues keep trading, so an as-of join with no age limit converts a Sunday
print with Friday's close. max_age_ns is required, and every conversion
carries the age of the rate it used.
Direction is not guessable from a name. Vendors disagree about which way round to publish a pair, and a rate applied upside-down is either wrong by a factor of thousands or — near parity — wrong by a few per cent and entirely plausible. Pairs carry an explicit base and quote, inversion is recorded in the result, and it can be refused outright.
A cross is not free, and no path is searched for. Going through a vehicle
currency multiplies two quotes and inherits both spreads and both staleness
windows. State the vehicle with via= or the conversion is refused: a library
that finds its own way through the currency graph is choosing which spreads
you pay, invisibly.
A converted return is not a converted price. `(1 + total) = (1 + asset)(1
- fx)
holds exactly; the familiar shorthand adds the two and drops the product.decompose_return` returns both and the difference between them.
$ mdnorm fx prices.csv rates.csv --from EUR --to USD --max-age 1m -o usd.csv
note: the rate moved +9.09% across this span, so a single-rate conversion
would have restated the whole series by a number that did not exist until the
end of it.
Prices live on a grid, and the grid is data
A venue does not accept any price. It accepts multiples of a tick, and the tick depends on the price band, the instrument and the year.
from mdnorm import TickTable, TickBand, grid_report, spread_in_ticks
table = TickTable([TickBand(D("0"), D("0.0001")),
TickBand(D("1"), D("0.01"))])
grid_report(prices, table).looks_raw # could the venue have quoted these?
spread_in_ticks(bid, ask, table) # 1.0 is the floor, not a tight market
A price off the grid is telling you something. Raw prints sit on the grid
by construction — the venue would not have accepted them otherwise. So a
series that does not is a mid, a VWAP, an average of venues, a back-adjusted
history, or an error, and those are indistinguishable by eye. grid_report is
one pass over the data and answers a question most pipelines never ask.
Back-adjustment takes a series off the grid permanently, and that is correct. An adjusted history is a returns object, not a price object. It stops being correct when someone rounds it back on to make it look tidy. The grid is the cheapest way to tell the two apart after the fact.
There is no default tick size. The familiar penny is wrong below a dollar
on most venues, wrong for sub-penny programmes, wrong for crypto by orders of
magnitude, and wrong for the same instrument before the last regime change —
so tick tables are point-in-time data, and TickSchedule refuses to answer
before the first one it was given.
Ties are not an edge case here. On a continuous scale an exact half is a
curiosity; on a tick grid a mid between adjacent ticks is a half-tick every
single time. Rounding has no default and no tie shortcut.
Round against yourself, or say that you did not. executable rounds a buy
down and a sell up, so the grid never makes an order more aggressive than the
strategy asked for. Rounding to the nearest tick does the opposite about half
the time, which lifts the fill rate in any backtest that fills limit orders at
their limit. The clearest case is the mid: a market quoted one tick wide has a
mid exactly half a tick from both sides, so it is not a price the venue could
ever accept, and filling there understates cost by half the spread on every
trade.
$ mdnorm ticks prices.csv --table ticks.csv
off the grid 2
note: 2 price(s) could not have been quoted on this grid, so this series is
not raw prints.
Corporate actions and contract rolls
A raw price series is not continuous. A 4-for-1 split divides the printed price by four overnight, a cash dividend drops it by the amount paid, and a futures roll steps it by the spread between the two contracts. None of them are market moves, but all of them look like returns:
from decimal import Decimal
from mdnorm import adjust_bars, split, dividend, roll, iso_to_ns
actions = [
split(iso_to_ns("2026-06-06T00:00:00Z"), Decimal("4")),
dividend(iso_to_ns("2026-05-09T00:00:00Z"), Decimal("0.25")),
]
clean = adjust_bars(bars, actions)
Back-adjustment leaves the most recent segment at the prices that actually printed and restates everything before each event, so the joins are seamless:
raw closes 500 502 498 504 │ 126 125.5 127 126.5
raw returns +0.4% -0.8% +1.2% │ -75.0% -0.4% +1.2% -0.4%
^ the split, not a crash
adj closes 125 125.5 124.5 126 │ 126 125.5 127 126.5
adj returns +0.4% -0.8% +1.2% │ +0.0% -0.4% +1.2% -0.4%
Splits scale volume as well as price. Dividends take their reference price
from the last print before the ex-date unless you pass one. Rolls support
both conventions — AdjustMethod.RATIO (default, preserves returns) and
AdjustMethod.DIFFERENCE (preserves price differences, the usual choice for
futures). Factors are composed as exact rationals, so a 1-for-2 followed by a
1-for-3 restates 600 to exactly 100 rather than 99.999...96.
Actions can come from a file, and the CLI wires it up:
$ mdnorm bars trades.csv --interval 1d --actions actions.csv -o adjusted.csv
$ mdnorm bars tape.jsonl --infer-sides --every-imbalance 500 -o imbalance.csv
$ mdnorm book deltas.csv --symbol BTC-USD -o quotes.jsonl
$ mdnorm nbbo quotes.jsonl --max-age 2s -o top.jsonl
$ mdnorm tca fills.csv --market tape.jsonl --decision-price 100
ts,kind,value,ref_price
2026-06-06T00:00:00Z,split,4,
2026-05-09T00:00:00Z,dividend,0.25,190.50
2026-03-14T00:00:00Z,roll,5312.50,5290.25
Who crossed the spread
Most trade tapes give you a price and a size but not the aggressor. That one
missing field is what separates a price series from an order-flow series, and
signed volume, order imbalance and imbalance bars are all defined in terms of
it. mdnorm.micro infers it, using the three rules the literature settled on:
from mdnorm import SideRule, infer_sides, trade_imbalance, mean_effective_spread
classified = infer_sides(events) # Lee-Ready by default
print(trade_imbalance(classified)) # -1 selling .. +1 buying
print(mean_effective_spread(classified)) # 2 * |price - mid|
SideRule.TICK compares each trade with the previous different price and
needs trades only. SideRule.QUOTE compares the trade with the prevailing
mid. SideRule.LEE_READY — the default — uses the quote rule and falls back
to the tick rule at the mid. A side reported by the venue always wins;
inference only fills gaps, and trades it cannot resolve stay None rather
than being guessed at. Published accuracy of these rules is roughly 75-85% on
liquid names, so treat an inferred side as an estimate.
roll_spread estimates the effective spread from trade prices alone, via the
serial covariance that bid-ask bounce induces. It needs no quotes, which makes
it a useful cross-check on the rest — and it returns None rather than zero
when the covariance comes out non-negative and the estimator is undefined.
Imbalance bars
Once trades carry a side, the sampling clock can follow order flow instead of time or volume:
from mdnorm import Pipeline
bars = Pipeline().infer_sides().imbalance_bars(Decimal("500")).run(events)
A bar runs until buyers have outbought sellers, or the reverse, by the
threshold. Balanced two-sided periods produce one long bar; a sustained
one-sided push produces several short ones. by="tick" measures the imbalance
in trade count rather than size. From the command line:
$ mdnorm bars tape.jsonl --infer-sides --every-imbalance 500 -o imbalance.csv
Rebuilding the order book
Exchanges do not send you a book. They send a snapshot and then a stream of deltas, and the book only exists if you apply every one of them, in order:
from mdnorm import BookDelta, OrderBook, Side, replay_book
book = OrderBook("BTC-USD", "binance")
book.apply_snapshot(ts, bids=[(D("100"), D("2"))], asks=[(D("101"), D("3"))], seq=10)
quotes = list(replay_book(book, deltas)) # one quote per change in the top
print(book.best_bid, book.spread, book.imbalance(levels=5))
Two failure modes make a reconstructed book silently untrue, and this implementation refuses to hide either.
A sequence gap means a message was missed, and no later update repairs the
damage — the book is simply wrong from then on, in a way that looks completely
normal. OrderBook raises SequenceGapError the moment a number is skipped,
naming how many updates went missing, because the correct response is to
resynchronise from a snapshot rather than carry on. Duplicated or replayed
messages are rejected the same way. Feeds without sequence numbers work fine;
pass strict_sequence=False to opt out entirely.
A crossed book — best bid at or above best ask — is not a market state but
a symptom: a dropped delete, a stale snapshot, two venues merged by mistake.
It is exposed as is_crossed, and the spread goes negative rather than being
quietly clamped to zero.
to_quote() turns the top of the book into an ordinary MarketEvent, so a
reconstructed book feeds straight into session filtering, trade classification
and effective spreads with nothing in between. From the command line:
$ mdnorm book deltas.csv --symbol BTC-USD --venue binance -o quotes.jsonl
One instrument, several venues
When something trades in more than one place, "the price" is a question. The consolidated top of book is the answer, and it is where three problems live that a maximum over venues will not warn you about:
from mdnorm import consolidate
top = consolidate(quotes, max_age_ns=2_000_000_000) # 2s staleness cutoff
A venue that goes quiet keeps voting. When a feed disconnects, its last
quote stays in the consolidation forever — and a stale price is very often the
best price, so the dead venue ends up setting the top of book. This is the
failure that produces a consolidated feed which looks excellent and is
fiction. max_age_ns retires a venue that has not spoken recently;
stale_venues() names them.
A consolidated book can appear crossed. A bid on one venue above the offer
on another looks like free money and is almost always clock skew between two
feeds timestamped by different machines. is_crossed reports it and
crossed_updates counts it, because the useful response is to check the
clocks rather than to trade the spread.
Ties need a rule. Equal best prices are broken by size, then by venue name, so the same input always produces the same output.
Which venue actually sets the price is a measurement in its own right, and
leadership counts it. The pieces compose: an order book becomes a quote,
quotes from several venues consolidate into one, and the result feeds trade
classification and effective spreads unchanged.
$ mdnorm nbbo quotes.jsonl --symbol BTC-USD --max-age 2s -o top.jsonl
Did I execute well?
Once the tape is clean the next question is about you rather than the market, and every standard benchmark has a way of quietly flattering the person running it:
from mdnorm import Fill, Side, evaluate
report = evaluate(my_fills, market_trades, decision_price=D("100"))
print(report.slippage_vs_vwap_bps, report.participation_rate)
Your own trades are in the benchmark. A VWAP over the public tape includes
the prints you just made, so you end up partly benchmarking yourself against
yourself — and the bigger your share of volume, the more the benchmark bends
toward your own average price. evaluate removes your fills from the tape
before computing anything; exclude_fills does it on its own if you want the
benchmark separately. In the library's own test suite, leaving them in turns a
100 VWAP into 109 and a losing execution into a winning one.
Participation decides whether the number means anything. Beating VWAP by two basis points on 0.1% of volume is a result; the same number on 30% of volume mostly measures your own impact. The summary always reports the two together, and the CLI says so out loud above 10%.
Sign conventions are stated, not assumed. Positive basis points always mean better than the benchmark — paying below it on a buy, selling above it on a sell. Mixed-side fills are refused rather than netted, because one number covering both directions has no meaning.
By default the window runs from your first fill to your last. That is right
for a worked order and wrong for a single fill — the only print in the window
is then your own — so start_ns and end_ns let you score against an
interval you chose instead.
TWAP skips intervals that never traded instead of carrying the last price forward, for the same reason nothing else here invents data.
$ mdnorm tca fills.csv --market tape.jsonl --decision-price 100
Several instruments, one time grid
Research wants a matrix — one row per timestamp, one column per instrument — and building it from independent tick streams is where look-ahead bias gets in, because every mistake here makes the backtest better rather than raising:
from mdnorm import Field, align
rows = align({"BTC": btc_events, "ETH": eth_events},
interval_ns=60_000_000_000, # a one-minute grid
max_age_ns=5 * 60_000_000_000) # nothing older than 5 minutes
rows[0].values # {"BTC": Decimal("60000"), "ETH": Decimal("3000")}
rows[0].ages_ns # how old each value was at that grid point
rows[0].complete # False if any column had nothing to show
The join only looks backwards. A value is visible at a grid point only if it was observed at or before it. "Nearest observation" is the expensive default in this area: on a one-minute grid it lets a print from 09:30:20 be read at 09:30:00, and twenty seconds of hindsight is enough to make a mediocre signal look tradeable.
A bar labelled 09:30 is not knowable at 09:30. It contains everything that
traded until 09:31, so joining bars on their label imports an interval of the
future. AsOfSeries.from_bars timestamps each bar at its end, and
align_bars therefore gives you the last closed bar per column — one
interval further back than the naive join, and the version you could have
traded.
Forward-filling has no natural end. A halted or delisted stream otherwise
contributes its last price forever, and a frozen price correlates with nothing,
which reads as diversification. With max_age_ns a quiet column becomes
None; the age is still reported, so row.stale (had data, too old) and
row.missing (never had data) stay distinguishable.
A feed you get late was not available on time. AsOfSeries.delayed(250ms)
shifts observation times forward by the delivery delay, so alignment reflects
when you could have acted rather than when the source stamped it. If you do
not know what your delay is, the next section measures it.
Nothing interpolates or smooths. align_on takes timestamps you supply, for
one row per print of a reference instrument, per signal, or per fill.
$ mdnorm align BTC=btc.csv ETH=eth.jsonl --interval 1m --max-age 5m -o matrix.csv
When the venue says it happened, and when you found out
AsOfSeries.delayed has always taken a delay and said, in its own docstring,
that a delay of zero is a claim about your infrastructure rather than a
default. It never offered a way to find out what yours is. This is that half:
from mdnorm import Arrival, delay_report, as_received, as_stamped, view_gap
report = delay_report(arrivals) # what the transport actually costs
report.median_ns, report.p95_ns # the typical case, and the one to size for
report.negative # rows received before they happened
report.out_of_order # messages that overtook the one before
knowable = as_received(arrivals) # the series you could have acted on
optimistic = as_stamped(arrivals) # the series research usually builds
view_gap(arrivals, grid).share # how often those two disagree
A venue timestamp is not an arrival. Keying research on it claims the information reached you instantly, and the error only ever points one way: every signal looks actionable slightly earlier than it was, every cross-venue lead is inflated by the difference in transport, and a fill is priced at a quote that had not reached the machine placing the order. Nothing fails. The result is simply better than it should be.
There is no default delay. A file with no receipt column gets no invented
one. State what you decided to believe with assume_delay_ns= and the report
comes back with assumed=True — a report that hides which of the two it used
is worse than no report.
A negative delay is a fact, not an outlier. Receipt before the venue stamp means the clocks disagree, which is usually the more interesting finding. It is counted separately and never clamped, because clamping turns a clock problem into a latency figure that looks fine.
The mean latency is the least useful summary there is. A transport
distribution has a tail and the mean mostly measures it. The report gives the
median and the p95 by nearest rank, so every figure it prints is a delay that
actually happened, and tail_ratio says whether the typical case and the bad
case are the same problem.
view_gap asks both views what they knew at each grid point and reports where
they differ, with largest_gain_ns — the most unearned foresight found. That
number is the one to compare against the horizon your signal acts on: a
quarter of a second is nothing to a daily rebalance and everything to a queue
position.
$ mdnorm arrival feed.csv --interval 1s
observations 184203
median delay 412us
p95 delay 3.9ms
p95 / median 9.47x
received before sent 0
out of order 37
for `AsOfSeries.delayed`: by_ns=412000 for the typical case, 3900000 for the
case worth sizing against.
Alpha is what is left after the things you already knew about
A strategy with a Sharpe ratio worth reporting is sometimes a strategy, and sometimes it is a factor everybody can already buy, wearing a new name. The return series of an exposure and the return series of an edge look identical:
from mdnorm import factor_regression, alpha_stream, sharpe_ratio
rep = factor_regression(strategy, {"market": mkt, "momentum": mom,
"value": val})
rep.r_squared # 0.8562
rep.alpha # 0.00010503 per period
rep.alpha_t_stat # 1.223 — not distinguishable from zero
rep.alpha_share # 0.18 — 18% of the mean return survives
sharpe_ratio(strategy) # 1.16 annualised
sharpe_ratio(alpha_stream(strategy, facs)) # 0.55 annualised
Five years of daily returns. The headline Sharpe is 1.16 and the market carries 0.00044607 of the 0.00058426 mean — three quarters of the return. What is left over is worth 0.55, and its t-statistic is 1.2.
The absence of an exposure is not evidence of alpha. It is evidence about your factor list. A residual that no factor explains means the strategy is orthogonal to the factors you supplied, which is a much smaller claim than the one people make with it. This module will never tell you a strategy has alpha; it tells you how much survives a list you chose.
No factor data ships with this library and none ever will. Bundling a
factor set would make every answer partly a property of whose definition of
momentum we vendored, and ROADMAP.md already rules out tying the
library to one feed. Bring your own series, and record where they came from —
mdnorm.provenance exists for that.
One trap is in the arithmetic rather than the data. A least-squares
residual computed with an intercept has a mean of exactly zero, always, so a
Sharpe ratio on it is zero whatever the alpha was. residuals returns that
series, for looking at the shape of what the factors missed; alpha_stream
returns the strategy with the factor contributions removed and the intercept
kept, whose mean is the alpha. That is the series to put a ratio on, and it
is the one nobody computes.
$ mdnorm exposure returns.csv --strategy strategy
observations 1260
factors 3
R squared 0.8562
mean return 0.00058426
alpha 0.00010503
t-statistic 1.223
share of the mean 18.0%
loadings
market beta 0.6851 t 81.34 carries 0.00044607
momentum beta 0.4157 t 28.19 carries 0.00003139
value beta 0.0119 t 0.68 carries 0.00000177
dominant factor market (0.00044607)
Sharpe, as reported 0.0729
Sharpe, alpha only 0.0346
dominant_factor ranks by the return a factor carried, not by its beta and
not by its t-statistic: a large loading on a factor that went nowhere carries
nothing, and a significant coefficient is a statement about precision rather
than about magnitude.
Choosing a factor list after seeing the strategy is a search. Four
candidates in every combination are fifteen regressions, and the one with the
flattering residual is the best of fifteen. mdnorm.multiverse will enumerate
that grid and hand you the count. The loadings are also full-sample and
constant — a strategy that was fully exposed in the first half and flat in
the second reports an average beta describing neither half, so run the
regression across mdnorm.windows and watch whether the betas move.
Five hundred names is not five hundred bets
mdnorm.independence counts how many independent observations an
overlapping-label study really has, along the time axis. This asks the same
question across the cross-section, and almost nobody asks it:
from mdnorm import correlation_matrix, breadth_report
rep = breadth_report(correlation_matrix(returns_by_symbol))
rep.names # 40
rep.average_correlation # 0.5035
rep.effective_bets # 3.628
rep.overstatement # 11.02x
rep.ratio_overstatement # 3.32x
Breadth enters performance arithmetic under a square root, so the error compounds twice. The fundamental law puts an information ratio at the information coefficient times the root of the number of independent bets. Forty names driven by one market are not forty bets and not twenty — they are three and a half, and a ratio computed on the position count is overstated by a factor of three.
The error is silent because the position count is a fact. There really are forty names, the trades really happened, the reconciliation really balances. Nothing in the accounting is wrong. What is wrong is the claim implied by quoting a statistic against forty, and no line of the books contradicts it.
Two counts are reported and they are not two estimates of one thing.
effective_bets is the participation ratio of the eigenvalues, (Σλ)²/Σλ² —
how concentrated risk is across independent directions. effective_observations
is n / (1 + (n-1)ρ̄) — what an average of n correlated series is worth as a
sample size, which is the number a cross-sectional t-statistic needs, and
BreadthReport.as_sample() hands it straight to deflate_t_stat. They
coincide only at the extremes: both give n for the identity and one when every
correlation is one. Three names at ρ = 0.5 are two bets and one and a half
observations, and as n grows at fixed ρ the first tends to 1/ρ² and the
second to 1/ρ. Quote the one that matches the claim being made.
$ mdnorm breadth panel.csv --eigenvalues --list-limit 5
names 40
observations 500
average correlation 0.5035
effective bets 3.628
effective observations 1.938
position count over 11.02x the bets
information ratio 3.32x overstated
eigenvalues
1 20.747915 51.87%
2 0.812768 2.03%
3 0.777735 1.94%
4 0.750583 1.88%
5 0.748383 1.87%
... (35 more)
One direction carries fifty-two per cent of the variance and the next thirty-nine share the rest. That is the whole finding, and it is visible before any of the summary numbers are computed.
A short sample flatters the count upward. With fewer observations than
names the sample correlation matrix is singular and part of its spectrum is
noise, which inflates the apparent number of bets. thin_sample reports the
condition; nothing here corrects for it, because the correction needs a model
of the return process and this library does not have one.
The eigenvalues come from a cyclic Jacobi rotation written for Decimal, so
there is still no runtime dependency. It stops when the off-diagonal mass
reaches the round-off floor of the working precision and raises if the floor
it reaches is large enough to matter, rather than returning a
half-diagonalised answer dressed up as a spectrum.
Every cleaning decision is a fork, and nobody counts the forks
The window is not the only thing that was chosen. So was the staleness threshold, the clipping sigma, whether the scale was ordinary or robust, whether repeated prints were dropped. Each decision was defensible and made once. Together they define a grid, and the published number is one cell of it:
from mdnorm import Choice, specifications, explore, choice_effect
choices = [
Choice("clip", [("none", None), ("5 sigma", D(5)), ("3 sigma", D(3))]),
Choice("scale", [("ordinary", False), ("robust", True)]),
Choice("stale", [("keep", False), ("drop", True)]),
]
curve = explore(specifications(choices), run_pipeline)
curve.highest # 0.8682 — clip nothing, ordinary scale, keep repeats
curve.lowest # 0.4254 — clip at 3 sigma, robust scale, drop repeats
curve.trials # 12
choice_effect(curve, "clip").spread # 0.3096 of the 0.4428 total
The grid multiplies, which is the point. Three decisions here; six binary ones would be sixty-four pipelines. Nothing samples the grid for you and there is no default set of decisions, because which forks a pipeline contains is a property of that pipeline and a library that guessed would be reporting on one it invented.
Which decision is doing the work is usually answerable, and that is the
useful output. choice_effect gives the median result under each option of
one decision; dominant_choice names the one that pulls them furthest apart.
Above, the clipping threshold accounts for 0.3096 of the 0.4428 spread — so
the sentence to write is not the number is unstable but the number is
mostly a function of a threshold we chose in a meeting.
A specification count is a trial count, with a caveat we will not bury.
SpecCurve.trials goes to deflated_sharpe_ratio the way
SensitivityReport.trials does, and it is an upper bound on the effective
number of trials: two pipelines differing in one choice out of six see nearly
the same data. Deflating by the raw count errs toward caution, which is the
direction to be wrong in, and it is still wrong. Nothing here estimates the
effective count, because that needs a model of how the choices correlate and
this library does not have one.
$ mdnorm multiverse pnl.csv --clip 5 3 --scale --stale --periods 252 --deflate
observations 1000
specifications 12
clip none | 5 sigma | 3 sigma
scale ordinary | robust
stale keep | drop
lowest 0.4254
median 0.7851
highest 0.8682
spread 0.4428
positive 12/12 (100.0%)
highest from clip=none, scale=ordinary, stale=keep
lowest from clip=3 sigma, scale=robust, stale=drop
attribution
clip none=0.8266 5 sigma=0.7929 3 sigma=0.5170 spread 0.3096
scale ordinary=0.7851 robust=0.7485 spread 0.0366
stale keep=0.8345 drop=0.7485 spread 0.0860
dominant decision clip (0.3096 of 0.4428)
trials 12
note: these specifications share a grid and are not independent, so this is
an upper bound on the effective number of trials. Deflating by it errs toward
caution rather than toward being right.
best specification deflated
as one trial 0.9579
as 12 trials 0.8874
best is named for the number and not for the pipeline. Which cleaning is
correct is not a question this library can answer — the cell that flatters a
result is simply the one most in need of a reason, and the parameters that
produced it belong in a mdnorm.provenance manifest.
Choosing where the sample starts is a trial
A backtest is reported as one number over one window. The window was chosen too — by when the data happened to begin, by which vendor file was to hand, or by somebody sliding the start forward until the curve looked right. The last of those is a search, and nobody counts it:
from mdnorm import trimmed_starts, sweep, sharpe_ratio
rep = sweep(returns, trimmed_starts(len(returns), step=21, count=24),
sharpe_ratio, kind=WindowKind.TRIMMED_START)
rep.full # -0.0329 — a losing strategy over everything
rep.highest # +0.0524 — profitable, starting eleven months in
rep.changes_sign # True
rep.trials # 25 — the number to hand a deflated Sharpe
A window count is a trial count. Twenty-four start dates are
twenty-four alternatives that could have been reported. Quoting the best of
them without deflating for that count is the same error as quoting the best of
twenty-four strategies, and it is harder to see because only one strategy was
ever written down. SensitivityReport.trials exists so the number reaches
deflated_sharpe_ratio instead of somebody's memory of how the window was
picked.
The spread is the finding, not the best value in it. A metric that is positive on seven windows out of twenty-four and negative on the other seventeen has not been measured badly — it has been measured, and the answer is that the headline figure is mostly a function of where the window opens.
Shorter windows are noisier, and that is part of what you see. A metric over a third of the data carries roughly √3 times the standard error, so some of the spread is sampling noise rather than instability. Nothing here separates the two; that would need a model of the return process and this library does not have one. Every window carries its observation count so the shrinkage is visible, and a wide spread is consistent with instability rather than proof of it.
$ mdnorm windows pnl.csv --metric sharpe --trim-start 21 --count 24 --deflate
observations 1000
windows 24 (trimmed_start)
shortest 496
longest 979
full sample -0.0329
lowest -0.0351
median -0.0168
highest 0.0524
spread 0.0875
positive 7/24 (29.2%)
note: the metric is positive on some windows and negative on others. That is
not a matter of degree.
trials 25
best window deflated
as one trial 0.9510
as 25 trials 0.4730
Read the last two lines together. The best window deflates to 0.95 if you pretend it was the only thing you ever looked at, and to 0.47 once the twenty-five windows are counted — and the strategy loses money over the full sample either way.
The sensitivity is usually to a handful of observations. If dropping the
first four months changes the answer, find out what was in those four months
before concluding anything about regimes. mdnorm.extremes counts how few
observations a result rests on, and a moved start date is often just a dropped
outlier wearing a different hat.
A fat tail and a fat finger look identical
One is the risk you are paid to carry, the other is a typo, and nothing in the number distinguishes them. So this measures what removing them would cost and removes nothing:
from mdnorm import flag_extremes, clip_effect, concentration
flag_extremes(returns, sigma=5, robust=True) # 30 found
flag_extremes(returns, sigma=5) # 0 found
concentration(returns, share=Decimal("0.5")) # 3 observations
The outliers hide inside the ruler used to find them. A z-score divides by a standard deviation computed from the same sample, and every extreme observation inflates it. Thirty contaminated points in a thousand raised the scale by 1.42x, which pushed all thirty from six sigma down to four and a bit — so at a five-sigma cut the ordinary score found none of them and the robust score found all thirty. That is masking, and it starts as soon as contamination is more than about one per cent.
So the scale is computed two ways and you choose. robust=True is the
median and the median absolute deviation scaled by 1.4826, which the extremes
cannot move. Right for detection, wrong for description — a robust scale
deliberately ignores the tail you may be trying to measure. Both are reported
and neither is assumed.
Clipping always lowers the measured volatility; what it does to the Sharpe
depends on which side the tail was on. Symmetric extremes leave the mean
alone and the ratio rises, which is the case people have in mind. A one-sided
tail takes the profit with it and the ratio falls — 0.88x on the run below.
Both are distortions of the same size, so clip_effect reports the shift
without asserting a sign and hands back no data.
Concentration is the question behind all of it. Three observations out of a thousand make half the total here. That strategy is a bet on three days, and whether those three were real is the only question that matters.
$ mdnorm extremes pnl.csv --sigma 5 --robust --tail 10 \
--concentration 0.5 --clip 3
observations 1000
ordinary centre +0.00034184 scale 0.01422197
robust centre +0.00024111 scale 0.01004799
note: the ordinary scale is 1.42x the robust one, which means the extremes
are inflating the ruler they would be measured against.
at 5 sigma
ordinary score 0
robust score 30
largest 10 by size
their total 0.12
whole total 0.3418381129
without them 0.2218381129
share of the total 35.10%
concentration 3 observation(s) make 50% of the total
clipping at 3 sigma
would touch 32
volatility 0.01422197 -> 0.01100780
understated to 0.7740x
Sharpe 0.8838x (lower)
note: the clip lowered the Sharpe rather than raising it, which means the
tail was one-sided and the profit went with it.
Nothing was clipped. winsorise exists and is separate, so trimming a sample
is a visible act rather than a side effect of measuring it — and the sigma you
picked is a parameter chosen after seeing the data, which is what
mdnorm.provenance is for.
A result you cannot reproduce is not a result
Every other module here refuses to guess a constant. metrics will not invent
the number of trials a search ran, coverage will not pick a gap threshold,
halts will not infer a pause, independence will not choose a truncation
lag. Each refusal hands the caller a decision — and until now nothing wrote
down what they decided:
from mdnorm import manifest, verify, read_manifest, write_manifest
m = manifest(command="sharpe", inputs=["pnl.csv"],
parameters={"trials": 500, "risk_free": "0.04"})
write_manifest(m, "run.json") # fingerprint ed731c772bdd
v = verify(read_manifest("run.json"), parameters={"trials": 50})
v.reproducible # False
v.drifts # pnl.csv: digest, trials: 500 -> 50
The parameters are the part that goes missing. An input that changes is usually noticed, because somebody had to change it. A trial count that was 500 in the run and 50 in the write-up is noticed by nobody, and it is the whole difference between a deflated Sharpe that survives and one that does not. A manifest records the arguments beside the data because they are the same kind of fact.
A fingerprint that includes the clock answers no question. Two runs of the
same pipeline over the same inputs with the same arguments must fingerprint
identically, so created_ns and the free-text note are excluded and only what
would change the numbers is covered. Outputs are excluded too, deliberately:
the same fingerprint with different results is the finding.
Floats are refused. 0.1 is not a value, it is a rendering of one, and the
rendering differs by platform. Pass a str or a Decimal and the manifest
records what you meant; pass a float and it raises rather than promising to
reproduce something it can only approximate.
An edited manifest is refused. read_manifest re-derives the fingerprint
from the contents and raises if the file disagrees with itself. A manifest
somebody has corrected by hand is worse than no manifest, because it carries
the authority of a record while stating something that never happened.
Nothing here judges. verify reports what moved and stops. Whether a
changed input is a correction or a corruption is not a question a library can
answer, and a tool that decided would be trusted for a judgment it is not
entitled to make.
$ mdnorm provenance run.json --verify --parameter trials=50
command sharpe
recorded with market-data-normalizer 1.35.0
fingerprint ed731c772bdd
inputs 1
result 2 difference(s)
pnl.csv: digest 'b264dd52615d' -> 'db1608907c45'
trials: parameter '500' -> '50'
note: nothing here says which side is right. A changed input may be a
correction or a corruption, and that is not a question this library can answer.
Exit status is non-zero when a run does not reproduce, so this drops into a scheduled check without any parsing.
The absence of a row is not the absence of an event
A feed that stops delivering and a market that stops trading produce the same thing: nothing. Every calculation downstream reads the silence as information:
from mdnorm import coverage_report, panel_coverage
rep = coverage_report(events, min_gap_ns=5 * MINUTE, calendar=cal,
halts=halts, start_ns=first, end_ns=last)
rep.explained_share # 97.27% of the silence was the venue being shut
rep.unexplained_ns # 5h54m nobody has accounted for
rep.longest_unexplained_ns # 3h10m — a feed that stopped mid-session
A gap has to be explained before it is a gap. Most silences are ordinary,
and a raw gap count is mostly weekends. explain_gaps subtracts closed
sessions using a TradingCalendar and pauses using the halts windows, then
reports the residual — which is the only part that says anything about the
feed. The three components are times rather than one label, because a Friday
outage running into a weekend is part missing data and part closed venue.
Give the bounds, or a feed that stopped is invisible. Without start_ns
and end_ns a gap only exists between two observations, so a symbol that goes
quiet and never returns produces nothing at all — its last print has nothing
after it to be distant from. That is the case worth catching, because
instruments stop printing when something has happened to them. With the
bounds, a named symbol that never appears is one gap the width of the period.
Names go missing when they are in trouble. A cross-sectional rank or
z-score over "the instruments that printed" is computed on a universe whose
width moves, and the ones that drop out are not a random sample.
panel_coverage counts the width at every point instead of averaging it away.
No default threshold, and the calendar is recorded. Five minutes without a
print is remarkable on a liquid future and unremarkable on a corporate bond,
so min_gap_ns is required — the same objection halts makes to inferring a
pause. A report built without a calendar carries calendar=False, because
counting every night as missing data is right for a venue that never closes
and badly wrong for one that does.
$ mdnorm coverage feed.csv --min-gap 5m --calendar us_2026.csv \
--session 09:30-16:00 --tz America/New_York --panel 1d \
--since 1773149400000000000 --until 1773432000000000000
events 5481
symbols 3
covered span 307h30m
gaps over 5m 15
total silence 216h24m
venue was shut 210h
halted 30m
unexplained 5h54m
longest unexplained 3h10m
explained 97.27% of the silence
A price you could not have traded at
When an instrument is halted the tape goes quiet, and a backtest reading that tape sees nothing unusual. The last print stands, the features keep updating off it, and every fill placed in that silence is a fill that could not have happened:
from mdnorm import halt_report, reopen_gaps, unfillable
halt_report(events, halts).halted_share # 25.6% of the covered span
reopen_gaps(events, halts)[0].move_bps # -1,834.8 across one pause
unfillable(decisions, halts).value_share # 91.6% of the money
Halts are concentrated in exactly the wrong place. An instrument is not paused on a quiet afternoon. It is paused on the day of the earnings leak, the guidance cut, the tender offer — the days carrying the largest moves in the sample. The fills a backtest invents during a halt are not a random slice of its trades; they come from the fattest part of the tail, and they land on the right side of it, because the strategy is reading a price that has not yet absorbed the news.
The count understates it and the value does not. One decision in two on
this input fell inside the pause, and those decisions carried 91.6 per cent of
the money. A value_share above a count_share is the signature of the whole
problem, which is why unfillable reports both and sums in absolute terms, so
a short and a long of the same size cannot cancel into a reassuring zero.
The reopening move belongs to nobody. A name halted at 41.15 and reopened
at 33.60 fell eighteen per cent with no tradable print in between. Whoever
held it took the loss; whoever "entered" during the pause entered at the stale
price and was marked at the new one, which is not a trade. reopen_gaps
prices every one of those.
Nothing is inferred. There is no rule here that a long enough quiet stretch is a halt — on an illiquid name that rule fires constantly, and the resulting statistic is a property of the threshold rather than of the market. Either the windows are supplied, or the report says it has none. Symbols with no halt record at all are named rather than assumed clean.
$ mdnorm halts trades.csv --halts halts.csv --decisions fills.csv
events 30
halts 1 across 1 symbol(s)
halted time 10m
longest halt 10m
share of covered 25.64%
reopening moves (no tradable price existed across these)
AAA 10m 41.15 -> 33.6 -1834.8 bps
decisions 4
unfillable 2 (50.00%)
by value 91.60%
note: the value share exceeds the count share, which means the decisions taken
while halted were the large ones.
Prints stamped inside a halt window are counted and not interpreted: usually a late report of a pre-halt execution, occasionally a cross that is allowed to print, sometimes a vendor with a broken clock.
A price that stopped moving is not a price that stopped being risky
align has warned since it was written that a frozen price is uncorrelated
with everything and therefore reads as diversification. It gave no way to find
out how much of that you have:
from mdnorm import staleness_report, smoothing_bias
rep = staleness_report(marks, min_run=3)
rep.unchanged_share, rep.longest_run # 75%, 31 observations
bias = smoothing_bias(returns)
bias.weight_current # 0.721 — the rest arrives tomorrow
bias.volatility_understated # 0.773x
bias.sharpe_inflation # 1.294x
Repeated values are a fact; staleness is an interpretation. An illiquid
instrument genuinely does not trade for an hour, and a vendor carrying
yesterday's mark forward produces identical rows. Nothing here decides which
you have — runs and staleness_report count, and what makes a flat stretch
suspicious is the instrument and the sampling interval, which is why there is
no default min_run.
Smoothing is where the money is. A reported series that partly reflects the previous period's move is a moving average of the true one, and a moving average has lower variance than what it averages. Lower measured volatility with the same mean is a higher Sharpe, a lower beta and a smaller correlation with everything else — four numbers moving the flattering way from one cause, with nothing raising an objection.
The adjustment is a model and says so. smoothing_bias assumes the
reported return is a two-period weighted average, infers the weights from the
first-order autocorrelation and reports the implied understatement. The result
carries modelled=True, so it can never be confused with the run counts,
which are arithmetic. A negative autocorrelation is bid-ask bounce rather than
staleness and returns weights of one and zero rather than pretending.
Where the model cannot fit, it refuses. A two-period average cannot
produce a first-order autocorrelation above one half, so a larger value is
evidence of something else — a trend, a longer memory, an overlapping sampling
window. fits comes back False and no figure is offered.
$ mdnorm staleness returns.csv --returns
observations 2999
unchanged 0 (0.00%)
autocorrelation +0.3368
weight on today 0.721
variance reported 0.5975 of the truth (modelled)
volatility 0.7730x understated
Sharpe 1.2937x overstated
Nothing is unsmoothed in place and no repeated value is dropped. Both would be corrections applied to data whose cause has not been established.
How many observations you actually have
A five-day forward return sampled every day gives you a thousand rows and about two hundred pieces of information. Every t-statistic, Sharpe, confidence interval and p-value computed on the thousand is overstated by roughly the square root of five, and nothing in the arithmetic complains:
from mdnorm import label_spans, effective_sample_size, deflate_t_stat
sample = effective_sample_size(label_spans(1_000, horizon=5))
sample.nominal, sample.effective # 1000, 200.8
sample.inflation # 2.232x on every t-statistic
deflate_t_stat(Decimal("2.1"), sample) # 0.941
forward_returns produces exactly those overlapping labels and
purged_splits already removes the training rows whose windows reach into a
test block. This is the other half of the same problem: purging stops the
overlap leaking across a split, and nothing stops it inflating the sample
within one.
Overlap is arithmetic, not an estimate. Given each label's window you can
count how many are live at every point. A label sharing its window with four
others is worth a fifth of an observation; summing that gives the effective
count exactly, with no model and no assumption. uniqueness exposes the
per-label figure and concurrency the step function underneath it.
Autocorrelation is an estimate, and it says so. For a return series with
no explicit windows there is only the sample autocorrelation, which is itself
noisy, so effective_sample_size_series marks its answer estimated. The
sum stops at the first non-positive autocorrelation — past that the terms are
noise whose signs cancel arbitrarily and can produce an effective sample
larger than the real one, which is the one direction this module exists to
rule out. On an AR(1) it lands on the closed form (1-φ)/(1+φ), which the
test suite checks.
No default lag, no default horizon, no silent correction. How far the
dependence reaches is a property of your data. deflate_t_stat returns the
adjusted figure and the report keeps both counts, because a statistic that
has quietly been divided by something is harder to argue with than one that
shows its working.
$ mdnorm independence --count 1000 --horizon 5 --t-stat 2.1
nominal sample 1000
effective sample 200.80 (exact)
ratio 20.1%
t-statistic inflated 2.232x
t-statistic adjusted 0.941
note: that crosses the conventional two-sigma line in the wrong direction.
The overlap did it, not the strategy.
The crosses are not points on the tape
The opening and closing auctions are single prints, at a single price, aggregating orders that never met each other in a book. Treating them as ordinary trades is wrong in a direction that flatters:
from mdnorm import auction_windows, auction_report, vwap_gap
windows = auction_windows(days, calendar) # from the calendar, not a constant
auction_report(trades, windows).volume_share # what went through the crosses
vwap_gap(trades, windows).difference_bps # what that does to the benchmark
An auction print has no aggressor. Nobody crossed a spread; a clearing price was computed. The tick rule and the quote rule will still return a side, because those functions always return a side, and it is an artefact of where the last continuous print happened to sit.
Execution benchmarks are where it costs money. A VWAP with the closing
cross in it is dominated by one print. A strategy that never touches the
auction, scored against that benchmark, is being measured against a price it
could not have obtained; a strategy that only trades the auction beats it by
construction. vwap_gap reports both numbers and the distance between them.
Auctions are not inferred. No condition-code guessing, no rule that a
print ten times the median must be a cross — on a busy day that rule
reclassifies ordinary blocks and the resulting statistic describes the
threshold rather than the market. Windows come from a TradingCalendar, so a
half-day's cross lands where the venue actually closed rather than three hours
later. The window extents default to zero: wide enough for a print stamped at
the bell and nothing else, because "thirty seconds, everyone uses that" is a
constant that differs by venue and by decade.
Nothing is deleted. split_auctions hands back both halves, and
largest_print_share is measured even with no windows at all — a file where
one print is a tenth of the day has a cross in it whether or not anything has
been told where.
$ mdnorm auctions trades.csv --calendar us_2026.csv --session 09:30-16:00 \
--tz America/New_York --ts-unit ns
trades 3730
in an auction 20
volume in auctions 67.86%
notional in auctions 67.89%
VWAP with auctions 100.103008
VWAP without 100.008100
benchmark difference +9.49 bps
Stored in nanoseconds is not measured in nanoseconds
Every timestamp here is an integer nanosecond. That is a storage decision. A vendor that stamps to the millisecond and hands you nanoseconds has multiplied by a million, and the extra six digits are zeros wearing the clothes of precision:
from mdnorm import detect_resolution, classification_risk
res = detect_resolution(ts)
res.granularity_ns, res.overstated_digits # 1_000_000, 6
res.tied_share # 76% of rows share a timestamp
risk = classification_risk(events)
risk.same_tick, risk.changed # exposure, and what it moves
Resolution is detectable, and detection is divisibility. A millisecond feed leaves every timestamp a multiple of a million. The module walks the decimal ladder — nanosecond up to a second — and reports the coarsest unit that divides everything. Only decimal units, because those are the units a clock is read in; a divisor of 2,000,000 would be arithmetic rather than a statement about the venue.
Divisibility needs enough observations to mean anything. Twenty distinct timestamps all being multiples of ten is a one-in-10²⁰ coincidence on a real nanosecond feed, so twenty is plenty — three is not. Below the threshold the answer is undetermined, which is not the same answer as one nanosecond. The threshold counts distinct values, since a thousand copies of one round number is one reading of the clock.
A tie is not an ordering. Rows sharing a timestamp are in the order the writer used — an unstable sort, a queue that interleaved, a buffer flushed however it was held. Reading sequence out of them is reading the writer.
Where it costs something is trade classification. The quote rule matches a
trade against the newest quote at or before it. When that quote carries the
same timestamp as the trade, which came first is not in the data.
classification_risk counts those trades, then re-runs the rule against the
last quote that is provably earlier and reports how many sides actually move.
$ mdnorm resolution trades.jsonl
distinct timestamps 3000
resolution 1ms
padding digits 6
tied timestamps 3704 (76.34%)
largest tie 2
trades 1852
same-tick quote 1852 (100.00%)
side would change 815 (44.01%)
Nothing here re-sorts, jitters, or invents a finer timestamp. The resolution you have is the resolution you have.
The shape of the day, fitted without the rest of the year
Volume, spread and volatility all follow a curve inside a session — heavy at the open, thin at midday, heavy into the close. Any statistic computed across a day without removing that curve is mostly measuring the time of day:
from mdnorm import Sample, US_EQUITY_RTH, session_profile, deseasonalise
shape = session_profile(volumes, US_EQUITY_RTH, bucket_ns=5 * 60 * 10**9)
shape.factor_at(0) # 2.09x — the first five minutes
shape.sessions, shape.excluded # what the curve is actually built on
adjusted = deseasonalise(volumes, US_EQUITY_RTH,
bucket_ns=5 * 60 * 10**9, min_sessions=20)
Removing the shape is the easy half; not reading the future while you do it
is the hard half. The usual recipe fits one profile over the whole sample
and divides every day by it, including the first. That profile contains days
that had not happened yet, so an unusually heavy open in January is judged
against a curve that already knows December. expanding_profiles gives each
session a profile built only from the sessions before it, and deseasonalise
uses it. full_sample_deseasonalise is the other one, kept deliberately: it
is the better estimate for describing a market and the wrong input to
anything that trades, and shipping both is what lets profile_leak measure
the difference instead of arguing about it.
No default bucket size. Five minutes over a 6½-hour session is 78 buckets; the same five minutes on a venue that never closes is 288. Finer buckets describe the curve better and put less evidence in each, and where that trade sits is a property of your data.
A thin bucket reports nothing rather than the average. Filling it with
the overall mean makes the adjusted series look well-behaved in exactly the
places where nothing is known about it, so min_observations sets the bar and
the bucket comes back empty below it. A point with no factor leaves the
output; a point silently divided by one would be a point claiming to have
been adjusted.
A short session is not a quiet one. Bucketing by offset from the open
puts a half-day's closing surge into a bucket that is mid-afternoon on every
other day. Given a TradingCalendar, early closes are left out and counted.
$ mdnorm seasonality volume.csv --session 09:30-16:00 --tz America/New_York --bucket 5m
sessions used 60
buckets 78 of 5m
heaviest bucket +6h25m into the session (2.09x)
lightest bucket +3h55m into the session (0.47x)
comparable samples 3120 of 4680
factor differs by >0.01: 2533 (81.19%)
median gap 3.59%
largest gap 30.46%
Features that cannot see the future
With the matrix built, the next step is turning prices into returns, z-scores, volatility and correlations. This is the second place look-ahead gets in, and it gets in just as quietly:
from mdnorm import ReturnMethod, column, returns, rolling_zscore, realized_volatility
px = column(rows, "BTC")
r = returns(px, method=ReturnMethod.LOG)
z = rolling_zscore(px, window=60) # trailing, never full-sample
vol = realized_volatility(r, window=60) # per period until you annualise it
qty = rolling_sum(volumes, window=60) # O(n), not O(n x window)
A full-sample z-score is look-ahead. Subtracting the mean and dividing by
the standard deviation of the whole series gives every observation knowledge
of the distribution it sits in — including the part that had not happened yet.
It is one line of code and it is everywhere. Every statistic here is trailing:
the value at i comes from values[i-window+1 : i+1] and nothing else. There
is a test that pins this as a property — change the tail of the input and every
earlier output must be byte-identical — and a second test that shows the
full-sample form failing it.
A partial window is not a result. Until the window fills you get None,
not a twenty-period statistic computed from three observations. A gap inside a
window propagates for the same reason: stepping over the hole would compute a
twenty-period number from nineteen and label it twenty.
A frozen series has no z-score and no correlation. Zero dispersion makes
both undefined, so both return None rather than 0. Reading that zero as a
correlation is how a dead feed becomes an apparent diversifier.
There is no default annualisation factor. √252 is right for daily bars on a
252-day calendar and wrong for almost everything else. realized_volatility
returns per-period volatility unless you pass a factor, and periods_per_year
makes you state the calendar rather than assume one — the same minute bars are
525,600 periods a year on a continuous venue and 98,280 on a cash equity
session.
The trailing sum is slid, and only where sliding is exact. rolling_sum
and rolling_mean no longer resum the window at every index, which took them
from O(n × window) to O(n) — window 250 now costs the same as window 60, 21×
faster than before. The reason libraries avoid this is drift: a running
Decimal total rounds differently from a fresh sum. That rounding is
observable, so every update runs with the Inexact flag cleared and is thrown
away the moment it would round, falling back to a full sum of the window. On
ordinary data every output is unchanged; where one does change it is because
the forward recomputation rounded an intermediate partial and the slid total
did not, and the test suite checks against rational arithmetic that the slid
answer is the exact one. The variance pass inside rolling_std and
rolling_zscore is deliberately not slid — the identity that would allow it
is a different sequence of roundings, and that is a trade we decline.
BENCHMARKS.md has the numbers and the argument.
$ mdnorm features matrix.csv --returns log --zscore 60 --vol 60 \
--interval 1m --sessions-per-year 365 --session-length 24h -o feats.csv
Labels, and a split that does not leak
A label is the one series in a research dataset that is allowed to look forward — it is the thing you are predicting. That makes it the series which quietly contaminates every split it touches:
from mdnorm import forward_returns, purged_splits
y = forward_returns(prices, horizon=5)
for split in purged_splits(len(prices), n_splits=5, horizon=5, embargo=60):
train, test = split.train, split.test
A label with a horizon makes neighbouring rows overlap. If the label at
row i spans the next five bars, rows i through i+5 all describe the same
stretch of future. Put row i in train and row i+3 in test and the model has
already seen most of the answer. Shuffling does not help — the rows genuinely
are different rows, they merely share an outcome. purged_splits drops the
training samples whose label window reaches into each test block, and reports
how many it dropped.
A gap after the test block is not enough, because features have memory. A
rolling statistic computed just after a test period is built partly from
observations inside it. The embargo removes the training rows immediately
following each block; set it to at least your longest feature window. It
defaults to 0 because the right value is a property of your features, not of
this function.
forward_returns looks forward on purpose. It is the only function in the
library that does, which is why it lives in mdnorm.labels rather than
mdnorm.features. Its output belongs on the left-hand side of a model; feeding
it back in as an input is not a subtle mistake.
The purging and embargo scheme follows López de Prado, Advances in Financial Machine Learning (2018), ch. 7.
$ mdnorm labels feats.csv --column BTC --horizon 5 --splits 5 --embargo 60 -o ml.csv
The instruments that existed then
Two of the biases this library guards against are about time. The third is about membership: a universe assembled today did not exist in the past.
from mdnorm import Universe, Listing, cross_section, cross_sectional_rank
pit = Universe([Listing("AAA", listed_ns=...), Listing("BBB", listed_ns=..., delisted_ns=...)])
ranks = cross_section(rows, cross_sectional_rank, universe=pit)
Survivorship bias produces no strange values anywhere. Take the names listed and liquid now, pull their history, rank them against each other over ten years, and every instrument in the study is one that survived. Unlike a look-ahead bug there is nothing odd to spot — the numbers are all real, the sample is just wrong.
Excluding a name is not the same as it having no data. A symbol that has
not listed yet, or delisted last month, belongs outside the cross-section
rather than inside it as a blank — because a blank gets treated as missing at
random, and the instruments that disappear from a market are the opposite of
random. mask_to_universe returns the number of cells it removed; over a long
window a count of zero usually means the listings file is present-day
membership.
The size of the cross-section changes, and that is correct. Percentile ranks are computed against the members present at that moment, so the denominator moves as instruments list and delist.
Ties share an average rank, missing names are ranked neither last nor middle, and a flat cross-section has no z-score rather than a row of zeros.
$ mdnorm universe matrix.csv --listings listings.csv --pct-rank -o pit.csv
$ mdnorm revisions gdp.csv -o published.csv
Two feeds that disagree
quality inspects one feed and reports what looks wrong inside it. A second
feed asks the question desks actually use to decide whether a vendor can be
trusted.
from mdnorm import reconcile, suggest_shift
report, mismatches = reconcile(primary, secondary,
relative_tolerance=D("0.0001"))
report.agreement # of the shared timestamps, how many matched
report.coverage_difference # timestamps only one of them had
The two kinds of disagreement are not one number. A timestamp one feed has
and the other does not is a coverage difference — a dropped message, a
filtered print, a venue one side does not carry. A timestamp both have with
different values is a content difference, and at least one of them is wrong
about something checkable. agreement is computed over shared timestamps
only, so a feed that simply carries less does not look like a feed that lies.
There is no default tolerance. Two feeds of the same instrument differ in the last digits for reasons that are not errors, and a constant deciding how much is acceptable is a judgement about your data rather than a property of it. With none given, values must match exactly — the strictest reading, and one that states its own assumption.
Zero overlap almost never means total disagreement. It usually means a
clock offset: one feed stamps at the venue, the other on receipt, exact
matching finds nothing in common, and the naive conclusion is that the feeds
are unrelated. suggest_shift looks for the constant offset that lines them
up and reports how much of the sample it would explain. It does not apply it —
a clock difference is a fact about two systems that somebody should confirm.
$ mdnorm reconcile primary.csv vendor.csv --relative 0.0001 -o breaks.csv
Who was in the index, and when they were told
universe applies a membership record. Producing one from the files a vendor
actually ships is a separate job, and it is where survivorship gets in.
from mdnorm import MembershipHistory, Basis, survivorship_gap
history = MembershipHistory.from_changes(changes)
history.members_at(t, basis=Basis.EFFECTIVE) # who was in the index
history.report() # what the file cannot say
survivorship_gap(history, t) # what a today-list would cost
Two dates, and they answer different questions. An addition is announced
on one day and takes effect on another. Who was in the index is the
effective date; when could anyone have known is the announcement. Rank on
one and trade the other and the file will never object, because both columns
are correct. Basis has no default, so the question has to be named.
A snapshot cannot express a deletion. Names that leave do not appear as
departures, they simply stop being listed, and the last file that showed them
is not the day they left. from_snapshots therefore dates each inferred change
at the later snapshot — never claiming membership earlier than the file
supports — and records the window it really fell inside. On a monthly file
that window is a month, which is longer than many holding periods.
A today-list is the classic error, and it is measurable.
survivorship_gap returns both directions: the names a today-list drops
(they left, usually not for good reasons) and the names it holds too early
(they had not joined yet). The two do not cancel — one removes losers and the
other adds winners — which is why the effect is large and one-directional.
The report names the tell. If nothing ever left, the file is a list of
today's members wearing a history's clothes, and mdnorm membership says so
out loud rather than computing quietly on it.
$ mdnorm membership index_changes.csv --at 1770000000000000000
Values that get corrected later
Every observation so far has had one timestamp: when it happened. A lot of real data has two — the period it describes, and the moment it became knowable — and then it gets revised.
from mdnorm import Revision, RevisionSeries
series = RevisionSeries([
Revision(event_ts_ns=q1, known_ts_ns=april, value=D("2.1")),
Revision(event_ts_ns=q1, known_ts_ns=may, value=D("1.6")), # revised down
])
series.as_of(event_ts_ns=q1, known_ts_ns=april_20) # 2.1 — what you knew
series.final(event_ts_ns=q1) # 1.6 — what is true now
Using the corrected value is look-ahead, and no timestamp check will catch
it. The row is dated correctly. The value is a real number that was genuinely
published. Nothing marks it as unavailable until three weeks later. Every guard
in mdnorm.align passes and the study is still wrong.
Two honest questions, two different objects. What was the newest published
number at time t is a feature — known_series() is keyed by publication time
and joins like any other stream. What did the whole table look like at time t
is a vintage — vintage_at(t) is keyed by event time and reproduces the sheet
as it appeared that day. Reading a vintage at the wrong moment gives a value
nobody had; there is a test that shows exactly that.
Measure it rather than assuming. revision_summary() reports how many
events were ever revised and how far first releases sat from final values. If
that number is large, every backtest built on final data has been reading
answers.
$ mdnorm revisions gdp.csv -o published.csv
A daily number on an intraday grid
A daily close, a settlement price, an overnight risk figure — slow series meet fast grids constantly, and they almost always arrive labelled with the period they describe rather than the moment they became knowable.
from mdnorm import PeriodSeries, leak_report, US_EQUITY_RTH, grid
series = PeriodSeries.from_sessions(daily_closes, US_EQUITY_RTH)
feature = series.knowable_series() # keyed at the close — safe to join
report = leak_report(series, grid(...)) # what the label join would have cost
A daily bar labelled Tuesday is not knowable on Tuesday morning. It is knowable once Tuesday's session closes — Tuesday evening, and later still if the number has to be published. Join it by its label and every minute of Tuesday sees a value that summarises, among other things, the rest of Tuesday.
The session decides the close, not the file. Daily bars are frequently
stamped midnight to midnight regardless of when the market was open.
from_sessions and from_daily_bars take a Session, so a 16:00 New York
close is 21:00 UTC in January and 20:00 in July without the caller thinking
about it.
Publication lag is a separate claim. A settlement price exists at the
close; it reaches you when the vendor sends it. publication_lag_ns is where
that goes and it defaults to zero, because a lag of zero is a statement about
your feed that only you can make.
Measure the leak instead of arguing about it. leak_report counts the grid
points where the label join shows a value that did not yet exist, and how far
ahead the worst one was read. On back-to-back periods the answer is every
point: the moment one value becomes readable the label has already moved to
the next. Whether that ruins a study depends on the signal, which is exactly
why the number is worth having.
$ mdnorm mixfreq daily.csv --interval 60000000000 --lag 900000000000 -o joined.csv
How much of the result is the search
Everything above is about getting the data right. The last step is a correct dataset that still produces a misleading number, because the number was chosen. A Sharpe ratio from one strategy is an estimate; the same ratio kept after trying two hundred parameter sets is a maximum, and the maximum of two hundred draws from noise is not small.
from mdnorm import sharpe_report
rep = sharpe_report(daily_returns, periods_per_year=D(252),
trials=500, trial_sharpe_variance=D("0.004"))
rep.sharpe_annualised # 0.56 — the figure that goes in the deck
rep.probabilistic # 0.92 — probability the true ratio is above zero
rep.deflated # 0.008 — after accounting for 500 attempts
rep.demonstrated # False — the sample is shorter than it needs to be
rep.warnings # what the headline number does not say
Ratios are per period until you state the calendar. sharpe_ratio divides
mean by standard deviation and stops; annualise_sharpe needs a factor, for
the same reason realized_volatility does. Being wrong by a constant is the
hardest kind of wrong to notice, because the shape of the series is unchanged.
A short track record is not evidence. min_track_record_length says how
many periods a ratio needs before it is distinguishable from the benchmark.
A strategy whose minimum is nine years and whose backtest is eighteen months
has not been demonstrated, however good the ratio looks.
Selection is measurable. expected_max_sharpe(trials, variance) is the
best ratio a search of that size produces from strategies that are all
worthless. deflated_sharpe_ratio measures your result against that instead
of against zero, following Bailey and López de Prado (2014). Pass the whole
search, not the survivors.
Nothing here returns a flattering placeholder. A series that never moved
has no Sharpe, a sample with no losing period has no measurable downside, a
curve that never fell has no drawdown — all None, not zero and not infinity.
Each of them is a statement about the sample being short.
$ mdnorm metrics pnl.csv --column ret --interval 1d \
--sessions-per-year 252 --session-length 6h \
--trials 500 --trial-variance 0.004
What the trade costs
mdnorm.execution measures what your fills actually cost. This is the other
question: what a backtest should charge itself for a trade it never made. It
is the crudest way a result flatters you and it survives every other check,
because nothing in the data is wrong — the strategy is simply being priced at
a level nobody trades at.
from mdnorm import CostModel, Fees, ImpactModel, Liquidity, estimate, capacity
model = CostModel(fees=Fees(taker_bps=D(1)),
impact=ImpactModel(coefficient=D("0.5"))) # no default
liq = Liquidity(adv=D(1_000_000), volatility=D("0.02"), spread_bps=D(4))
b = estimate(model, notional=D(500_000), quantity=D(20_000), liquidity=liq)
b.commission_bps # 1.0
b.spread_bps # 2.0 — half of the quoted spread, because you crossed
b.impact_bps # 14.1 — 2% of daily volume, square-root law
b.total_bps # 17.1
capacity(D(20), model=model, liquidity=liq) # 28,900 a day at a 20 bps edge
Zero cost is not a default, it is a claim. A backtest that charges nothing has asserted that it trades at the midpoint, in unlimited size, for free. Written down that way nobody would sign it.
A cost that does not depend on size is not a cost model. A flat five basis
points says a strategy can trade a thousand dollars and a billion on identical
terms, so every capacity question has the same answer. estimate says so in
its warnings every time an impact model is absent.
There is no default impact coefficient. The square-root law is well
supported; the constant in front of it is not universal, and a plausible wrong
one rescales every cost in the report while changing nothing about its shape.
Calibrate it against your own fills — that is what evaluate is for.
The useful output is not the cost. breakeven_participation is the
fraction of daily volume at which the edge is exactly consumed, and capacity
is the same figure as a quantity. A two-basis-point edge that breaks even at
0.3% of volume is a different object from the same edge breaking even at 30%,
and no Sharpe ratio distinguishes them.
$ mdnorm costs pnl.csv --column ret --turnover-column turnover \
--cost-bps 5 --edge-bps 20 --adv 1000000 --volatility 0.02 \
--spread-bps 4 --fee-bps 1 --impact-coefficient 0.5
A ticker is not an identifier
canonical_symbol makes BTCUSDT and XBT/USD agree on a spelling. This is
the other problem: the same spelling, at two different times, meaning two
different things. Exchanges reuse ticker strings — a company delists and its
symbol is reassigned, a venue renames a pair and the old name reappears
elsewhere.
from mdnorm import SymbolAssignment, SymbolMap, key_by_instrument, series_segments
smap = SymbolMap([
SymbolAssignment("ABC", "US0000000001", start_ns=t0, end_ns=t1),
SymbolAssignment("ABC", "US0000000002", start_ns=t2), # reused later
])
smap.reused_symbols() # [("ABC", 2)] — the finding
smap.instrument_at("ABC", t_mid) # None: in the gap it named nothing
rows, counts = key_by_instrument(rows, smap)
counts["reassigned"] # rows the string would have mis-joined
segments, unresolved = series_segments("ABC", timestamps, smap)
The bias is a join, not a bad value. Every price in a spliced series genuinely traded, at its own timestamp, under the ticker it carries. What is wrong is the assumption that the column header names one thing — made once, silently, when the matrix is built.
Reuse looks like a merger, and mergers look profitable. A delisting is usually a fall and a new listing starts at a normal price, so splicing one onto the other inserts a jump. Half the time it is upward, and an upward jump in a name you were holding is indistinguishable from a takeover premium. The series does not look broken; it looks lucky.
A gap is not filled with the next owner. Between the delisting and the
reassignment the ticker named nothing, and instrument_at returns None there
rather than the instrument that took the letters afterwards. That substitution
is the splice.
Overlaps are refused. One ticker bound to two instruments at the same
moment is a broken reference file, and picking one of them quietly is how the
error reaches a study. SymbolMap raises instead.
No reuse in a long history is a finding, not a pass. A file with one open-ended binding per ticker cannot express reuse at all, so a zero means the file rather than the market — the same shape of diagnostic as a purge that removes nothing.
$ mdnorm instruments symbol_map.csv trades.csv --segments ABC -o keyed.csv
Data quality
from mdnorm.quality import find_issues, clean
find_issues(events) # list of QualityIssue (outlier / gap / out_of_order / non_positive)
cleaned, issues = clean(events) # drop bad ticks & invalid rows, keep a report
clean removes price outliers and non-positive price/size records and returns
the surviving events plus everything it flagged.
Serialization
from mdnorm import to_records
to_records(events) # list of flat dicts (Decimals as strings)
to_records(bars, as_float=True) # numeric output for DataFrames
to_records (and event_to_dict / bar_to_dict) flatten events and bars into
plain, JSON-serialisable dicts — drop straight into pandas.DataFrame, a
csv.DictWriter, or json.dumps.
Consolidating streams
from mdnorm import merge_streams, dedupe
timeline = dedupe(merge_streams(binance_events, coinbase_events))
merge_streams interleaves multiple venue feeds into one timestamp-ordered
timeline; dedupe drops exact duplicate events left behind by reconnects and
replays.
CSV files
from mdnorm import read_csv_trades, write_records_csv
events = read_csv_trades("trades.csv", venue="coinbase") # file -> events
write_records_csv(bars, "bars.csv", as_float=True) # events/bars -> file
read_csv_trades parses a whole CSV of trades into normalized events;
write_records_csv writes events or bars back out. Standard library only.
NDJSON / JSON Lines
from mdnorm import write_jsonl, read_jsonl_events
write_jsonl(events, "events.jsonl") # one JSON object per line
events2 = read_jsonl_events("events.jsonl") # lossless round-trip
# large files: stream lazily, .gz handled transparently
for e in iter_jsonl_events("dump.jsonl.gz"):
...
Pipelines
Declare a processing chain once, reuse it everywhere:
from decimal import Decimal
from mdnorm import Pipeline
pipe = (
Pipeline()
.dedupe()
.clean(max_return=Decimal("0.1"))
.time_bars(60_000_000_000) # 1-minute bars
.fill_gaps()
)
bars = pipe.run(events)
print(pipe.last_issues) # quality report from clean()
Command line
The common conversions ship as a zero-dependency CLI:
$ mdnorm bars trades.csv --venue binance --interval 1m -o bars.csv
$ mdnorm quality trades.csv --max-gap 5m
$ mdnorm convert trades.csv -o trades.jsonl
$ mdnorm bars trades.csv --interval 1d --actions actions.csv -o adjusted.csv
$ mdnorm align BTC=btc.csv ETH=eth.csv --interval 1m --max-age 5m -o matrix.csv
$ mdnorm features matrix.csv --returns log --zscore 60 --vol 60 -o feats.csv
$ mdnorm labels feats.csv --column BTC --horizon 5 --splits 5 -o ml.csv
$ mdnorm universe matrix.csv --listings listings.csv --pct-rank -o pit.csv
$ mdnorm revisions gdp.csv -o published.csv
$ mdnorm metrics pnl.csv --column ret --trials 500 --trial-variance 0.004
$ mdnorm costs pnl.csv --cost-bps 5 --edge-bps 20 --adv 1e6 --volatility 0.02
$ mdnorm instruments symbol_map.csv trades.csv -o keyed.csv
$ mdnorm calendar us_2026.csv --session 09:30-16:00 --tz America/New_York
$ mdnorm fx prices.csv rates.csv --from EUR --to USD --max-age 1m -o usd.csv
$ mdnorm ticks prices.csv --table ticks.csv
$ mdnorm arrival feed.csv --interval 1s
$ mdnorm seasonality volume.csv --session 09:30-16:00 --bucket 5m
$ mdnorm resolution trades.jsonl
$ mdnorm auctions trades.csv --calendar us_2026.csv --session 09:30-16:00
$ mdnorm independence --count 1000 --horizon 5 --t-stat 2.1
$ mdnorm staleness marks.csv --min-run 3
$ mdnorm halts trades.csv --halts halts.csv --decisions fills.csv
$ mdnorm coverage feed.csv --min-gap 5m --calendar us_2026.csv
$ mdnorm provenance run.json --verify --parameter trials=500
$ mdnorm extremes pnl.csv --sigma 5 --robust --tail 10
$ mdnorm windows pnl.csv --metric sharpe --trim-start 21 --deflate
Also available as python -m mdnorm.
The unified schema
@dataclass(frozen=True, slots=True)
class MarketEvent:
symbol: str # canonical "BASE-QUOTE", e.g. "BTC-USD"
venue: str # source venue
event_type: EventType # TRADE | QUOTE
ts_ns: int # nanoseconds since Unix epoch (UTC)
price: Decimal | None
size: Decimal | None
side: Side | None # BUY | SELL
# ... plus bid/ask fields for quotes
Benchmarks
Throughput for the hot paths, measured by a script in this repository rather
than asserted: BENCHMARKS.md. The headline finding is that
exact Decimal arithmetic costs 3.1× a float loop, not the order of magnitude
usually assumed — so the cost of this library is mostly Python and the
algorithm, not the exactness.
$ python bench/benchmark.py
Roadmap
What exists, what has been asked for, and what we have decided against is in ROADMAP.md — including the native Rust port two people have now asked for, with an honest account of what it would and would not solve.
Design notes
- Money is
Decimal. Prices and sizes never touch binary floats, so42000.10stays42000.10. - Time is integer nanoseconds, UTC. One comparable integer regardless of whether the source gave seconds, milliseconds, or a FIX timestamp string.
- Symbols are canonicalized to
BASE-QUOTE, with venue aliases resolved (XBT→BTC) and quote currencies detected longest-match-first soUSDTwins overUSD. - Normalizers are pure functions — one raw record in, one
MarketEventout — which keeps them trivial to unit-test and compose into any streaming or batch pipeline.
Architecture
raw feed ──► normalizer ─────────────► MarketEvent ──► your pipeline
(CSV / (from_csv_row / (unified, (research,
WS JSON / from_ws_json / immutable) backtest,
FIX) from_fix) execution)
│
├── symbols.canonical_symbol() BTCUSDT → BTC-USDT
├── SymbolMap.instrument_at() which instrument the ticker named then
├── timeutil.*_to_ns() any time → ns UTC
├── adjust.adjust_events() splits/divs/rolls
├── TradingCalendar.is_open() the holidays and half-days
├── FxRates.convert() a price in another currency
├── grid_report() is this a print or a derived number
├── micro.infer_sides() who crossed the spread
├── book.OrderBook() deltas → live book → quotes
├── consolidate() many venues → one best bid/offer
├── evaluate() your fills vs the market
├── align() N instruments → one time grid
├── returns() / rolling_*() features, trailing windows only
├── purged_splits() folds whose labels do not overlap
├── Universe.members_at() who was actually listed then
├── RevisionSeries.as_of() which version you had then
├── sharpe_report() and how much of it is the search
└── capacity() the size at which the edge runs out
Examples
Three runnable scripts in examples/, standard library only:
demo.py |
one trade in CSV, WebSocket JSON and FIX collapsing to the same event |
is_this_file_real.py |
prints against mids, VWAPs and an adjusted history, on the tick grid |
nothing_looks_forward.py |
edit the tail of a series; a trailing z-score does not move, a full-sample one moves everywhere |
Tests
pip install pytest
pytest -q
The suite includes a cross-venue equivalence test proving CSV, WebSocket and FIX representations of one trade collapse to an identical event, and a causality property applied across the feature layer: change the tail of an input, and every output before the change must be byte-identical.
CI also runs mypy, and it is clean. The package ships a PEP 561 py.typed
marker, so your type checker will use its annotations rather than ignore them.
That marker was missing for most of this project's life while the packaging
metadata claimed otherwise. Removing the false claim and then earning it back
took one release each; what the second one mostly consisted of was writing
down invariants the code already enforced — a trade cannot exist without a
price, a window past the gap guard holds no None. See
CONTRIBUTING.md for where cast is and is not acceptable.
License
MIT © HarvestGroup360 (AMII LTD). See LICENSE.
Maintained by HarvestGroup360 as part of our open quantitative-infrastructure tooling.
1 maintainer
Release files for market-data-normalizer 1.40.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| market_data_normalizer-1.40.0.tar.gz | 427.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| market_data_normalizer-1.40.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 694.4 kB
Release files / market_data_normalizer-1.40.0.tar.gz
| Download URL | market_data_normalizer-1.40.0.tar.gz |
|---|---|
| Size | 427.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
69944ad25f594319570c01ff0dba88df0d6af0c532b39f210c9c4d9321a6c61e
|
|
BLAKE2b-256 checksum How to use checksums |
9d8d80a5a8e587c4f87340b69daed217a19e9d900fa3632824141bd8e8d0b19d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 15, 2026.
Transparency logRelease files / market_data_normalizer-1.40.0-py3-none-any.whl
| Download URL | market_data_normalizer-1.40.0-py3-none-any.whl |
|---|---|
| Size | 266.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
09783c9f4d7feddc92c2c111cc02a26831ae660ee13b68f898c254a2f43a07fd
|
|
BLAKE2b-256 checksum How to use checksums |
a20aba1f7eb3f9742d06210cbf23816cf0cd8dc67478d86333741365e237383a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 15, 2026.
Transparency log
