itch-book

NASDAQ TotalView-ITCH 5.0 feed handler and limit order book reconstruction in C++23. Header-only, no dependencies beyond fixed-decimal for exact prices. Parses the raw BinaryFILE day dumps NASDAQ publishes at emi.nasdaq.com/ITCH and maintains per-symbol books with FIFO order queues, aggregate price levels and best bid/offer tracking.

On a full trading day (12302019.NASDAQ_ITCH50, 268.7M messages, 8.25 GB) it replays parse + full book apply for all 8,907 symbols at ~17.3M messages/s single-threaded (58 ns/message) inside ~1.4 GB of book structures, with zero unresolved order references and zero crossed books at the close.

Usage

#include <itch/book_manager.hpp>
#include <itch/mapped_file.hpp>
#include <itch/parser.hpp>

itch::MappedFile file("12302019.NASDAQ_ITCH50");
itch::BookManager<> books;
itch::ParseResult r = itch::parse(file.bytes(), books);

itch::Bbo q = books.bbo(locate);            // best bid/offer for a symbol
const auto* book = books.book(locate);      // full depth, FIFO queues per level

Handlers are plain structs; implement only the callbacks you need. Messages you skip cost one length lookup, nothing is decoded for them:

struct Trades {
    void on_trade(const itch::Trade& t) { /* ... */ }
};
Trades h;
itch::parse(file.bytes(), h);

Input does not have to be one whole buffer. StreamParser reassembles frames that arrive split across arbitrary chunk boundaries (socket reads, packet payloads); whole frames inside a chunk are still parsed in place, only a partial tail is ever copied:

itch::StreamParser stream(books);
while (read_chunk(buf)) stream.feed(buf);

The manager can also emit a time-and-sales stream: E executions print at the resting order's price, which only the book knows, plus printable C, non-cross trades, crosses and broken-trade voids, all in feed order:

itch::BookManager tape(nullptr, [](const itch::TradePrint& t) { /* ... */ });

Build and test:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build

Tools: itch-replay <file> [--book] (per-type counts, or full book replay with stats), gen-synthetic <out> <messages> [symbols] [seed] (deterministic test feed), parse_throughput / book_throughput / apply_latency benches (the last needs -DITCH_BENCH_LATENCY=ON, x86 only). GCC and Clang; MSVC is out because the price type needs __int128.

Python

The same core ships as a Python package: pip install itch-book, wheels for Linux x86_64 and aarch64 (manylinux_2_28), macOS 11+ arm64 and x86_64, Windows x64, Python 3.10+; it depends on numpy and tzdata. A source build needs a C++23 compiler. It reads raw or gzipped day files straight from emi.nasdaq.com and hands out columnar batches as numpy arrays, zero-copy, so Polars, pandas and pyarrow ingest them without conversion.

import itch_book as ib

feed = ib.open("20190730.BX_ITCH_50.gz")          # session date from the filename
for batch in feed.batches(tables=("bbo", "symbols"), rows=1_000_000):
    df = ib.to_polars(batch.bbo)                  # ts_event as Datetime("ns", "UTC")
feed.stats                                        # message counts and book invariants

Every table carries ts_event (int64 ns UTC: New York midnight of the session plus the ITCH timestamp), seq (ordinal among decoded messages) and locate. Prices are float64 by default (every ITCH Price(4) is exact in a double) or the raw int64 mantissa with price_type="fixed"; a missing price is NaN / 0. Single-character columns come out as S1; to_polars turns them into strings. rows is a lower bound per batch: batches are cut at chunk boundaries.

batches(..., symbols=("AAPL", "MSFT")) keeps every book (executes and replaces need the order they refer to, wherever it lives) and emits rows only for the selected locates; feed.stats stays feed-wide, and stats["selected"] says how many names matched the day's directory (a miss is a warning). depth rows carry no trigger columns; join messages on seq for the event that produced a snapshot. On the BX day 99.8% of book-changing messages touch the top ten levels, so depth at N=10 is effectively one row per event there.

trades: an E prints at the resting order's price, a C at the message price and only when printable, side is the resting side for E/C and N otherwise (the P side field is always B on the wire and carries nothing), order_id is the resting order for E/C and 0 otherwise, size is uint64 because cross sizes are 8 bytes. A B row carries only match_number; the trade it voids may sit in an earlier batch, so anti-join over the day.

messages is order-by-order with the resting state looked up before the message is applied: side, locate and (for E/X/D) price come from the resting order, so a D/X/E row is self-contained; remaining is what is left after apply, clamped at zero. action is A add, F fill (E/C), C cancel (X/D), M replace (U, where order_id is the new reference and old_order_id the old). Rows the book did not apply say so: side == N means the reference was unknown and nothing changed; an A or U with remaining == 0 was rejected (zero shares or price). An A or U onto a live reference evicts it first. Non-printable C executions are here with printable == False and absent from trades. mpid indexes feed.mpids (0 = none) and is only set on F rows. test_tables.py replays this table with those rules and reproduces bbo exactly on random days that include unknown references, over-sized executes and duplicate references.

symbols is the stock directory keyed by locate. Decompression runs on a reader thread in Python's zlib; the parser and books run in C++ with the GIL released.

BX 2019-07-30 (391 MB gzip, 28.7M messages, 8,849 symbols) on the machine above, gunzip included: bbo alone 2.3 s (19.1M rows), the five row tables without depth 2.5 s (messages 23.8M rows, trades 925k), every book invariant at zero. depth at N=10 for all 8,849 symbols is the one expensive table: 7.0 s for 23.8M rows of 63 columns; with three symbols selected the whole run is back to 2.3 s. One abi3 wheel covers 3.12 and later, 3.10 and 3.11 get their own; Windows builds with clang-cl (MSVC has no __int128).

itch2parquet

pip install 'itch-book[cli]' adds a command that writes the tables as Parquet (pyarrow, zstd, one row group per batch) and takes care of getting the data:

itch2parquet list                                   # what emi.nasdaq.com has, with sizes and session dates
itch2parquet fetch 20190730.BX_ITCH_50.gz --dir data # resumes a partial download, checks the published md5
itch2parquet verify data/20190730.BX_ITCH_50.gz     # replays the day and prints the book invariants
itch2parquet convert data/20190730.BX_ITCH_50.gz out # bbo + trades by default
itch2parquet convert FILE out --tables messages,depth --symbols AAPL,MSFT --depth 5 --price-type fixed

Every row table gets a dictionary-encoded symbol column next to locate, trades gets a broken flag (true on a print that a later B voided; the B rows stay; the BX day above has no B at all, so that path is exercised by the tests only), and symbols.parquet / system_events.parquet are always written. Files are written to .part and renamed at the end, so a failed run leaves nothing behind, and stale table files from an earlier run in the same directory are removed first. The footer of each file carries itch_book.* key-value metadata: a schema version, the source file name and its md5, session date, time zone, price_type and price_scale (dollars = stored value / scale), the table list and symbol filter, tool version, creation time, and the full stats dictionary (message counts, unresolved references, crossed_books and live_orders as of the end of the file, last system event), so a Parquet file states where it came from and whether the book that produced it was clean. The session date comes from the filename (both emi naming schemes) and is printed when inferred; --date overrides it. trades is held in memory for the whole day to compute broken (about 50 bytes per print: ~100 MB for BX, ~1 GB for a NASDAQ day); the other tables stream. Unsigned columns are stored with Parquet unsigned annotations, which polars, pyarrow, duckdb and pandas read directly and some older JVM readers do not. There is no per-symbol partitioning; filter the tables afterwards. The BX day above converts to bbo (267 MB) + trades (15 MB) in 6.6 s including the md5 pass.

Wire format notes

Every message sits behind a 2-byte big-endian length prefix; a zero length marks end of session. The parser treats the prefix as authoritative: known types are additionally checked against their fixed spec length (one table lookup), unknown or mismatched frames are skipped by length and counted, never parsed. Nasdaq adds message types over the years (O, Direct Listing with Capital Raise, arrived in 2023), and parsers that abort on unknown bytes die on the first file recorded after their spec revision.

All multi-byte fields are big-endian at odd offsets. Fields are decoded with memcpy into an integer plus std::byteswap; at -O2 GCC and Clang compile that to the same single mov + bswap a reinterpret_cast of a packed struct would produce, but without the unaligned-access UB, so the hot path runs clean under UBSan and works on strict-alignment targets. The 6-byte timestamps are copied as 6 bytes; the popular trick of one 8-byte load at offset 5 shifted right reads past the end of the buffer on the final 12-byte message of a file.

Dispatch is a switch on the type byte into a compile-time handler concept (if constexpr (requires { h.on_add(...); })), so decode inlines straight into book application. No virtual calls anywhere on the hot path.

Prices are ITCH Price(4), 32-bit unsigned with four implied decimals, and land in fixed_decimal::Fixed<4, PriceTag, int64_t>: exact integer mantissa arithmetic, no floats, from_raw costs nothing.

Book design

• Books live in a flat vector indexed by stock locate (the spec defines it as a dense, day-scoped array index), so no symbol hashing ever happens per message.
• Price levels per side are a sorted vector with the best level at the back. Ask prices are stored negated so both sides share one ascending comparator and the same scan-from-back loop. Adds and deletes overwhelmingly hit within a few levels of the touch, so the linear scan typically ends in 1–5 comparisons; a deep insert pays an O(levels) memmove, which the latency table below quantifies.
• Level records (aggregate shares, order count, FIFO head/tail) are pooled per book behind 32-bit handles with a LIFO freelist: no allocation per level after warm-up, and handles stay valid across vector growth.
• Orders carry their level handle, so executes, cancels, deletes and replaces never search the book: one order lookup, one level dereference. Messages that mutate an existing order are more than half of a NASDAQ day (deletes alone are ~43%), which is why this is the property worth paying for.
• FIFO queues are intrusive doubly-linked chains of order references per level, so queue position is reconstructible. That is the part aggregate-only books throw away.
• The order-reference index exploits that ITCH refs are day-unique and near-dense: a paged direct index (8,192 refs per page) instead of a hash. The default store keeps pages of 32-bit handles into a recycled order pool; a page is freed to a spare list the moment its last order dies, so resident memory is bounded by the live window of the ref space, not by the day's 118M adds. A cap on the accepted ref space (kMaxRef) keeps a corrupt or adversarial feed from growing the page table without bound.
• reserve() on the manager and the store pre-sizes everything for a strict zero-allocation steady state, verified by a test that counts global operator new calls across 400k messages after warm-up: zero.
• Trading-action state (H) is tracked per locate and queryable (trading_state(locate)), but books are deliberately not gated on it: Nasdaq keeps order maintenance flowing during halts, so a handler that stops applying messages on H resumes with a corrupt book.

Robustness rules: unknown refs are counted and ignored, duplicate adds replace the stale order, over-sized executes clamp, zero-share or zero-price messages are rejected. Each path is unit-tested and mirrored exactly by the reference implementation used for differential testing.

Correctness

• Differential test: a deliberately naive reference book (std::map levels, std::unordered_map orders, ~100 lines) consumes the same synthetic feeds as the fast book; full book states (every level, every side, live-order counts) are compared at checkpoints. Runs over 3 seeds × 400k messages × all three order-store variants.
• Structural invariants (Book::validate): sorted sides, level aggregates equal to the sum of their FIFO chain, link consistency, order counts.
• Real-data smoke: the full NASDAQ and BX days replay with zero missing refs, zero duplicates, zero rejects, zero clamps, and zero crossed books at the close.
• Fuzzing: a libFuzzer harness drives parse + book apply in CI (ASan+UBSan); a deterministic mutation test (bit flips + truncations over a synthetic feed) runs in the regular suite. The framing layer never reads outside the buffer by construction; decode only happens after the length check.
• CI: GCC, Clang, ASan+UBSan, fuzz, all on every push.

Benchmarks

Machine: AMD Ryzen 9 9950X3D (Zen 5), Windows 11, GCC 16.1 -O3, single thread, no core isolation. Input: 12302019.NASDAQ_ITCH50 (268,744,780 messages, 8.25 GB) fully resident in a RAM buffer, so no IO or page-cache effects in the measured loop. Reproduce with parse_throughput <file> and book_throughput <file> <variant>.

Parse only:

Parse + apply, whole day, all symbols (best of repeated runs; "structures" is peak RSS minus the input buffer):

Where the factors come from. Replacing std::map levels with the sorted vector is ~1.6× (touch-local scans instead of pointer chasing). Replacing the hash ref-index with paged direct indexing is another ~3×: one arithmetic dereference, no hashing, no probe chains, no rehash stalls, and near-monotonic refs keep the hot pages cached. The flat hash (fibonacci hashing, linear probing, backward-shift deletion) isolates how much of the unordered_map cost is the container itself: dropping per-node allocation and bucket-chain chasing buys ~1.7×, but it still hashes, probes and moves 40-byte slots on every delete, where the direct index just dereferences. When the key space is day-unique and near-dense, indexing beats even a good hash. The inline variant stores whole order records in the pages and skips the second indirection, but at ~10 GB of sparse pages the TLB pressure eats the win; the pooled variant keeps the live set compact and is both faster and 7× smaller. The itch-replay --book tool (mmap file, BBO tracking on) does the same day at 12.8 M msg/s.

Per-operation apply latency (rdtsc via tsc-latency, uncorrected, includes the ~10 ns timestamp-pair floor; ns):

The reduce p50 of 40 ns is the O(1) level-handle path. The p99.99 band is deep sorted- vector memmoves and fresh page allocations; the millisecond maxima are OS scheduler preemptions. Nothing was pinned or isolated, and a single uncorrected run over 268M messages will catch a few.

For context, published single-threaded parse+apply numbers elsewhere: charles-cooper/itch-order-book reports 61 ns/tick (~16.4 M msg/s) on a 2012 i7-3820 with aggregate-only levels and a 4.4 GB preallocated ref array; CppTrader reports 3.2 M msg/s for its reference book and ~9.8 M for its stripped benchmark variant on an i7-4790K. Different hardware and different feature sets, so the numbers are not directly comparable; this implementation keeps FIFO queues, bounded memory and feed-robustness checks on at all times.

Limitations

• Replay, not a live feed handler. StreamParser reassembles frames split across arbitrary chunk boundaries, but there is no MoldUDP64/SoupBinTCP session layer on top, no A/B feed arbitration, no gap or retransmission requests.
• Book-affecting messages, trades and trade voids (P/Q/B) and trading actions (H) are decoded; NOII, RegSHO, LULD and the other administrative types are framed and counted but not decoded.
• Order references are trusted to be locate-consistent (the order's stored locate wins over the message header on E/X/D/U, so a corrupt feed cannot cross-corrupt books).
• Single-threaded by design; shard symbols across instances above the library if needed.
• Latency numbers above are from an unpinned desktop Windows box with boost clocks on.

What I would do differently in production

MoldUDP64 with A/B arbitration and gap-fill feeding StreamParser; pinned cores, huge pages for the order pool, and an io_uring read path on Linux; per-symbol sharding with an SPSC handoff per shard.

License

MIT