tracebook-conformance 0.6.0

Matching-engine conformance testing, reproducible failure reduction, and honest local benchmarks

Tracebook

Conformance testing and reproducible failure analysis for matching engines.

Tracebook runs the same normalized order-lifecycle trace against an inspectable reference engine and your Rust, C++, Java, Python, or other candidate. It stops at the first semantic difference, explains what drifted, and reduces the failure to a deterministic JSONL reproducer.

Give it an engine. Get back the smallest trace that proves where it disagrees.

Quick start · Real failure · Profiles · Qualification · Adapters · CI · Architecture · Research

Quick Start

Tracebook requires Python 3.10-3.13. Install the dependency-light conformance distribution and materialize the current hash-verified adversarial suite:

If this environment already contains tracebook-sim 0.5.x, remove that legacy package owner before installing either 0.6.0 distribution:

python -m pip uninstall -y tracebook-sim

python -m pip install "tracebook-conformance==0.6.0"
tracebook-conformance sample ./tracebook-suite-v2 --suite-version v2
tracebook-conformance --help

Suite v2 is the current default; the explicit option makes the copied contract unambiguous if a later release adds another suite. These commands do not require a source checkout, NumPy, or psutil. Install tracebook-sim==0.6.0 instead when you need the simulator, benchmark, profiling, visualization, replay, or corpus commands. Users migrating that distribution from 0.5.x must use the uninstall-first handoff above:

python -m pip install "tracebook-sim==0.6.0"

The package-boundary decision documents the one-time ownership handoff and recovery command. A candidate engine remains a separate process by design. Once it speaks the versioned NDJSON protocol, run a deterministic campaign:

tracebook-conformance campaign \
  --profile fifo-limit-v1 \
  --seed 42 \
  --traces 25 \
  --events-per-trace 200 \
  --candidate-cmd './matching-engine --tracebook-stdio' \
  --corpus-dir .tracebook/corpus \
  --stop-after-first \
  --junit-output .tracebook/conformance.xml

Exit code 0 means every requested trace conformed. Exit code 1 means a semantic difference was preserved with the original failure, reduced reproducer, semantic diff, coverage evidence, campaign metadata, and JUnit result. Exit code 2 is reserved for an invalid configuration, adapter protocol, process, or filesystem failure.

For a working adapter before writing your own, use the native orderbook-rs walkthrough or the small Python process example. The Python example demonstrates protocol framing around Tracebook's own reference adapter; it is not an estimate of real engine-integration effort. Native integrations must faithfully translate lifecycle operations, numeric representations, ordered snapshots, and source IDs. The frozen design-partner study records adapter files ranging from 672 to 1,056 Rust lines and an 865-line Go adapter plus 130 lines of tests. Engine maintainers and independent evaluators can also open a structured engine qualification report. Successful and blocked attempts are both useful: the report records time to evidence, adapter size, protocol questions, profile boundaries, and whether the result belongs in the candidate project's CI.

A Real Four-Event Rust Failure

Flash's matching-engine benchmark reported a historical FIFO defect in an orderbook-rs dependency: after a partial fill, the oldest maker's remainder moved behind a later maker at the same price. Upstream tracked the defect as orderbook-rs #88 and fixed it in PR #131.

Flash's merged schema-v1 export localizes its canonical stream at sequence 15738. Tracebook converts the exact 15,739-message prefix and reduces it to four actual workload events. Independently, seed 42 generates a portable Tracebook probe whose first incorrect trade is event 173; that trace also reduces to four events while preserving the same maker-priority defect.

The reduced trace then becomes an ordinary CI regression: it must reproduce against the affected revision and pass against orderbook-rs 0.12.0.

• Read the provenance and reduction case study
• Inspect the Flash schema-v1 divergence
• Inspect the Flash-derived four-event regression
• Inspect the four-event JSONL reproducer
• Review the native Rust adapter

This is a retrospective import of a defect Flash discovered, not a Tracebook discovery claim. The value Tracebook adds is deterministic generation, first-divergence localization, minimization, replay, and a reusable regression artifact.

What It Compares

Surface	Comparison
Outcomes	Applied or rejected status plus stable rejection reason
Trades	Ordered maker/taker source IDs, side, price, and quantity
Resting state	Full price-time queue order, remaining quantity, owner, and symbol
Lifecycle	New, cancel, reduce, replace, clear, duplicate, and inactive requests
Instructions	Limit, market, IOC, FOK, and configured self-trade prevention
Isolation	Independent multi-symbol books and canonical state hashes
Evidence	Candidate-independent semantic coverage for compared events

Tracebook transfers compact observations after each event and requests a full queue snapshot only at divergence and completion. Candidate execution time is not presented as engine latency because it includes process, serialization, and adapter overhead.

Portable Semantic Profiles

Profiles are versioned contracts. Existing names do not silently acquire new behavior, so seeds and campaign hashes remain reproducible.

Profile	Contract
fifo-limit-v1	FIFO limit orders, partial fills, cancel, reduce, replace, clear, duplicates, inactive requests, and multiple symbols
fifo-full-v1	fifo-limit-v1 plus market, IOC, and FOK instructions
fifo-partial-fill-v1	Portable FIFO lifecycle plus the real partial-fill continuation probe
Bundled suite v2 (current default)	Nine adversarial fixtures covering FIFO, instructions, encounter-based STP, tick grids, deep cancellation, multiple symbols, and pro-rata

Campaigns use specified SplitMix64 trace seeds and candidate-independent generation. Immutable suite v1 remains available explicitly for reproducing its original eight-case contract. Reports distinguish semantic workload coverage from Python source coverage.

Read the profile, protocol, hashing, and minimizer contracts.

Profile Qualification

Use one command for the fixed and generated evidence inside the contract your engine actually claims:

tracebook-conformance qualify \
  --profile fifo-limit-v1 \
  --seed 42 \
  --traces 25 \
  --events-per-trace 200 \
  --candidate-cmd './matching-engine --tracebook-stdio' \
  --output-dir .tracebook/qualification

The atomic bundle includes qualification.json, qualification.xml, selected fixed-suite results, campaign results, semantic coverage, and any minimized failure. Qualification selection contract v1 uses bundled suite v2 by default; these are separate version boundaries. fifo-limit-v1 qualification does not run STP or pro-rata cases, so unsupported features stay explicit without becoming false failures.

Read the research-grounded roadmap and adoption experiment.

Architecture

flowchart LR
    G["Versioned campaign"] --> R["Runner"]
    T["Normalized trace"] --> R
    R --> O["Reference engine"]
    R --> A["NDJSON adapter"]
    A <--> E["Candidate engine"]
    O --> D["Semantic diff"]
    A --> D
    D --> P["JSON / JUnit"]
    D --> M["Minimizer"] --> B["Failure bundle"]

The process boundary keeps the candidate independent of Tracebook's Python implementation. Adapters translate only between native engine operations and the canonical protocol.

Open the detailed component map.

Engine Adapters

Candidate	Native surface	Evidence
orderbook-rs 0.12.0	Rust FIFO lifecycle, market/IOC/FOK, STP, deterministic trade IDs, consumption-order queue snapshots	Conformant generated FIFO campaign; 7/9 standard cases with pro-rata and one STP policy difference explicit; semantics reviewed in upstream issue #203 and Tracebook issue #57
gocronx/matcher 0.2.0	Pinned Rust FIFO limit lifecycle over native events and snapshot format v1	fifo-limit-v1 qualification passes 3/3 fixed cases, 25/25 generated traces, and 10/10 capabilities; upstream observation and replacement questions remain explicitly pending in issue #7
Historical orderbook-rs issue #88	Exact affected Rust dependency behind an opt-in Cargo feature	Flash sequence 15738 and generated event 173 independently reduce to four-event regressions
faulty-orderbook-adapter	Real Rust engine plus one documented injected queue-priority fault	Synthetic negative control reduced from event 173 to five causal events
PythonMatchingEngine	Pinned FIFO limit lifecycle, cancellation, reduction, replacement, clear, and queue snapshots	Compatible trace passes; unsupported instructions, STP, tick grid, and pro-rata remain visible differences
Your engine	Any language that can read and write newline-delimited JSON	Start from the adapter contract and gate only the profile your engine claims

A divergence means two configured contracts disagree. It does not, by itself, declare either project incorrect.

Failure Artifacts

Every stopped campaign writes an atomic, content-addressed bundle:

Artifact	Purpose
campaign.json	Seed, profile, generator version, trace count, campaign hash, and coverage
failure.json	Failure class, exact semantic path, engines, IDs, and reproduction expectation
original.jsonl	Complete generated history through the first divergence
reduced.jsonl	Deterministic minimized reproducer
minimization.json	Run budget, reduction evidence, and one-minimal status
JUnit XML	Portable test-report projection; the sample workflow uploads it, and a separate CI reporter is required for pull-request annotations

Replay a stored expectation without regenerating anything:

tracebook-conformance reproduce \
  .tracebook/corpus/<failure-id>/reduced.jsonl \
  --candidate-cmd './matching-engine --tracebook-stdio'

Continuous Integration

This minimal job installs the public PyPI release. It assumes that a faithful stdio adapter already exists and can be built in the candidate repository; the adapter is the substantive engine-specific integration work. Replace the placeholder build and candidate commands:

name: Matching engine conformance

on: [pull_request]

jobs:
  conformance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v6
        with:
          python-version: "3.12"
      - run: python -m pip install "tracebook-conformance==0.6.0"
      - run: make build
      - run: |
          tracebook-conformance qualify \
            --profile fifo-limit-v1 \
            --seed 42 \
            --traces 25 \
            --events-per-trace 200 \
            --candidate-cmd './build/matching-engine --tracebook-stdio' \
            --output-dir artifacts/qualification

The repository includes a copy-paste workflow with artifact upload and a short CI integration guide. qualify prints a concise human summary—profile, fixed and generated counts, semantic coverage, PASS/FAIL, and reduced-trace details—to the job log. The workflow uploads the lossless JSON and JUnit bundle for deeper review; it does not install a JUnit annotation reporter.

Reference Semantics

The bundled Python engine is intentionally small and inspectable. It supports:

• FIFO and pro-rata allocation;
• limit, market, IOC, and FOK orders;
• cancellation, priority-preserving reduction, and priority-losing replacement;
• configurable self-trade prevention;
• detached queue snapshots and deterministic record/replay;
• stable structured outcomes for validation and lifecycle failures.

It is the executable oracle for declared Tracebook profiles, not a production exchange engine.

Additional Workbench Tools

The conformance wedge sits on top of useful lower-level tooling rather than removing it:

Tool	Use
Normalized event replay	Replay CSV, JSON, or JSONL lifecycle streams with source-ID preservation
Coinbase Exchange L3	Validate snapshot/feed sequence and normalize public Level 3 messages
Verified corpora	Bind local datasets to manifests, golden states, and reproducible import benchmarks
Performance reports	Separate generation, matching, lifecycle, memory, and monitoring measurements
Local dashboard	Inspect simulation and profiling artifacts without a hosted service

These tools support correctness investigations. Tracebook does not claim that local Python timings predict production Rust or C++ latency.

Research And Citation

The first public conformance field note records the 15,739-to-four-event reduction, external semantic review, public- package qualification evidence, negative guided-generation result, and claim boundaries in one citable narrative.

Use CITATION.cff for software citation metadata. Release-specific evidence remains bound to its profile, candidate revision, workload identity, and artifact hashes.

Project Boundaries

Tracebook is not:

• a production exchange or broker gateway;
• a complete strategy backtester or portfolio platform;
• a realistic multi-agent market economy;
• a source of redistributable exchange data;
• a universal performance ranking for matching engines.

Read the product position and comparison with adjacent projects.

Platform Support

The release matrix runs Python 3.10-3.13 on Ubuntu. Atomic campaign and qualification bundles rely on descriptor-relative filesystem operations, so Ubuntu is currently the only release-tested platform for that evidence path. The implementation fails closed when those operations are unavailable; Windows qualification artifact publication is not currently supported. Manually measured macOS runs exist, but macOS is not a release-gated support target. Other library and CLI surfaces may work elsewhere without constituting a cross-platform support claim.

Contributing

Tracebook supports Python 3.10 through 3.13 on the platform boundary above and is alpha software. The real four-event orderbook-rs case study currently requires a source checkout.

git clone https://github.com/Taz33m/tracebook.git
cd tracebook
python -m venv venv
source venv/bin/activate
python -m pip install -e ".[dev]"
python -m pip install -e "./packaging/tracebook-sim[analysis,capture,dashboard]"
make quality

Start with the contribution guide, the command reference, or an issue. Security reports follow the private process in SECURITY.md.

License

MIT