Market Data Reliability & Acceleration Platform (MDRAP)

A high-performance financial market infrastructure platform designed to ingest, validate, accelerate, and reconcile noisy, delayed, duplicated, and inconsistent market data from disparate exchanges and internal feeds into a unified, ultra-low-latency canonical stream with mathematical reliability scoring, cryptographic lineage auditing, and institutional execution analytics.

📖 Complete Documentation: See the Comprehensive Operator & User Manual for full syntax, flags, hotkeys, and role-based workflows.

High-Level Architecture

flowchart TD
    subgraph INGESTION ["1. Market Ingestion & Direct Streaming Feeds"]
        POLY[Polygon.io WebSocket<br/>US Equities & Crypto Q/T/AM]
        DBN[Databento Binary DBN<br/>CME/Nasdaq MBP-1/10 & Trades]
        B[Binance / Coinbase / Kraken<br/>Crypto WebSockets & REST]
        SIM[Deterministic Feed Simulator<br/>Seeded Faults & Injections]
        FSUP[Streaming Feed Supervisor<br/>Thread-Safe Low-Contention Queue]
    end

    subgraph SECURITY ["2. Security & Gatekeeper (Spec §19)"]
        RL[Token Bucket Rate Limiter<br/>20,000 eps per IP/Key]
        SAN[Regex & Range Payload Sanitizer]
        HMAC[HMAC-SHA256 Signature Verification<br/>Constant-Time Digest]
        RBAC[RBAC Entitlement Guard<br/>VIEWER / OPERATOR / ADMIN]
    end

    subgraph PIPELINE ["3. Validation, Acceleration & Consensus Pipeline"]
        GW[Gateway & Normalization<br/>RawEvent -> CanonicalEvent]
        QE[7-Rule Quality Engine<br/>Schema, Dedup, Gap, Order, Stale, Crossed, 3-Sigma]
        FP[Native C Hot Path Accelerator<br/>8,192 Symbols | 18.6M eps | 50.0 ns]
        WD[Source Watchdog & Failover Circuit Breaker<br/>Silence & Degradation Monitoring]
        BBO[Synthetic Consolidated BBO<br/>5-Venue Multi-Exchange NBBO]
        DEPTH[Consolidated L2 Order Book<br/>Multi-Venue Depth Aggregation & VWAP Curves]
    end

    subgraph STORAGE ["4. Columnar & Batched Storage, Archive & Audit (Spec §14, §19, §26)"]
        CAN[(canonical_events<br/>WAL SQLite Batch)]
        QUAR[(quarantine<br/>Never Silently Drop)]
        LIN[(lineage<br/>Transformation Lineage Proof)]
        AUD[(audit_log<br/>Merkle Hash Chained)]
        ARC[[Immutable Raw JSONL Archive<br/>Write-Ahead Partitioned Log]]
        COL[(DuckDB Columnar Store<br/>SIMD Resampling & Parquet Export)]
    end

    subgraph PRESENTATION ["5. Presentation, IPC & Institutional Export"]
        DAEMON[Headless Streaming Daemon<br/>Non-blocking Socket IPC]
        SHM[Binary Shared Memory Transport<br/>Zero-Copy Ring Buffer]
        LIVE[In-Place Live Terminal Ticker<br/>Cursor-Repositioned Rich HUD]
        CHART[Visual Candlestick Terminal Chart<br/>Unicode Wicks & Outlier Percentile Scaling]
        EXCEL[Institutional 5-Tab Excel Exporter<br/>XLSX Financial Model & CSV Packages]
    end

    POLY & DBN & B & SIM --> FSUP --> RL
    RL --> SAN --> HMAC --> RBAC --> GW
    GW --> ARC
    GW --> QE
    QE <--> FP
    QE --> BBO & DEPTH
    QE --> WD
    BBO & DEPTH --> DAEMON & SHM
    QE --> CAN & QUAR & LIN & AUD
    CAN --> COL
    BBO & DEPTH & COL --> LIVE & CHART & EXCEL

Key Platform Capabilities

1. 7-Rule Data Quality Engine (Spec §7)

• Structural Schema Validation: Rejects malformed JSON and missing sequence/timestamp attributes.
• Sliding-Window Deduplication: Identifies exact and sliding-window duplicate packet bursts without memory bloat.
• Monotonic Sequence Gap Detection: Detects missing exchange packets and penalizes feed reputation.
• Out-of-Order Sequencing: Catches retrograde arrival events across jittery network paths.
• Timestamp Staleness Evaluation: Flags lagging feeds exceeding max latency thresholds.
• Crossed Quote Detection: Flags invalid book states where $\text{Bid} > \text{Ask}$.
• Statistical Price Sanity Checks: Evaluates sudden price jumps ($>3\sigma$) using Welford's online variance algorithm.
• Strict Quality Priority: Non-downgradable status progression: INVALID > SUSPICIOUS > VALID. Quarantines bad data; never silently drops events.

2. Native C Hot-Path Accelerator (fastpath.c)

• Pure C implementation compiled with GCC -O3 into a native shared library (fastpath.dll).
• Capacity Expanded to 8,192 Symbols ($2^{13}$) and 32 feed sources with dynamically allocated, SIMD-aligned contiguous memory arrays.
• Zero-Division Bitshift Slot Indexing: Computes slot offsets in 1 CPU cycle: (source_id << 13) | instrument_id.
• Ultra-High Throughput: Evaluates 18,669,082 events/sec (50.0 nanoseconds/event) in batch mode.
• Seamless, transparent boundary fallback to pure Python if instrument universe exceeds 8,192 symbols.

3. Direct High-Throughput Streaming Feed Handlers (src/polygon_feed.py, src/databento_feed.py, src/feed_handler.py)

• Polygon.io WebSocket Connector: Streams high-frequency US Equities and Crypto quotes (Q), trades (T), and aggregate bars (AM) with API key authentication, multiplexed subscription channels, exponential backoff reconnection, and built-in offline wire-format mock generators.
• Databento Binary Encoding (DBN) Ingestion: Sub-microsecond binary record decoding using struct.Struct with C-struct layouts for Databento DBN formats (MBP-1, MBP-10, TradeMsg), nanosecond UTC epoch timestamps, fixed-point price scaling ($10^9$), dynamic symbol resolution, and live TCP / .dbn file / synthetic binary packet streaming.
• Unified Streaming Feed Supervisor: Coordinates multiple streaming providers into a bounded, low-contention queue with ring eviction to guarantee real-time latency and zero stale queue backlog. Ingestion telemetry tracks throughput (eps), dropped frames, and provider health.

4. DuckDB Columnar Time-Series Storage & SIMD Analytics (src/columnar.py, Spec §14, §26)

• High-Throughput Embedded Columnar Store: Embedded in-process DuckDB analytical engine with SIMD-vectorized execution for ultra-fast billion-tick historical queries.
• Zero-Copy SQLite Sync: Directly attaches operational SQLite databases via DuckDB's native SQLite scanner (ATTACH '...' AS sqldb (TYPE SQLITE)) and bulk copies 300,000+ ticks in ~2.8s into columnar storage.
• Vectorized Resampling & Aggregation:
  • Resamples trade ticks into OHLCV candles via arg_min(price, exchange_timestamp) and arg_max(price, exchange_timestamp) in a single pass without window functions or self-joins (37.2x faster than SQLite).
  • Computes exact institutional VWAP (sum(P*Q) / sum(Q)) and total notional across tens of thousands of trades in <10ms (63.3x faster than SQLite).
  • Vectorized bid-ask spread analytics and crossed-market anomaly tracking.
  • Sub-microsecond latency quantile extraction (p50, p90, p95, p99, p99.9) across millions of records via quantile_cont().
  • Discrete price-rung volume profile distribution.
• Apache Parquet Compressed Export: Direct export of tick universes to compressed .parquet files with Zstandard (zstd), Snappy, or GZIP compression (299k ticks compressed to 1.52 MB).

5. Consolidated Level-2 Market Depth & Real-Time VWAP Slicing (src/depth.py)

• Real-time aggregation of multi-venue order books into a consolidated L2 depth ladder.
• Dynamic VWAP Slippage Curve calculation: computes estimated executed price, basis point slippage, and market impact across any requested order size.
• Real-time bid/ask liquidity imbalances and multi-venue depth visualization via cli.py depth and cli.py vwap.

6. Institutional Financial Model & 5-Tab Excel Exporter (src/exporter.py)

• Translates live ticks, order book depth, and quality metrics into institutional-grade Microsoft Excel (.xlsx) workbooks:
  • Tab 1: Executive Summary & Microstructure KPIs: Total volume, VWAP, spreads, crossed quote count, tick count.
  • Tab 2: Consolidated Market Depth: Multi-venue aggregated bid/ask ladders with depth visualization.
  • Tab 3: VWAP Slippage Curve: Execution slippage schedule across order tranches.
  • Tab 4: Quality & Quarantine Audit: Detailed record of rejected/quarantined events with exact failure reasons.
  • Tab 5: OHLCV Candlesticks: 5-second candle aggregates (Open, High, Low, Close, Volume, Trades).
• Automatic fallback to structured CSV report directories if openpyxl is not installed.
• One-command generation and instant launch: mdrap export AAPL --open.

7. In-Place Live Terminal Ticker & Candlestick Charts (src/terminal_display.py)

• Zero-Scroll In-Place Display: Updates live market quotes and candlestick charts in-place using ANSI cursor repositioning without cluttering terminal history.
• High-Resolution Candlestick Visualization: Renders 3-character columns (█, │, ┼) with distinct body margins and box-drawing wicks.
• Outlier-Resilient Percentile Scaling: Visual bounds clamped to the 10th–90th price percentiles so extreme anomalies never crush normal candles into a flat line.
• Aligned Volume Histogram: Synchronized volume bars underneath each candle column.
• Dedicated modes: full live ticker dashboard (mdrap live), ticker-only mode (mdrap live AAPL --ticker-only), and standalone historical chart viewer (mdrap chart AAPL).

8. Synthetic Consolidated NBBO & Multi-Market Connectors (src/live.py, src/bbo.py)

• Ingests real-time prices across major global crypto exchanges (Binance, Coinbase, Kraken, OKX, Bybit) and global equities (AAPL, MSFT, NVDA, TSLA, SPY, QQQ, GOLD).
• Computes global tightest bid/ask spread, mid-price, and real-time venue attribution with crossed-market flags.

9. Live Watchdog & Automated Source Failover (src/watchdog.py)

• Real-time source reliability tracking with silence detection and degradation alerts.
• Automated failover circuit breaker: dynamically evicts silent or corrupt feeds from the consolidated book with hysteresis recovery.

10. Immutable Raw Event Archive & Deterministic Replay (src/archive.py)

• Date- and source-partitioned write-ahead JSONL log capturing every raw event before processing.
• Deterministic event replay engine allows historical backtesting and auditing through the complete pipeline.

11. Enterprise Security & Cryptographic Merkle Audit (src/security.py)

• HMAC-SHA256 Feed Authentication: Anti-spoofing signature verification with pre-shared feed secrets.
• Role-Based Access Control (RBAC): Privilege boundaries across VIEWER, OPERATOR, and ADMIN.
• Token Bucket Rate Limiting: Shields pipeline against denial-of-service and quote flooding ($20,000\text{ eps}$).
• Tamper-Evident Merkle Audit Log: Cryptographically chained SHA-256 hash trail in SQLite with standalone verification via mdrap audit --verify.

12. Binary Shared Memory IPC Transport (src/shm.py, src/protocol.py)

• Zero-copy lock-free ring buffer for ultra-low latency IPC between the ingestion daemon and trading algorithms.

Architectural Progression & Benchmarks

Measured on identical 10,000-event workloads (seed=42) with fixed ground-truth errors:

Latency Hierarchy & Physical Bounds

• Native C Batch (50.0 ns) vs Pipeline (15.7 µs): The 50.0 ns figure measures the C accelerator alone operating on pre-batched contiguous arrays in CPU L1 cache. The 15.7 µs figure is the same accelerator measured end-to-end inside the full pipeline (gateway → quality engine → reconciliation → storage).
• Physics of the "~5 Nanosecond" Myth: At 4.0 GHz, one CPU cycle is 0.25 nanoseconds; 5 nanoseconds is exactly 20 CPU cycles. Software running on general-purpose operating systems cannot receive network packets, parse payloads, and evaluate state in 5 nanoseconds (PCIe bus transfer from NIC to RAM alone takes 100–250 ns). Sub-20ns latencies are only physically possible in dedicated hardware FPGA gate logic (e.g. AMD Xilinx UltraScale+).

The 3 Measurable Platform Latency Tiers

Quick Start

Installation

Clone the repository and install optional dependencies:

git clone https://github.com/Aryan-20-04/mdrap.git
cd mdrap

# MDRAP has ZERO mandatory dependencies (runs 100% on standard library).
# Install optional visualization, financial exporter, and test packages:
pip install -r requirements.txt

Compile the Native C accelerator (optional — transparent pure-Python fallback is included):

python build_fastpath.py

1. Interactive Wall Street & Quant Terminal

Run mdrap (or .\mdrap.bat) with zero arguments to enter the pre-warmed shell:

.\mdrap.bat

mdrap> LIVE AAPL         # Live market stream with in-place updating table & candlestick chart
mdrap> CHART BTC/USD     # Standalone visual candlestick chart with volume histogram
mdrap> DEPTH AAPL        # Consolidated Level-2 market depth ladder
mdrap> VWAP AAPL 1000    # Calculate VWAP slippage curve for 1,000 shares
mdrap> EXPORT AAPL --open# Export 5-tab financial model workbook to Excel and open it
mdrap> BTC BBO           # 5-Venue Consolidated NBBO across Binance, Coinbase, Kraken, OKX, Bybit
mdrap> TOP               # Launch real-time full-screen service cockpit
mdrap> STRESS            # Run multi-directional stress tests and 1M-1B scale analysis
mdrap> AUDIT             # Cryptographically verify tamper-evident Merkle hash chain
mdrap> ?                 # Open clean 4-quadrant command palette

You can also run all commands directly from PowerShell / CMD / Bash:

.\mdrap.bat live AAPL                 # In-place terminal ticker & candlestick chart
.\mdrap.bat live AAPL --ticker-only  # Clean single-table ticker view
.\mdrap.bat chart AAPL                # Unicode candlestick chart
.\mdrap.bat depth AAPL                # L2 market depth ladder
.\mdrap.bat vwap AAPL --size 500      # Execution slippage schedule
.\mdrap.bat export AAPL --open        # Generate Excel model (.xlsx) & open immediately
.\mdrap.bat bbo BTC/USD               # 5-Venue crypto NBBO quote
.\mdrap.bat top                       # Terminal service cockpit
.\mdrap.bat stress --module quality   # Benchmark C hotpath (18.6M eps)

2. Headless Daemon & Live IPC Streaming

In Terminal 1, start the background ingestion daemon:

# High-speed simulated multi-venue feed
.\mdrap.bat daemon --speed 2000

# Or live multi-venue market feeds (Binance, Coinbase, Kraken, OKX, Bybit)
.\mdrap.bat daemon --live

In Terminal 2, stream clean canonical ticks directly to stdout or pipe into trading algorithms:

# Formatted ANSI color stream
.\mdrap.bat sub BTC/USD

# Raw JSON stream for automated algorithmic bots or jq
.\mdrap.bat sub BTC/USD --json | jq '{bid: .bbo.bid, ask: .bbo.ask}'

In Terminal 3, launch the real-time terminal monitor:

.\mdrap.bat top

Keyboard-First Speed Ergonomics

MDRAP provides sub-second keyboard ergonomics inspired by Bloomberg terminals (<TICKER> <FUNCTION> <GO>), eliminating long CLI commands for high-speed trading desks and quant operations:

1. Wall Street 2-Token Mnemonic Shell

Inside the interactive shell (.\mdrap.bat or ./mdrap), type:

• AAPL C -> Candlestick Chart HUD
• BTC D -> Consolidated Level-2 Depth Book Ladder
• AAPL V -> Institutional Real-Time VWAP Slippage Curve
• AAPL P -> Polygon.io Streaming WebSocket Feed
• ES B -> Databento Binary DBN Fast Streaming Feed
• AAPL X -> 5-Tab Financial Model Excel Export (Auto-Opens)
• AAPL (ticker only) -> Instant Consolidated NBBO Quote
• 1 to 9 -> Instant 1-Key Launches (1 = Live BTC, 2 = NBBO, 3 = Cockpit, 4 = Chart, etc.)

2. Live In-Stream Hotkeys (Non-Blocking Keystrokes)

During any live stream (mdrap live, mdrap top, mdrap depth), hands never leave the keyboard:

• [Space] -> Freeze / Unfreeze Frame: Pauses the live rendering so you can inspect fast-moving prints and L2 depth levels without them scrolling away. Pressing [Space] again resumes real-time updates.
• [q] or [Esc] -> Instant Clean Exit: Cleanly restores terminal cursor without Python stack traces.
• [c] -> Toggle Candlestick HUD: Show or hide the inline technical chart.
• [d] -> Toggle Level-2 Depth Ladder: Show or hide the consolidated depth rungs.
• [Tab] / [1-9] -> Switch Active Symbol Focus: Cycle or jump between active universe tickers on the fly.

3. Single-Letter OS CLI Shortcuts

From your terminal (PowerShell, CMD, or bash):

mdrap c AAPL    # Candlestick Chart
mdrap d BTC     # Level-2 Depth Ladder
mdrap v AAPL    # Real-Time VWAP Curve
mdrap p AAPL    # Polygon.io Streaming Feed
mdrap b ES      # Databento DBN Streaming Feed
mdrap x AAPL    # 5-Tab Excel Export
mdrap AAPL      # Instant Best Bid & Offer Quote

CLI Command Reference

Verification & Testing

MDRAP includes a rigorous test suite of 238 automated unit, integration, security, and chaos tests covering 100% of pipeline stages:

# Run the complete automated test suite
pytest tests/ -v

# Run the comprehensive platform verification scorecard
.\mdrap.bat test-all

All tests execute with deterministic seeds and verify ground-truth fault detection, boundary conditions, C fallback mechanisms, and zero memory leaks.

Repository Structure

mdrap/
├── cli.py               # Unified CLI, interactive quant shell, and command dispatcher
├── mdrap.bat            # Windows zero-config launcher script
├── build_fastpath.py    # Native C accelerator build script (GCC / Clang / MSVC)
├── config.yaml          # Externalized quality thresholds, anomaly windows & security policies
├── pyproject.toml       # PEP 518/621 project configuration, scripts & package packaging
├── requirements.txt     # Optional runtime & dev dependencies (pure stdlib default)
├── LICENSE              # MIT License
├── .gitignore           # Production-grade gitignore for Python, C artifacts, data, and reports
│
├── src/                 # Core MDRAP Platform Engine
│   ├── analytics.py     # 5s OHLCV candles, bid-ask spread tracking, Welford realized volatility
│   ├── archive.py       # Immutable write-ahead JSONL archive & deterministic replay
│   ├── bbo.py           # Synthetic Consolidated BBO (NBBO) multi-venue engine
│   ├── benchmark.py     # Micro-benchmark harness & ground-truth scoring
│   ├── broker.py        # Thread-safe in-memory streaming bus with backpressure
│   ├── chaos.py         # Automated failure injection & chaos drill suite (Spec §15)
│   ├── client.py        # Low-latency streaming client SDK with reconnect logic
│   ├── columnar.py      # DuckDB columnar engine, zero-copy SQLite scanner & Parquet exporter
│   ├── config.py        # Central configuration manager & asset-class override resolver
│   ├── dashboard.py     # Real-time terminal pipeline telemetry HUD
│   ├── databento_feed.py# Databento DBN binary decoding (MBP-1, MBP-10, Trades) & streaming
│   ├── depth.py         # Consolidated L2 depth aggregation & VWAP slippage curve engine
│   ├── exporter.py      # Institutional 5-tab Excel (.xlsx) & CSV financial model exporter
│   ├── fastpath.c       # Native C hot path accelerator (8,192 symbols, GCC -O3)
│   ├── fastpath.dll     # Pre-compiled high-performance native C shared library
│   ├── fastpath.py      # C ctypes wrapper with transparent pure-Python boundary fallback
│   ├── feed_handler.py  # Unified streaming supervisor (Polygon, Databento, Crypto WebSockets)
│   ├── gateway.py       # Ingestion gateway, timestamp recorder, and schema normalizer
│   ├── live.py          # Multi-exchange connectors (Binance, Coinbase, Kraken, OKX, Bybit, Equities)
│   ├── metrics.py       # High-resolution hardware nanosecond latency & percentile telemetry
│   ├── models.py        # CanonicalEvent, RawEvent, QualityStatus, Reason dataclasses
│   ├── pipeline.py      # V1 synchronous baseline pipeline (ground-truth reference)
│   ├── pipeline_v2.py   # V2 decoupled streaming pipeline with bounded queue broker
│   ├── polygon_feed.py  # Polygon.io streaming WebSocket connector (Quotes, Trades, Bars)
│   ├── protocol.py      # Binary serialization & framing protocol for IPC
│   ├── quality.py       # 7-rule data quality evaluation engine (Spec §7)
│   ├── reconciliation.py# Multi-feed cross-reconciliation & dynamic reliability scoring
│   ├── security.py      # HMAC-SHA256 signing, RBAC, Token Bucket rate limiter, Merkle audit log
│   ├── service.py       # Headless streaming daemon, authenticated socket, and service cockpit
│   ├── shm.py           # Lock-free binary shared memory ring buffer IPC
│   ├── simulator.py     # Deterministic feed simulator with seeded anomaly injections
│   ├── storage.py       # Batched SQLite store (canonical, quarantine, lineage, audit, health)
│   ├── stresstest.py    # Multi-directional stress benchmarks & 1B-scale profiling
│   ├── term.py          # Auto-responsive terminal styling with stdlib fallback
│   ├── terminal_display.py # In-place live terminal ticker & ANSI candlestick chart renderer
│   ├── watchdog.py      # Live source watchdog, silence detection & automated failover
│   └── ws_feed.py       # Async WebSocket live market feed connector
│
├── tests/               # 238 Automated Unit & Integration Tests (100% Passing)
│   ├── test_analytics.py
│   ├── test_archive.py
│   ├── test_bbo.py
│   ├── test_chaos.py
│   ├── test_cli.py
│   ├── test_client.py
│   ├── test_columnar.py
│   ├── test_config.py
│   ├── test_databento_feed.py
│   ├── test_depth.py
│   ├── test_entitlements.py
│   ├── test_export.py
│   ├── test_fastpath.py
│   ├── test_feed_handler.py
│   ├── test_hardening.py
│   ├── test_keyboard_shortcuts.py
│   ├── test_live.py
│   ├── test_pipeline_integration.py
│   ├── test_polygon_feed.py
│   ├── test_protocol.py
│   ├── test_quality.py
│   ├── test_security.py
│   ├── test_service.py
│   ├── test_shm.py
│   ├── test_stresstest.py
│   ├── test_system_limitations.py
│   ├── test_terminal_display.py
│   ├── test_v2_streaming.py
│   ├── test_vwap.py
│   ├── test_watchdog.py
│   └── test_ws_feed.py
│
├── docs/                # Architecture & Platform Specifications
│   ├── USER_GUIDE.md         # Comprehensive Operator & User Manual (all commands, flags, workflows)
│   ├── architecture.md       # Full platform architecture specification (V1–V4)
│   ├── audit-log-format.md   # Merkle tree hash chain format & audit specification (§19)
│   ├── benchmark-methodology.md # Scientific measurement standards & latency hierarchy
│   ├── data-model.md         # Canonical event schema & lineage data model
│   └── quality-rules.md      # 7-rule data quality evaluation definitions & fault scoring
│
└── benchmarks/          # Immutable benchmark runs, JSON reports, and cProfile traces

License

This project is licensed under the MIT License — see the LICENSE file for details.