orderpulse 0.2.11

High-performance exchange feed parser and orderflow analytics engine with Rust and Python bindings

FastReader

High-performance binary market-data parser built in Rust with Python bindings using PyO3.

FastReader is designed for low-latency parsing and sequential processing of binary order and trade packets. It provides efficient market-data extraction, order book generation, top-level market depth calculation, and Python interoperability.

---

Table of Contents

1. Overview
2. Key Features
3. Architecture
4. Project Structure
5. Core Components
6. Data Flow
7. Installation
8. Build Instructions
9. Python Integration
10. API Reference
11. Usage Examples
12. Orderbook Engine
13. Performance Considerations
14. Error Handling
15. Example Output
16. Recommended Improvements
17. Future Enhancements
18. License

---

Overview

FastReader is a Rust-powered binary parser optimized for market-data systems where throughput and deterministic processing matter.

The library:

• Parses binary order packets
• Parses binary trade packets
• Supports sequential message streaming
• Generates real-time orderbook snapshots
• Calculates mid-price dynamically
• Exposes Rust functionality to Python through PyO3

The project follows a modular architecture for maintainability and scalability.

---

Key Features

High Performance Parsing

Uses Rust memory safety and zero-cost abstractions for fast packet decoding.

Sequential Message Processing

Supports message-by-message iteration using internal indexing.

Python Bindings

Exposes Rust backend directly to Python.

Token Filtering

Allows filtering by instrument token during initialization.

Orderbook Reconstruction

Maintains orderbook state and generates top bid/ask levels.

Mid Price Calculation

Calculates real-time midpoint from best bid and ask.

Modular Design

Separated parsing, structures, and orderbook logic.

---

Architecture

                    +----------------------+
                    |   Binary Market Data |
                    |      (.bin file)     |
                    +----------+-----------+
                               |
                               v
                +----------------------------+
                |  read_trd_ord_only.rs      |
                |  Binary Packet Parser      |
                +-------------+--------------+
                              |
                              v
                +----------------------------+
                |       structure.rs         |
                |  Message / Packet Models   |
                +-------------+--------------+
                              |
                              v
                +----------------------------+
                |          lib.rs            |
                |  Python Interface Layer    |
                |  Sequential Processing     |
                +-------------+--------------+
                              |
          +-------------------+-------------------+
          |                                       |
          v                                       v
+---------------------+               +----------------------+
| Message Formatting  |               | orderbook.rs         |
| Human Readable View |               | OrderBook Engine     |
+---------------------+               +-----------+----------+
                                                  |
                                                  v
                                   +----------------------------+
                                   | Mid Price + Top 5 Levels   |
                                   +----------------------------+
                                                  |
                                                  v
                                   +----------------------------+
                                   | Python Consumer / Strategy |
                                   +----------------------------+

---

Project Structure

fastreader/
│
├── src/
│   ├── lib.rs
│   ├── structure.rs
│   ├── read_trd_ord_only.rs
│   └── orderbook.rs
│
├── Cargo.toml
├── pyproject.toml
└── README.md

---

Core Components

1. lib.rs

Main library entry point.

Responsibilities:

• Python module registration
• Expose Rust APIs to Python
• Message formatting
• Sequential processing
• Orderbook integration

Key Objects:

ReadMsgFromBinary

Main public interface exposed to Python.

---

2. structure.rs

Defines all packet structures.

Typical responsibilities:

• Binary packet layout
• Message enums
• Header structures
• Order packet structures
• Trade packet structures

Example:

pub enum Message {
    Order(OrderPacket),
    Trade(TradePacket),
}

---

3. read_trd_ord_only.rs

Binary parser implementation.

Responsibilities:

• Read binary file
• Deserialize packets
• Detect message type
• Construct Message enums

Core Function:

read_messages(path)

Returns:

Vec<Message>

---

4. orderbook.rs

Maintains live orderbook state.

Responsibilities:

• Process order updates
• Process trade updates
• Maintain bid/ask levels
• Generate top market depth
• Calculate mid-price

Main Engine:

OrderBookManager

---

Data Flow

Binary File
    ↓
Packet Parsing
    ↓
Message Enum Creation
    ↓
Sequential Processing
    ↓
Orderbook Updates
    ↓
Market Depth Extraction
    ↓
Python Strategy Layer

---

Installation

Rust Requirements

Install Rust:

curl https://sh.rustup.rs -sSf | sh

Verify:

rustc --version
cargo --version

---

Python Requirements

pip install maturin

---

Build Instructions

Development Build

maturin develop

Release Build

maturin develop --release

Build Wheel

maturin build --release

---

Python Integration

The module is exposed as:

import fastreader

Main class:

fastreader.ReadMsgFromBinary

---

API Reference

Constructor

ReadMsgFromBinary

reader = fastreader.ReadMsgFromBinary(
    path,
    token=None
)

Parameters:

Parameter	Type	Description
path	str	Binary market-data file
token	int	Optional instrument filter

---

Methods

all_messages

Returns all messages.

reader.all_messages(limit=None)

Example:

msgs = reader.all_messages(10)

for msg in msgs:
    print(msg)

---

order_messages

Returns only order messages.

reader.order_messages(limit=None)

Example:

orders = reader.order_messages(5)

---

trade_messages

Returns only trade messages.

reader.trade_messages(limit=None)

Example:

trades = reader.trade_messages(5)

---

next_message

Returns next sequential message.

msg = reader.next_message()

Example:

while True:
    msg = reader.next_message()

    if msg == "END":
        break

    print(msg)

---

print_messages

Print sequential messages internally.

reader.print_messages(limit=10)

---

orderbook

Generates orderbook snapshots.

reader.orderbook()

Output contains:

• local timestamp
• exchange timestamp
• mid price
• top 5 bid levels
• top 5 ask levels

---

Usage Examples

Basic Parsing

import fastreader

reader = fastreader.ReadMsgFromBinary(
    "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin"
)

msgs = reader.all_messages(5)

for msg in msgs:
    print(msg)

---

Token Based Filtering

import fastreader

reader = fastreader.ReadMsgFromBinary(
    "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin",
    token=1333
)

msgs = reader.all_messages()

---

Sequential Processing

import fastreader

reader = fastreader.ReadMsgFromBinary(
    "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin"
)

while True:

    msg = reader.next_message()

    if msg == "END":
        break

    print(msg)

---

Generate Orderbook

import fastreader

reader = fastreader.ReadMsgFromBinary(
    "/nas/50.30/NSE_CM/Feed_CM_StreamID_2_29_12_2025.bin" , token = 1333
)

reader.orderbook()

---

Orderbook Engine

The orderbook system processes:

• order inserts
• modifications
• cancellations
• trade executions

It continuously maintains:

• best bid
• best ask
• market depth
• mid-price

Example output:

local_ts,
exch_ts,
mid_price,
bid1_price,bid1_qty,
ask1_price,ask1_qty,
...

---

Performance Considerations

Why Rust?

Rust provides:

• memory safety
• no garbage collection pauses
• deterministic performance
• low latency
• high throughput

Parsing Strategy

Current implementation loads all messages into memory.

Advantages:

• fast sequential access
• simpler architecture
• efficient repeated iteration

Tradeoff:

• high memory usage for very large files

---

Error Handling

Errors from binary parsing are converted into Python exceptions.

Example:

.map_err(|e|
    pyo3::exceptions::PyRuntimeError
        ::new_err(e.to_string())
)?;

Python side:

try:
    reader = fastreader.ReadMsgFromBinary(
        "bad.bin"
    )
except RuntimeError as e:
    print(e)

---

Example Output

Order Message

Order Message: SeqNo1, msg_len72,
Msg_Type'O', Exch_ts123456789,
local_ts123456999,
order_id1001,
Token26000,
order_Type'B',
Price25000,
Quantity100,
missed0

Trade Message

Trade Message: SeqNo2, msg_len64,
Msg_Type'T', Exch_ts123456790,
local_ts123457000,
order_id_buy1001,
order_id_sell2002,
Token26000,
Price25010,
Quantity50,
missed0

---

Recommended Improvements

1. Streaming Parser

Avoid loading all messages into memory.

Recommended:

Iterator-based packet parsing

Benefit:

• lower memory usage
• scalable for huge datasets

---

2. Parallel Processing

Current processing is sequential.

Potential upgrade:

• Rayon
• Tokio
• multi-threaded token partitions

---

3. Structured Return Types

Currently methods return formatted strings.

Recommended:

Return structured Python dictionaries:

{
    "seq_no": 1,
    "price": 25000,
    "qty": 100
}

Benefit:

• easier pandas integration
• faster downstream analytics

---

4. CSV / Parquet Export

Add direct exporters.

Example:

reader.to_parquet("data.parquet")

---

5. Async Streaming

Support real-time exchange feeds.

Possible integrations:

• Kafka
• Redis Streams
• WebSockets
• UDP multicast

---

6. Benchmarking

Add Criterion benchmarks.

Measure:

• parsing latency
• throughput
• memory footprint
• orderbook update speed

---

Production Architecture Recommendation

Exchange Feed
      ↓
Binary Capture Layer
      ↓
Rust Decoder Engine
      ↓
Shared Memory Queue
      ↓
Orderbook Engine
      ↓
Strategy Engine
      ↓
Risk Layer
      ↓
Execution Engine

---

Future Enhancements

• zero-copy parsing
• SIMD optimizations
• memory mapped files
• snapshot recovery
• incremental replay
• persistent orderbook state
• exchange gateway integration
• real-time analytics
• Python dataframe support
• Arrow / Polars integration

---

License

MIT License

---

Summary

FastReader is a professional-grade foundation for building:

• HFT infrastructure
• market replay systems
• backtesting engines
• orderbook analytics
• market microstructure research
• execution simulators
• exchange feed handlers

The modular Rust + Python design makes it highly extensible for quantitative trading systems.