orderpulse 0.2.12

High-performance exchange feed parser and orderflow analytics engine with Rust and Python bindings

High-Performance Market Data Processing Engine

Overview

This project is a high-performance Rust-based market data processing engine with Python bindings powered by PyO3. The system is designed to:

• Read binary market feed data
• Parse order and trade messages
• Filter messages by instrument token
• Stream messages sequentially
• Build and maintain an in-memory order book
• Expose Rust performance to Python users

The architecture follows a clean separation of concerns:

Layer	Responsibility
Binary Reader	Reads raw exchange feed data
Message Parser	Converts bytes into typed messages
Message Abstraction	Unified enum-based message handling
Order Book Engine	Maintains bid/ask state
Python Interface	Exposes Rust APIs to Python

---

Architecture

High-Level Architecture

                 +----------------------+
                 |  Binary Feed File    |
                 +----------+-----------+
                            |
                            v
                 +----------------------+
                 | read_messages()      |
                 | Binary Parser        |
                 +----------+-----------+
                            |
                            v
                 +----------------------+
                 | Vec<Message>         |
                 | Message Enum Layer   |
                 +----------+-----------+
                            |
          +----------------+----------------+
          |                                 |
          v                                 v
+----------------------+      +---------------------------+
| ReadMsgFromBinary    |      | OrderBookBuilder          |
| Query/Filtering API  |      | Order Book Construction   |
+----------+-----------+      +-------------+-------------+
           |                                  |
           v                                  v
+----------------------+      +---------------------------+
| MessageBatch         |      | OrderBookManager          |
| Batch Selection API  |      | Matching Engine State     |
+----------------------+      +---------------------------+

---

Core Modules

1. orderbook

Contains:

• OrderBookManager
• Bid/ask maintenance logic
• Top-of-book calculations
• Market depth generation

Responsibilities

• Process incoming order messages
• Update order book state
• Maintain price levels
• Generate top bid/ask snapshots

---

2. read_trd_ord_only

Contains:

• read_messages()

Responsibilities

• Read binary exchange feed files
• Decode binary structures
• Produce typed messages

---

3. structure

Contains:

• Message enum
• Packet structures
• Exchange packet layouts

Responsibilities

Defines the low-level message schema.

Example:

pub enum Message {
    Order(OrderPacket),
    Trade(TradePacket),
}

This abstraction allows the engine to process heterogeneous messages uniformly.

---

4. orderbook_processing

Contains advanced order book processing utilities.

Likely responsibilities:

• Aggregation
• Snapshot generation
• Derived analytics
• Depth calculations

---

5. tsc

Most likely contains:

• Timestamp counter utilities
• Performance timers
• Latency measurement helpers

Useful for:

• HFT systems
• Benchmarking
• Exchange latency analysis

---

Main Components

1. format_message()

fn format_message(msg: &Message) -> String

Purpose

Converts internal message structures into human-readable strings.

This function acts as:

• A debugging utility
• A logging formatter
• A Python-readable serialization layer

---

Flow

Message Enum
     |
     +--> Order Message
     |
     +--> Trade Message

---

Order Message Formatting

Extracts:

• Sequence number
• Message length
• Message type
• Exchange timestamp
• Local timestamp
• Order ID
• Token
• Price
• Quantity
• Missed packet flag

Example

let output = format_message(&msg);
println!("{}", output);

Sample Output

Order Message: SeqNo12345, msg_len64, Msg_Type'B', Exch_ts123456789,
local_ts123456790, order_id99999, Token26000,
order_Type'B', Price24500, Quantity50, missed0

---

Trade Message Formatting

Extracts:

• Buy order ID
• Sell order ID
• Trade price
• Trade quantity

Example

let output = format_message(&trade_msg);

---

2. MessageBatch

#[pyclass]
pub struct MessageBatch {
    messages: Vec<Message>,
}

Purpose

Acts as a lightweight batch container for filtered or selected messages.

This is a powerful abstraction because it:

• Avoids copying entire datasets repeatedly
• Enables chainable workflows
• Supports Python interoperability
• Allows selective order book construction

---

Methods

len()

fn len(&self) -> usize

Returns total messages in batch.

Example

batch.len()

Output

1200

---

is_empty()

fn is_empty(&self) -> bool

Checks whether the batch contains messages.

Example

batch.is_empty()

Output

False

---

to_list(limit=None)

fn to_list(&self, limit: Option<usize>) -> Vec<String>

Converts messages into formatted strings.

Example

batch.to_list(5)

Output

[
  "Order Message: ...",
  "Trade Message: ..."
]

---

3. ReadMsgFromBinary

#[pyclass]
pub struct ReadMsgFromBinary {
    messages: Vec<Message>,
    current_index: usize,
}

Purpose

Primary interface for:

• Loading exchange binary feeds
• Filtering by token
• Iterating messages
• Performing analytics
• Feeding order book engine

This is effectively the:

Core ingestion engine

---

Constructor

new(path, token=None)

fn new(path: String, token: Option<u32>) -> PyResult<Self>

Responsibilities

1. Read binary file
2. Parse messages
3. Optionally filter by token
4. Store parsed messages

---

Example

Python Usage

reader = ReadMsgFromBinary(
    path="market_feed.bin",
    token=26000
)

---

Internal Flow

Binary File
    |
    v
read_messages()
    |
    v
Vec<Message>
    |
    +--> Optional Token Filter
    |
    v
ReadMsgFromBinary

---

Analytics Functions

total_messages()

fn total_messages(&self) -> usize

Returns total parsed messages.

Example

reader.total_messages()

---

total_orders()

fn total_orders(&self) -> usize

Counts only order messages.

Example

reader.total_orders()

---

total_trades()

fn total_trades(&self) -> usize

Counts only trade messages.

Example

reader.total_trades()

---

summary()

fn summary(&self)

Prints complete dataset statistics.

Example

reader.summary()

Output

Total Messages: 500000
Total Orders: 420000
Total Trades: 80000

---

Cursor-Based Streaming

The engine supports sequential message streaming.

This design is critical for:

• Large datasets
• Replay systems
• Event-driven processing
• Memory-efficient workflows

---

reset_cursor()

fn reset_cursor(&mut self)

Resets stream position.

Example

reader.reset_cursor()

---

get_next_msg()

fn get_next_msg(&mut self) -> String

Returns next message in sequence.

Example

msg = reader.get_next_msg()

Output

"Order Message: ..."

---

Message Selection APIs

get_all_messages(limit=None)

Returns all messages.

Example

reader.get_all_messages(10)

---

get_order_messages(limit=None)

Returns only order messages.

Example

reader.get_order_messages(20)

---

get_trade_messages(limit=None)

Returns only trade messages.

Example

reader.get_trade_messages(20)

---

Batch Selection APIs

These methods return MessageBatch objects.

---

select_all_messages()

fn select_all_messages(&self) -> MessageBatch

Example

batch = reader.select_all_messages()

---

select_order_messages()

Returns only order messages.

Example

orders = reader.select_order_messages()

---

select_trade_messages()

Returns only trade messages.

Example

trades = reader.select_trade_messages()

---

select_next_messages(limit)

fn select_next_messages(&mut self, limit: usize) -> MessageBatch

Returns the next N messages from stream.

Example

chunk = reader.select_next_messages(1000)

This is ideal for:

• Chunk processing
• Streaming pipelines
• Backtesting engines

---

4. OrderBookBuilder

#[pyclass]
pub struct OrderBookBuilder;

Purpose

Constructs order books from:

• Entire datasets
• Selected batches
• Streaming chunks

This is the bridge between:

Message Layer ---> Matching Engine

---

Constructor

new()

fn new() -> Self

Example

ob = OrderBookBuilder()

---

Order Book Creation

create_orderbook_all_messages()

fn create_orderbook_all_messages(&self, reader: PyRef<ReadMsgFromBinary>) -> usize

Processes all messages.

Example

builder.create_orderbook_all_messages(reader)

---

create_orderbook()

fn create_orderbook(&self, batch: PyRef<MessageBatch>) -> usize

Processes only selected messages.

Example

batch = reader.select_order_messages()
builder.create_orderbook(batch)

---

Internal Processing Pipeline

Message
   |
   v
OrderBookManager::process_order_message()
   |
   v
Update Bid/Ask Levels
   |
   v
Generate Top Levels
   |
   v
Emit Snapshot

---

build_from_messages()

fn build_from_messages(messages: &[Message]) -> usize

Responsibilities

• Initialize order book manager
• Process every message
• Update order book state
• Generate top levels
• Track processed rows

---

Performance Characteristics

Why Rust?

This system is architected for:

• Low latency
• Deterministic memory management
• High throughput
• Zero-cost abstractions
• Safe concurrency

---

Why PyO3?

PyO3 enables:

• Python interoperability
• Quant workflow integration
• Data science compatibility
• Jupyter support

This gives:

Rust Speed + Python Flexibility

---

Design Patterns Used

1. Enum-Based Message Dispatch

match msg {
    Message::Order(..) => {}
    Message::Trade(..) => {}
}

Benefits:

• Type safety
• Fast dispatch
• Cleaner branching

---

2. Batch Processing Pattern

MessageBatch avoids unnecessary re-parsing.

Benefits:

• Reduced memory allocations
• Faster analytics
• Reusable datasets

---

3. Streaming Iterator Pattern

current_index acts as an internal cursor.

Benefits:

• Efficient sequential access
• Large file support
• Replay compatibility

---

4. Separation of Concerns

Each module has a single responsibility.

Benefits:

• Easier testing
• Better maintainability
• Cleaner scaling

---

Example End-to-End Workflow

Python Example

from your_module import ReadMsgFromBinary, OrderBookBuilder

# Load binary feed
reader = ReadMsgFromBinary(
    path="market_feed.bin",
    token=26000
)

# Dataset statistics
reader.summary()

# Select only orders
orders = reader.select_order_messages()

# Build order book
builder = OrderBookBuilder()

builder.create_orderbook(orders)