orderpulse 0.2.25

High-performance exchange feed parser and orderflow analytics engine with Rust and Python bindings

OrderPulse / rmoney-orderbook

rmoney-orderbook is a Rust-powered Python extension module for reading exchange binary order/trade messages and building an orderbook snapshot from them.

The library is written in Rust for speed and exposed to Python using PyO3. Python users can load raw binary market-data files, process order/trade packets, apply filters, and get bid/ask snapshots as normal Python dictionaries.

---

Purpose

This package is designed for high-performance market-data processing.

It helps convert raw exchange binary files into useful orderbook information.

Raw binary file
        ↓
Rust binary parser
        ↓
Order / Trade messages
        ↓
OrderbookBuilder
        ↓
OrderBookManager
        ↓
Python snapshot output

The Python user does not need to manually decode the binary file.

---

Main Features

• Read order/trade binary files.
• Cache all parsed messages in memory.
• Process messages into an orderbook.
• Apply message-type filters.
• Build orderbook from cached messages or stream-like loader.
• Return top bid/ask levels.
• Return best bid, best ask, spread, and mid price.
• Expose Rust performance through a clean Python API.

---

Installation

After publishing to PyPI:

pip install rmoney-orderbook

For local development using maturin:

maturin develop --release

Build wheel:

maturin build --release
pip install target/wheels/*.whl

---

Python Import

from rmoney_orderbook import MessageCacheReader, StreamingBinaryLoader, OrderbookBuilder

---

Architecture Overview

The library contains three Python-facing classes:

MessageCacheReader
StreamingBinaryLoader
OrderbookBuilder

Internally, the Rust project is organized like this:

mod orderbook;
mod orderbook_processing;
mod read_trd_ord_only;
mod structure;
mod tsc;

Internal Module Purpose

Module	Purpose
structure	Defines binary packet structures such as Message, OrderPacket, TradePacket, and PeekStructure.
read_trd_ord_only	Reads binary order/trade files and converts them into Rust Message values.
orderbook	Contains the orderbook engine such as OrderBookManager, PriceLevel, and Side.
orderbook_processing	Contains helper processing logic.
tsc	Contains timing/cycle utility logic.
lib.rs	Exposes Rust classes and methods to Python using PyO3.

---

Class 1: MessageCacheReader

Purpose

MessageCacheReader is the RAM-based reader.

It loads the full binary file into memory and stores all parsed messages.

This is useful when:

• the file fits in RAM,
• you want to reuse the same data multiple times,
• you want fast repeated backtesting,
• you want to inspect parsed messages,
• you want summary information about the loaded file.

---

Internal Rust Structure

#[pyclass]
pub struct MessageCacheReader {
    file_path: Option<String>,
    messages: Arc<Vec<Message>>,
}

Field Explanation

Field	Type	Purpose
file_path	Option<String>	Stores the source file path after loading.
messages	Arc<Vec<Message>>	Stores all parsed messages in memory.

Arc<Vec<Message>> is used so the vector can be safely shared without copying the full message list.

---

Constructor

reader = MessageCacheReader()

Purpose

Creates an empty cache reader.

Initial State

file_path = None
messages = []

Expected Output

<MessageCacheReader object>

---

load_to_cache(file_path)

Purpose

Loads the full binary file into memory.

Python Example

from rmoney_orderbook import MessageCacheReader

reader = MessageCacheReader()

count = reader.load_to_cache("/data/market_feed.bin")

print(count)

Expected Output

1250000

This means 1,250,000 messages were loaded successfully.

---

What Happens Internally?

Python passes file path
        ↓
Rust calls binary reader
        ↓
Binary packets are converted into Message values
        ↓
Messages are stored in memory
        ↓
Message count is returned to Python

---

get_all_messages()

Purpose

Returns all cached messages as readable strings.

This is mainly useful for debugging and validating whether the binary file is being decoded correctly.

Python Example

reader = MessageCacheReader()
reader.load_to_cache("/data/market_feed.bin")

messages = reader.get_all_messages()

print(messages[0])
print(messages[1])

Expected Output

Order Message: SeqNo 1001, MsgLen 38, MsgType 'N', ExchTs 1718000000000000000, LocalTs 1718000000000100000, OrderId 987654321, Token 26000, Side 'B', Price 2255000, Quantity 75, Missed 0
Trade Message: SeqNo 1002, MsgLen 45, MsgType 'T', ExchTs 1718000000000200000, LocalTs 1718000000000300000, BuyOrderId 987654321, SellOrderId 987654322, Token 26000, Price 2255050, Quantity 25, Missed 0

---

get_cache_summary()

Purpose

Returns metadata about the loaded file.

Python Example

reader = MessageCacheReader()
reader.load_to_cache("/data/market_feed.bin")

summary = reader.get_cache_summary()

print(summary)

Expected Output

{
    "file_source": "/data/market_feed.bin",
    "total_messages": 1250000,
    "total_orders": 1000000,
    "total_trades": 250000,
    "memory_usage_bytes": 90000000
}

Output Explanation

Key	Meaning
file_source	File path loaded into cache.
total_messages	Total number of parsed messages.
total_orders	Number of order messages.
total_trades	Number of trade messages.
memory_usage_bytes	Approximate memory used by parsed messages.

---

Class 2: StreamingBinaryLoader

Purpose

StreamingBinaryLoader is a stream-style loader.

It is designed to process messages sequentially instead of exposing the full message list to Python.

In the current implementation, it stores messages internally and advances using an index. From the Python user’s perspective, it behaves like a source that can be consumed by OrderbookBuilder.

---

Internal Rust Structure

#[pyclass]
pub struct StreamingBinaryLoader {
    file_path: Option<String>,
    messages: Vec<Message>,
    index: usize,
}

Field Explanation

Field	Type	Purpose
file_path	Option<String>	Stores opened file path.
messages	Vec<Message>	Stores parsed messages internally.
index	usize	Tracks current read position.

---

Constructor

loader = StreamingBinaryLoader()

Purpose

Creates an empty streaming loader.

Initial State

file_path = None
messages = []
index = 0

---

open_stream(file_path)

Purpose

Opens and prepares the file for stream-style processing.

Python Example

from rmoney_orderbook import StreamingBinaryLoader

loader = StreamingBinaryLoader()

count = loader.open_stream("/data/market_feed.bin")

print(count)

Expected Output

1250000

This means the file was opened and messages are ready for sequential processing.

---

Internal Logic

File path is received
        ↓
Binary reader parses messages
        ↓
Messages are stored inside loader
        ↓
index is reset to 0
        ↓
Message count is returned

---

Internal get_next_message()

This method is used internally by OrderbookBuilder.

It is not exposed directly to Python.

Purpose

Returns the next message and moves the internal pointer forward.

Logic

Read message at current index
        ↓
Increment index
        ↓
Return message

If no messages are left, it returns None.

---

Class 3: OrderbookBuilder

Purpose

OrderbookBuilder is the main engine exposed to Python.

It consumes parsed messages from MessageCacheReader or StreamingBinaryLoader and updates the internal orderbook state.

---

Internal Rust Structure

#[pyclass]
pub struct OrderbookBuilder {
    manager: OrderBookManager,
    allowed_message_types: Option<Vec<u8>>,
}

Field Explanation

Field	Type	Purpose
manager	OrderBookManager	Internal bid/ask orderbook engine.
allowed_message_types	Option<Vec<u8>>	Optional message-type filter.

---

Constructor

builder = OrderbookBuilder()

Purpose

Creates a new empty orderbook builder.

Initial State

manager = empty OrderBookManager
allowed_message_types = None

When allowed_message_types is None, all messages are processed.

---

apply_filter(logic_criteria=None)

Purpose

Filters which message types should be processed.

For example, if you only want to process add/modify/delete order messages, you can pass only those message type codes.

Python Example

builder = OrderbookBuilder()

builder.apply_filter(["N", "M", "X"])

Meaning

Only process message types:
N
M
X

Messages with other types will be skipped.

---

Why Strings Are Converted to Bytes

Python users pass strings:

["N", "M", "X"]

But binary packets store message type as bytes:

b'N'
b'M'
b'X'

So internally, Rust converts:

"N" -> b'N'
"M" -> b'M'
"X" -> b'X'

---

If No Filter Is Applied

builder = OrderbookBuilder()

All messages are processed.

Equivalent internal state:

allowed_message_types = None

---

build_from_list(reader)

Purpose

Builds the orderbook from a MessageCacheReader.

This is the RAM-based workflow.

Python Example

from rmoney_orderbook import MessageCacheReader, OrderbookBuilder

reader = MessageCacheReader()
reader.load_to_cache("/data/market_feed.bin")

builder = OrderbookBuilder()

processed = builder.build_from_list(reader)

print(processed)

Expected Output

1250000

This means 1,250,000 messages were processed.

---

Internal Flow

Loop through cached messages
        ↓
Check if message is Order or Trade
        ↓
Apply filter if filter exists
        ↓
Call process_order_message() or process_trade_message()
        ↓
Update orderbook state
        ↓
Return processed count

---

build_from_source(source, limit=None)

Purpose

Builds the orderbook from either:

MessageCacheReader

or:

StreamingBinaryLoader

This method is flexible because Python can pass either source type.

---

Example with MessageCacheReader

reader = MessageCacheReader()
reader.load_to_cache("/data/market_feed.bin")

builder = OrderbookBuilder()

processed = builder.build_from_source(reader)

print(processed)

Expected Output

1250000

---

Example with StreamingBinaryLoader

loader = StreamingBinaryLoader()
loader.open_stream("/data/market_feed.bin")

builder = OrderbookBuilder()

processed = builder.build_from_source(loader, 100000)

print(processed)

Expected Output

100000

This means only the first 100,000 messages were processed because a limit was provided.

---

Why limit Is Useful

limit is useful for:

• debugging,
• testing,
• partial processing,
• avoiding long runs on huge files,
• checking first few messages before full processing.

Example:

builder.build_from_source(loader, 5000)

This processes only 5,000 messages.

---

Wrong Source Type Error

If a wrong object is passed:

builder.build_from_source("wrong")

Expected error:

TypeError: build_from_source expects MessageCacheReader or StreamingBinaryLoader

---

get_snapshot(token, levels=None)

Purpose

Returns the current orderbook snapshot for a specific token.

Python Example

snapshot = builder.get_snapshot(26000, 5)

print(snapshot)

Expected Output

{
    "token": 26000,
    "found": True,
    "mid_price": 2255025,
    "best_bid": (2255000, 150),
    "best_ask": (2255050, 200),
    "spread": 50,
    "bids": [
        (2255000, 150),
        (2254950, 100),
        (2254900, 75),
        (2254850, 50),
        (2254800, 25)
    ],
    "asks": [
        (2255050, 200),
        (2255100, 125),
        (2255150, 80),
        (2255200, 40),
        (2255250, 20)
    ]
}

---

Output Explanation

Key	Meaning
token	Instrument token requested.
found	Whether orderbook data exists for this token.
mid_price	Mid price returned by the orderbook manager.
best_bid	Top bid level as (price, quantity).
best_ask	Top ask level as (price, quantity).
spread	best_ask_price - best_bid_price.
bids	Top bid levels.
asks	Top ask levels.

---

If Token Is Not Found

snapshot = builder.get_snapshot(99999, 5)

print(snapshot)

Expected Output

{
    "token": 99999,
    "found": False,
    "mid_price": 0,
    "best_bid": None,
    "best_ask": None,
    "spread": None,
    "bids": [],
    "asks": []
}

---

Complete Workflow 1: Cache-Based Processing

Use this when the file can fit in RAM.

from rmoney_orderbook import MessageCacheReader, OrderbookBuilder

FILE_PATH = "/data/market_feed.bin"
TOKEN = 26000

reader = MessageCacheReader()

loaded_count = reader.load_to_cache(FILE_PATH)

print("Loaded messages:", loaded_count)

summary = reader.get_cache_summary()

print("Cache summary:", summary)

builder = OrderbookBuilder()

processed_count = builder.build_from_list(reader)

print("Processed messages:", processed_count)

snapshot = builder.get_snapshot(TOKEN, 5)

print("Snapshot:", snapshot)

Expected Output

Loaded messages: 1250000
Cache summary: {'file_source': '/data/market_feed.bin', 'total_messages': 1250000, 'total_orders': 1000000, 'total_trades': 250000, 'memory_usage_bytes': 90000000}
Processed messages: 1250000
Snapshot: {'token': 26000, 'found': True, 'mid_price': 2255025, 'best_bid': (2255000, 150), 'best_ask': (2255050, 200), 'spread': 50, 'bids': [(2255000, 150), (2254950, 100), (2254900, 75), (2254850, 50), (2254800, 25)], 'asks': [(2255050, 200), (2255100, 125), (2255150, 80), (2255200, 40), (2255250, 20)]}

---

Complete Workflow 2: Filtered Processing

Use this when you want to process only selected message types.

from rmoney_orderbook import MessageCacheReader, OrderbookBuilder

reader = MessageCacheReader()
reader.load_to_cache("/data/market_feed.bin")

builder = OrderbookBuilder()

builder.apply_filter(["N", "M", "X"])

processed = builder.build_from_list(reader)

print("Processed filtered messages:", processed)

snapshot = builder.get_snapshot(26000, 5)

print(snapshot)

Expected Output

Processed filtered messages: 1000000
{'token': 26000, 'found': True, 'mid_price': 2255025, 'best_bid': (2255000, 150), 'best_ask': (2255050, 200), 'spread': 50, 'bids': [(2255000, 150), (2254950, 100), (2254900, 75), (2254850, 50), (2254800, 25)], 'asks': [(2255050, 200), (2255100, 125), (2255150, 80), (2255200, 40), (2255250, 20)]}

---

Complete Workflow 3: Stream-Style Processing with Limit

Use this when you want to process only a limited number of messages.

from rmoney_orderbook import StreamingBinaryLoader, OrderbookBuilder

loader = StreamingBinaryLoader()

loaded = loader.open_stream("/data/huge_market_feed.bin")

print("Loaded into stream source:", loaded)

builder = OrderbookBuilder()

processed = builder.build_from_source(loader, 100000)

print("Processed:", processed)

snapshot = builder.get_snapshot(26000, 5)

print(snapshot)

Expected Output

Loaded into stream source: 5000000
Processed: 100000
{'token': 26000, 'found': True, 'mid_price': 2255025, 'best_bid': (2255000, 150), 'best_ask': (2255050, 200), 'spread': 50, 'bids': [(2255000, 150), (2254950, 100), (2254900, 75), (2254850, 50), (2254800, 25)], 'asks': [(2255050, 200), (2255100, 125), (2255150, 80), (2255200, 40), (2255250, 20)]}