orderpulse 0.2.24

High-performance exchange feed parser and orderflow analytics engine with Rust and Python bindings

fastReader Library Guide

fastReader is a high-performance Python extension (implemented in Rust via PyO3) for reading binary order/trade feed files and building an in-memory order book.

This README is generated from the current API in src/lib.rs and explains:

• how users should import and use the library,
• what each public function does,
• expected output shape/examples,
• and architecture with a practical workflow.

Correct Python Import

The module name is defined by this function in src/lib.rs:

#[pymodule]
fn fastReader(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {

So in Python, import like this:

import fastReader as fr

You can use any alias (fr, reader, etc.), but fastReader must match the module name.

Architecture (How Data Flows)

Binary feed file (.bin)
   |
   v
read_trd_ord_only(path)
   |- mmap file
   |- detect msg type (N / M / X / T)
   |- parse packed structs (OrderPacket / TradePacket)
   `- return Vec<Message>
        |
        +--> MessageCacheReader (full cache in memory)
        |
        `--> StreamingBinaryLoader (cursor-based sequential consumption)
                        |
                        v
                 OrderbookBuilder
                        |
                        |- apply_filter(...) [optional]
                        |- build_from_list(...) or build_from_source(...)
                        `- get_snapshot(token, levels)

Internal Components

• MessageCacheReader
  • Stores parsed messages in Arc<Vec<Message>> for shared read access.
• StreamingBinaryLoader
  • Holds parsed messages + an index pointer (index) for sequential playback.
• OrderbookBuilder
  • Owns OrderBookManager and replays messages into order-book state.

Message Types

• N: New order
• M: Modify order
• X: Cancel order
• T: Trade

Quick Start

import fastReader as fr

file_path = "/data/feed.bin"

# 1) Load and cache
reader = fr.MessageCacheReader()
loaded = reader.load_to_cache(file_path)
print("Loaded:", loaded)

# 2) Build order book from cached data
builder = fr.OrderbookBuilder()
applied = builder.build_from_list(reader)
print("Applied:", applied)

# 3) Snapshot top levels
snap = builder.get_snapshot(token=1660, levels=3)
print(snap)

Expected output example:

Loaded: 4231887
Applied: 4231887
{'token': 1660, 'found': True, 'mid_price': 234550, 'best_bid': (234500, 1500), 'best_ask': (234600, 800), 'spread': 100, 'bids': [(234500, 1500), (234400, 3200), (234300, 700)], 'asks': [(234600, 800), (234700, 2100), (234800, 4500)]}

API Reference (Every Public Function)

1) MessageCacheReader

MessageCacheReader()

Creates an empty cache reader.

reader = fr.MessageCacheReader()

Expected output: no direct output.

load_to_cache(file_path: str) -> int

Parses the binary file and stores all messages in cache.

count = reader.load_to_cache("/data/feed.bin")
print(count)

Expected output example:

4231887

Errors:

• Raises RuntimeError (from Rust I/O error) if file cannot be opened/read.

get_all_messages() -> list[str]

Returns formatted string representation of every cached message.

msgs = reader.get_all_messages()
print(msgs[0])
print(msgs[1])

Expected output example:

Order Message: SeqNo 42, MsgLen 10, MsgType 'N', ExchTs 100000, LocalTs 200000, OrderId 55, Token 1001, Side 'B', Price 500, Quantity 100, Missed 0
Trade Message: SeqNo 99, MsgLen 10, MsgType 'T', ExchTs 300000, LocalTs 400000, BuyOrderId 101, SellOrderId 202, Token 1001, Price 505, Quantity 20, Missed 0

get_cache_summary() -> dict

Returns summary statistics.

summary = reader.get_cache_summary()
print(summary)

Expected output shape:

{
  'file_source': '/data/feed.bin',
  'total_messages': 4231887,
  'total_orders': 3854321,
  'total_trades': 377566,
  'memory_usage_bytes': 226411296
}

Field meaning:

• file_source: last loaded file path or None
• total_messages: total cached messages
• total_orders: count of order messages (N/M/X)
• total_trades: count of trade messages (T)
• memory_usage_bytes: approximate cache memory as len(messages) * sizeof(Message)

2) StreamingBinaryLoader

StreamingBinaryLoader()

Creates an empty stream loader.

loader = fr.StreamingBinaryLoader()

Expected output: no direct output.

open_stream(file_path: str) -> int

Loads messages and resets stream cursor to start.

total = loader.open_stream("/data/feed.bin")
print(total)

Expected output example:

4231887

Notes:

• After open_stream, data is consumed through OrderbookBuilder.build_from_source(loader, limit=...).
• Internal cursor advances as messages are consumed.

3) OrderbookBuilder

OrderbookBuilder()

Creates a fresh order-book builder.

builder = fr.OrderbookBuilder()

Expected output: no direct output.

apply_filter(logic_criteria: list[str] | None = None) -> None

Sets optional message-type filter.

builder.apply_filter(["N", "M", "X"])  # process only order lifecycle
builder.apply_filter(["T"])            # process only trades
builder.apply_filter(None)             # clear filter (default all)

How it works:

• The first character of each string is used.
• Valid practical values: N, M, X, T.

Expected behavior:

• None: all messages processed
• ['N','M','X']: trade messages skipped
• ['T']: only trades processed

build_from_list(reader: MessageCacheReader) -> int

Replays cached messages into order book.

applied = builder.build_from_list(reader)
print(applied)

Expected output example:

4231887

If filter is set, return value is number of messages actually applied (after filtering).

build_from_source(source, limit: int | None = None) -> int

Accepts either:

• MessageCacheReader
• StreamingBinaryLoader

Examples:

# Source: cache reader
n1 = builder.build_from_source(reader)
print(n1)

# Source: streaming loader, process only first 500k messages
n2 = builder.build_from_source(loader, limit=500000)
print(n2)

Expected output example:

4231887
500000

Error case:

• Raises TypeError if source is neither MessageCacheReader nor StreamingBinaryLoader.

get_snapshot(token: int, levels: int = 5) -> dict

Returns top-of-book snapshot for one token.

snap = builder.get_snapshot(token=1660, levels=3)
print(snap)

Expected output when token exists:

{
  'token': 1660,
  'found': True,
  'mid_price': 234550,
  'best_bid': (234500, 1500),
  'best_ask': (234600, 800),
  'spread': 100,
  'bids': [(234500, 1500), (234400, 3200), (234300, 700)],
  'asks': [(234600, 800), (234700, 2100), (234800, 4500)]
}

Expected output when token does not exist:

{
  'token': 9999999,
  'found': False,
  'mid_price': 0,
  'best_bid': None,
  'best_ask': None,
  'spread': None,
  'bids': [],
  'asks': []
}

Field explanation:

• found: whether token book exists
• mid_price: midpoint based on best bid/ask
• best_bid / best_ask: (price, qty) tuples
• spread: best_ask_price - best_bid_price
• bids / asks: top N levels

Practical End-to-End Scenarios

A) Full Replay (All Messages)

import fastReader as fr

reader = fr.MessageCacheReader()
reader.load_to_cache("/data/feed.bin")

builder = fr.OrderbookBuilder()
builder.apply_filter(None)
builder.build_from_list(reader)

print(builder.get_snapshot(1660, 5))

B) Replay Only First N Stream Messages

import fastReader as fr

loader = fr.StreamingBinaryLoader()
loader.open_stream("/data/feed.bin")

builder = fr.OrderbookBuilder()
processed = builder.build_from_source(loader, limit=1_000_000)
print("Processed:", processed)
print(builder.get_snapshot(1660, 3))

C) Order-Only Book (Ignore Trades)

import fastReader as fr

reader = fr.MessageCacheReader()
reader.load_to_cache("/data/feed.bin")

builder = fr.OrderbookBuilder()
builder.apply_filter(["N", "M", "X"])
builder.build_from_list(reader)

print(builder.get_snapshot(1660, 5))

Performance and Usage Notes

• get_all_messages() can be expensive on huge files because it builds Python strings for every message.
• Use get_cache_summary() for fast sanity checks.
• build_from_source(loader, limit=...) is useful for point-in-time reconstruction.
• OrderbookBuilder keeps state across calls; create a new instance for a clean replay.

Troubleshooting

Import error for module name

If import rmoney_orderbook fails, that is expected after renaming module to fastReader. Use:

import fastReader as fr

Type error in build_from_source

Pass only MessageCacheReader or StreamingBinaryLoader instances.

Empty snapshot

If found is False, confirm:

• token exists in your feed,
• replay consumed relevant messages,
• filter did not exclude needed message types.