disco-data-logger 0.1.2

Data logger for Disco simulations

🧾 disco-data-logger

High-performance, C++/NumPy-backed data logger
for Disco discrete-event and Monte Carlo simulation programs.

---

Overview

disco-data-logger provides a fast, compressed, and lightweight data recording layer
for large-scale Disco simulations and other computational experiments.

It is optimized for capturing sparse numerical state updates and accumulators during simulation runs,
and writing them efficiently to disk as Zstandard-compressed segment files.
Each simulation entity or measurement can log its data independently through labeled streams.

It combines:

• A C++/pybind11 core for high-throughput buffering and compression.
• Python API for easy stream registration and control.
• Built-in Parquet export for analysis and aggregation after runs.

---

✨ Features

• Sparse vector logging powered by graphblas.Vector.
• Fixed-point quantization for compact and deterministic encoding.
• Buffered, lock-free write path (ring buffer + writer thread).
• Zstandard compression (vendored, no external dependencies).
• Segment rotation for large simulation outputs.
• JSON metadata for each stream (organisation, model, experiment, …).
• Periodic vector streams that emit state snapshots or accumulator sums once per period.
• Integrated Parquet export for post-run analytics.
• Arrow-based collector to filter finished loggers and emit RecordBatches directly.
• MIT-licensed and designed for in-cluster (on-disk/in-memory) use.

---

🚀 Installation

pip install disco-data-logger

pyarrow ships with the package, so Parquet export works out of the box.

---

📚 Documentation

• Collector – decode completed loggers, filter streams with label_selector, and write Arrow RecordBatch outputs efficiently.
• Periodic vector stream logging – step-by-step guide for configuring periodicity, choosing between state and accumulator modes, and verifying the emitted sparse data.
• ENGINEERING_SPEC.md – project history, motivation, and architectural overview.

---

🛠️ Development

Set up a virtual environment (for example, python -m venv .venv && source .venv/bin/activate), then install the project in editable mode with its development extras:

pip install -e '.[dev]'

[!NOTE] The tools namespace is provided by the separate disco-tools dependency. After installing it for the first time you must deactivate and reactivate your environment (e.g., deactivate followed by source .venv/bin/activate) so that pytest can discover the package properly.

Once the environment is ready, run the unit tests with:

pytest