disco-data-logger 0.1.4

Data logger for Disco simulations

🧾 disco-data-logger

High-performance, C++/NumPy-backed data logger
for Disco discrete-event and Monte Carlo simulation programs.

---

Overview

disco-data-logger provides a fast, compressed, and lightweight data recording layer
for large-scale Disco simulations and other computational experiments.

It is optimized for capturing sparse numerical state updates and accumulators during simulation runs,
and writing them efficiently to disk as Zstandard-compressed segment files.
Each simulation entity or measurement can log its data independently through labeled streams.

It combines:

• A C++/pybind11 core for high-throughput buffering and compression.
• Python API for easy stream registration and control.
• Built-in Parquet export for analysis and aggregation after runs.

---

✨ Features

• Sparse vector logging powered by graphblas.Vector.
• Fixed-point quantization for compact and deterministic encoding.
• Buffered, lock-free write path (ring buffer + writer thread).
• Zstandard compression (vendored, no external dependencies).
• Segment rotation for large simulation outputs.
• JSON metadata for each stream (organisation, model, experiment, …).
• Periodic vector streams that emit state snapshots or accumulator sums once per period.
• Integrated Parquet export for post-run analytics.
• Arrow-based collector to filter finished loggers and emit RecordBatches directly.
• MIT-licensed and designed for in-cluster (on-disk/in-memory) use.

---

🚀 Installation

pip install disco-data-logger

pyarrow ships with the package, so Parquet export works out of the box.

---

📚 Documentation

• Collector – decode completed loggers, filter streams with label_selector, and write Arrow RecordBatch outputs efficiently.
• Periodic vector stream logging – step-by-step guide for configuring periodicity, choosing between state and accumulator modes, and verifying the emitted sparse data.
• ENGINEERING_SPEC.md – project history, motivation, and architectural overview.

---

🛠️ Development

Set up a virtual environment (for example, python -m venv .venv && source .venv/bin/activate), then install the project in editable mode with its development extras:

pip install -e '.[dev]'

[!NOTE] The tools namespace is provided by the separate disco-tools dependency. After installing it for the first time you must deactivate and reactivate your environment (e.g., deactivate followed by source .venv/bin/activate) so that pytest can discover the package properly.

Once the environment is ready, run the unit tests with:

pytest

---

License

This project is licensed under the Apache License, Version 2.0.

You are free to use, modify, and distribute this software, including for commercial purposes, subject to the terms of the license.

Contributions

By contributing to this project, you agree that your contributions will be licensed under the Apache License 2.0 and that you have the right to submit the work under those terms.

This project uses a Contributor License Agreement (CLA) to ensure clarity of intellectual property and patent rights.