🧾 disco-data-logger
High-performance, C++/NumPy-backed data logger
for Disco discrete-event and Monte Carlo simulation programs.
Overview
disco-data-logger provides a fast, compressed, and lightweight data recording layer
for large-scale Disco simulations and other computational experiments.
It is optimized for capturing sparse numerical state updates and accumulators during simulation runs,
and writing them efficiently to disk as Zstandard-compressed segment files.
Each simulation entity or measurement can log its data independently through labeled streams.
It combines:
- A C++/pybind11 core for high-throughput buffering and compression.
- Python API for easy stream registration and control.
- Built-in Parquet export for analysis and aggregation after runs.
✨ Features
- Sparse vector logging powered by
graphblas.Vector. - Fixed-point quantization for compact and deterministic encoding.
- Buffered, lock-free write path (ring buffer + writer thread).
- Zstandard compression (vendored, no external dependencies).
- Segment rotation for large simulation outputs.
- JSON metadata for each stream (
organisation,model,experiment, …). - Periodic vector streams that emit state snapshots or accumulator sums once per period.
- Integrated Parquet export for post-run analytics.
- Arrow-based collector to filter finished loggers and emit RecordBatches directly.
- MIT-licensed and designed for in-cluster (on-disk/in-memory) use.
🚀 Installation
pip install disco-data-logger
pyarrow ships with the package, so Parquet export works out of the box.
📚 Documentation
- Collector – decode completed loggers, filter streams with
label_selector, and write ArrowRecordBatchoutputs efficiently. - Periodic vector stream logging – step-by-step guide for
configuring
periodicity, choosing betweenstateandaccumulatormodes, and verifying the emitted sparse data. - ENGINEERING_SPEC.md – project history, motivation, and architectural overview.
🛠️ Development
Set up a virtual environment (for example, python -m venv .venv && source .venv/bin/activate),
then install the project in editable mode with its development extras:
pip install -e '.[dev]'
Once the environment is ready, run the unit tests with:
pytest
License
This project is licensed under the Apache License, Version 2.0.
You are free to use, modify, and distribute this software, including for commercial purposes, subject to the terms of the license.
Contributions
By contributing to this project, you agree that your contributions will be licensed under the Apache License 2.0 and that you have the right to submit the work under those terms.
This project uses a Contributor License Agreement (CLA) to ensure clarity of intellectual property and patent rights.
1 maintainer
Metadata
1 maintainer
Release files for disco-data-logger 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| disco_data_logger-0.1.5.tar.gz | 32.3 kB | With provenance | Details for disco_data_logger-0.1.5.tar.gz |
Built distributions (wheels)
Total release size: 11.1 MB
Release files / disco_data_logger-0.1.5.tar.gz
| Download URL | disco_data_logger-0.1.5.tar.gz |
|---|---|
| Size | 32.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1cad6ef51b91e47fbf4bfecd95a9eb02489222294f326b1dd63a932dfe33588b
|
|
BLAKE2b-256 checksum How to use checksums |
1f1260564e893f5cd22dae7c29c51ba0a370336489c150789580df519dc30c9b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp313-cp313-win_amd64.whl
| Download URL | disco_data_logger-0.1.5-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 864.8 kB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
d7d59e0e850ab082f94e21d7c1018c7c22719ac769bbcfa1c7f35c2e30ac53c6
|
|
BLAKE2b-256 checksum How to use checksums |
e49513afeb50ae9ee0ce5a2a27319867208bd4062d419f0904fba28237f78267
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | disco_data_logger-0.1.5-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.13 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
2a043ce901e4aa628de57644cfdb2a8ab078fd01dccdfcccd35deda3fbee75d9
|
|
BLAKE2b-256 checksum How to use checksums |
432e242ef6f3015f207b2cbcf31c04f534fc569103e78caeb64bb0377b91288b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | disco_data_logger-0.1.5-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 762.8 kB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
223e7e5eacebb75ed05c636f1cd14763a2df04c08c6ae5c512d08672b2748e00
|
|
BLAKE2b-256 checksum How to use checksums |
44d1d06ad6f70eb30c0433d96dd57142427b7788fa276fd71d6361f5f3e80088
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp313-cp313-macosx_10_13_x86_64.whl
| Download URL | disco_data_logger-0.1.5-cp313-cp313-macosx_10_13_x86_64.whl |
|---|---|
| Size | 966.0 kB |
| Tags | CPython 3.13 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
94efe19aec76c91132585a9de03aa5057b7af3b151c8b400c1a0cc9e402ce6ba
|
|
BLAKE2b-256 checksum How to use checksums |
d6530aed9d67f856148bb37bcc3c8fb9c66fd535c3e02164409db05d2b545c33
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp312-cp312-win_amd64.whl
| Download URL | disco_data_logger-0.1.5-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 864.7 kB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
5539861a61b4c9a01aa15e415e3c9acb70bdedd2c5cbfdd45681c9f012987561
|
|
BLAKE2b-256 checksum How to use checksums |
cda5104e9508ea441bdc249c9f9c16d22cc4a82122dc04ccb43fb6a5dcee3c17
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | disco_data_logger-0.1.5-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.12 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
2be55cab81648140fcb6d48ff85ec650c0b76a6acec8bdedb2e0f917d88d9ec0
|
|
BLAKE2b-256 checksum How to use checksums |
4fee718a8e7afebd9863721fb9b5010464866287d3be78a854a15bfdc183a0a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp312-cp312-macosx_11_0_arm64.whl
| Download URL | disco_data_logger-0.1.5-cp312-cp312-macosx_11_0_arm64.whl |
|---|---|
| Size | 762.7 kB |
| Tags | CPython 3.12 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
e8f0a2fa8ab6e37c4230938e9ce7d080a23a1b8868e2cd44897c41b620714c9d
|
|
BLAKE2b-256 checksum How to use checksums |
2f9e23191a31c598d80431514c2c4c809a6b7c0a3ff955d4b8fb6b04dc9a2fcf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp312-cp312-macosx_10_13_x86_64.whl
| Download URL | disco_data_logger-0.1.5-cp312-cp312-macosx_10_13_x86_64.whl |
|---|---|
| Size | 965.9 kB |
| Tags | CPython 3.12 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
09b5bcffb0ac4c3644088abcc503eceeaee2f6ff4e76a717945ef8942e176f52
|
|
BLAKE2b-256 checksum How to use checksums |
b23160ba38ba8943f2a6fffc7e772378c22ad96d326f7b16adf662f2c8a01200
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp311-cp311-win_amd64.whl
| Download URL | disco_data_logger-0.1.5-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 862.9 kB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
40a5b68e2e591deb27909c4eb38a9407363d66e03ac0cb225af12ec0787590ac
|
|
BLAKE2b-256 checksum How to use checksums |
cda872a309fb6d2467eed157ed99ccc51f773dee52b1130e899c354bdc526695
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | disco_data_logger-0.1.5-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.1 MB |
| Tags | CPython 3.11 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
d7ba2c07d31417c0f847e20e68f9e0b3084a771e47d3a07445500e2a854f6772
|
|
BLAKE2b-256 checksum How to use checksums |
532617263831669f8c371780803cc36f216421ab363dbf29e220e91db9190c7e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | disco_data_logger-0.1.5-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 761.0 kB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
a999114c2fd5b43eb0ffd8c512241e774cb58e78e7788e4e9f083bb79bd515b8
|
|
BLAKE2b-256 checksum How to use checksums |
55821732d45258067b6acfc8888eee80a7b0307325e3fc4d788536e02bb33bd6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logRelease files / disco_data_logger-0.1.5-cp311-cp311-macosx_10_9_x86_64.whl
| Download URL | disco_data_logger-0.1.5-cp311-cp311-macosx_10_9_x86_64.whl |
|---|---|
| Size | 962.6 kB |
| Tags | CPython 3.11 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
2c45a4074fb7a912f427e60f00645e54486075c342386a7752dbf1f992981a7d
|
|
BLAKE2b-256 checksum How to use checksums |
50574b184dfd7bb8a5c2d0327b981f557dde2af647621665fa75f3f72ce7de45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logSecurity for disco-data-logger 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 13 files in this release have been uploaded with provenance metadata.
Attestation for disco_data_logger-0.1.5.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 28, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
