A binary data log library
Project description
binlog
A rust library for creating and managing logs of arbitrary binary data. Presently it's used to collect sensor data. But it should generally be helpful in cases where you need to store timeseries data, in a nearly (but not strictly) append-only fashion.
The underlying storage of logs are pluggable via a few traits. Binlog includes built-in implementations via sqlite, redis, and in-memory-only. Additionally, python bindings allow you to use (a subset of) binlog from python.
Usage
From Rust
A small example:
use binlog::{Entry, Error, Range, RangeableStore, SqliteStore, Store};
use std::borrow::Cow;
use string_cache::Atom;
/// Demonstrates the sqlite store, with results in `example.db`. You may want to delete that before
/// running this to see the results of this on an empty database.
fn main() -> Result<(), Error> {
// Create a new datastore with sqlite backing. The result will be stored in example.db, with
// default compression options. In-memory is also possible via
// `binlog::MemoryStore::default()`.
let store = SqliteStore::new("example.db", None)?;
// Add 10 entries.
for i in 1..11u8 {
let entry = Entry::new_with_timestamp(i as i64, Atom::from("sqlite_example"), vec![i]);
store.push(Cow::Owned(entry))?;
}
// Queries are done via `range`. Here we grab entries with any timestamp and any name.
let range = store.range(.., Option::<String>::None)?;
// Count the number of entries.
println!("initial count: {}", range.count()?);
// We can also iterate on the entries.
for entry in range.iter()? {
println!("entry: {:?}", entry?);
}
// Remove the entries with 4 <= ts <= 6 and with the name `sqlite_example`.
store.range(4..=6, Some(Atom::from("sqlite_example")))?.remove()?;
// Now get the range of entries with 5 <= ts and with the name `sqlite_example`.
let range = store.range(5.., Some(Atom::from("sqlite_example")))?;
println!("count after range deletion: {}", range.count()?);
for entry in range.iter()? {
println!("entry: {:?}", entry?);
}
Ok(())
}
From Python
A small example:
from binlog import binlog
store = binlog.SqliteStore("example.db")
store.push(binlog.Entry(1, "pytest_push", [1, 2, 3]))
Stores
Stores implement the Store
trait, and zero or more optional extensions depending on their supported functionality. A few stores implementations are built-in to binlog
:
In-memory-only
The in-memory-only store has no means of persistence, but offers the full log functionality. This is also used internally for fuzzing other implementations against.
Redis
The redis implementation is enableable via the redis-store
feature. Under the hood, it uses redis streams. It supports subscriptions, but not ranges.
Sqlite
The sqlite implementation is enableable via the sqlite-store
feature. It supports ranges, but not subscriptions.
Testing
Unit tests
Tests can be run via make test
. This will also be run in CI.
Benchmarks
Benchmarks can be run via make bench
.
Fuzzing
A fuzzer is available, ensuring the the sqlite and in-memory datastores operate identically. Run it via make fuzz
.
Checks
Lint and formatting checks can be run via make check
. Equivalent checks will also run in CI.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file binlog-0.5.0.tar.gz
.
File metadata
- Download URL: binlog-0.5.0.tar.gz
- Upload date:
- Size: 34.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9db3bff7d4cae803625c31e4922077eec8bde97750b512745de92831fcf57fc9 |
|
MD5 | 7ef6188aaad4a2c0000518b239666240 |
|
BLAKE2b-256 | 5ee9cec725214d5b6c029a9834675c67784db5cd4ebb4c1a84492e2a63931f5d |
File details
Details for the file binlog-0.5.0-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.whl
.
File metadata
- Download URL: binlog-0.5.0-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.whl
- Upload date:
- Size: 2.5 MB
- Tags: PyPy, manylinux: glibc 2.5+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2bc464c76734673ed44ee50e9848f82cabe452dc6a1b5eedbf9507950fe05e4f |
|
MD5 | 9d61ef66a7a02b1ee9041a9881a15fd3 |
|
BLAKE2b-256 | 09a7ff9276c46fa18a2dceeb07577d01b9a1010c855017ce3a82913bb2f30d0b |
File details
Details for the file binlog-0.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.whl
.
File metadata
- Download URL: binlog-0.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.whl
- Upload date:
- Size: 2.5 MB
- Tags: CPython 3.10, manylinux: glibc 2.5+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17a5133b668d99cbbbffe943fa07a72d585d5ae52d76f1aa001c740dd8acbc2a |
|
MD5 | dbe5bdc043ee0078e13bcb854843c060 |
|
BLAKE2b-256 | 4dbe9090a4a745c7a988b9825a80fd2ecdcaace17b9360aaf55f54fec168fc00 |
File details
Details for the file binlog-0.5.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
.
File metadata
- Download URL: binlog-0.5.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
- Upload date:
- Size: 2.0 MB
- Tags: CPython 3.10, macOS 10.9+ universal2 (ARM64, x86-64), macOS 10.9+ x86-64, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | eaac765457a2a5a514234ee46d6b5e404c4e97bf24cfea4d1bb4770adadad6b4 |
|
MD5 | 4261a3b8237d1b35d5a84c176af8579f |
|
BLAKE2b-256 | 2238e63bff731756044236c83b252787b17ca3a76303122ba3e6ad648e4fe525 |
File details
Details for the file binlog-0.5.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
.
File metadata
- Download URL: binlog-0.5.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
- Upload date:
- Size: 2.5 MB
- Tags: CPython 3.9, manylinux: glibc 2.5+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 124ae3802657c9d1ed0f99d8ee654d2fcbc9e8e286e708ee2838cb8e392e336a |
|
MD5 | 164afd554e25ee1753f7a3042a5b2c3d |
|
BLAKE2b-256 | c4d2143adac2a1550603e910e65a9bd8180bc7c78b03a3699511b63b68ac3286 |
File details
Details for the file binlog-0.5.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
.
File metadata
- Download URL: binlog-0.5.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
- Upload date:
- Size: 2.0 MB
- Tags: CPython 3.9, macOS 10.9+ universal2 (ARM64, x86-64), macOS 10.9+ x86-64, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b22f0c60574a30f6fd7ca5a4beb7f5b0a407801df84591600704b6b50f691d9 |
|
MD5 | 788e365395dc3ea03c8035ca2dd4781f |
|
BLAKE2b-256 | 0f46074135c144210e94f02df3caefd9b13a73cbf6332ffb7f4382eb3ca18856 |
File details
Details for the file binlog-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
.
File metadata
- Download URL: binlog-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
- Upload date:
- Size: 2.5 MB
- Tags: CPython 3.8, manylinux: glibc 2.5+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | de8d875404b7cddb319bb7ed55d81f716242d9fc05c5741696b5118531f34e87 |
|
MD5 | 2536c1f1305193a53a1f47a17478a231 |
|
BLAKE2b-256 | 660db8f712476773944d3d819cfe4a23352e38f8c708b48578fcda72ff0afc06 |
File details
Details for the file binlog-0.5.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
.
File metadata
- Download URL: binlog-0.5.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
- Upload date:
- Size: 2.0 MB
- Tags: CPython 3.8, macOS 10.9+ universal2 (ARM64, x86-64), macOS 10.9+ x86-64, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c703b49873051ab5885658c8ddb41d21a214847478cd3e96d03c77971a5a4f38 |
|
MD5 | ccaad6f03f3ba6c34e3fbe97cc17f12d |
|
BLAKE2b-256 | 4f7b5746d843903bca4aa50445db86eefe09ef40d6eae7ae646e433015eeb935 |
File details
Details for the file binlog-0.5.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
.
File metadata
- Download URL: binlog-0.5.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
- Upload date:
- Size: 2.5 MB
- Tags: CPython 3.7m, manylinux: glibc 2.5+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 72802d51eef32735a9e60e99209c6672f07345f9d20d7ecabf15c30ac96c21c2 |
|
MD5 | 3edd8e42eaf717a7b6754ac7eb93212a |
|
BLAKE2b-256 | 846b8161b30fbaf62846e1b812dd781bbbcf05c53c56c62ed92d52abc81e9907 |
File details
Details for the file binlog-0.5.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
.
File metadata
- Download URL: binlog-0.5.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
- Upload date:
- Size: 2.0 MB
- Tags: CPython 3.7m, macOS 10.9+ universal2 (ARM64, x86-64), macOS 10.9+ x86-64, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/0.12.15
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e2fbb7150010836f83fc979228fb9c7c723028c0387466ff7eb35b9426b5f6d |
|
MD5 | d4926ee138be8cbeb83e863e7e1f8054 |
|
BLAKE2b-256 | bb9186f818e69a85f94587f882ad584c4473bd7a818d9c3205cec7b6d356d540 |