Skip to main content

A binary data log library

Project description

binlog

Test crates.io API docs

A rust library for creating and managing logs of arbitrary binary data. Presently it's used to collect sensor data. But it should generally be helpful in cases where you need to store timeseries data, in a nearly (but not strictly) append-only fashion.

The underlying storage of logs are pluggable via a few traits. Binlog includes built-in implementations via sqlite, redis, and in-memory-only. Additionally, python bindings allow you to use (a subset of) binlog from python.

Usage

From Rust

A small example:

use binlog::{Entry, Error, Range, RangeableStore, SqliteStore, Store};
use std::borrow::Cow;
use string_cache::Atom;

/// Demonstrates the sqlite store, with results in `example.db`. You may want to delete that before
/// running this to see the results of this on an empty database.
fn main() -> Result<(), Error> {
    // Create a new datastore with sqlite backing. The result will be stored in example.db, with
    // default compression options. In-memory is also possible via
    // `binlog::MemoryStore::default()`.
    let store = SqliteStore::new("example.db", None)?;

    // Add 10 entries.
    for i in 1..11u8 {
        let entry = Entry::new_with_timestamp(i as i64, Atom::from("sqlite_example"), vec![i]);
        store.push(Cow::Owned(entry))?;
    }

    // Queries are done via `range`. Here we grab entries with any timestamp and any name.
    let range = store.range(.., Option::<String>::None)?;
    // Count the number of entries.
    println!("initial count: {}", range.count()?);
    // We can also iterate on the entries.
    for entry in range.iter()? {
        println!("entry: {:?}", entry?);
    }

    // Remove the entries with 4 <= ts <= 6 and with the name `sqlite_example`.
    store.range(4..=6, Some(Atom::from("sqlite_example")))?.remove()?;

    // Now get the range of entries with 5 <= ts and with the name `sqlite_example`.
    let range = store.range(5.., Some(Atom::from("sqlite_example")))?;
    println!("count after range deletion: {}", range.count()?);
    for entry in range.iter()? {
        println!("entry: {:?}", entry?);
    }

    Ok(())
}

From Python

A small example:

from binlog import binlog
store = binlog.SqliteStore("example.db")
store.push(binlog.Entry(1, "pytest_push", [1, 2, 3]))

Stores

Stores implement the Store trait, and zero or more optional extensions depending on their supported functionality. A few stores implementations are built-in to binlog:

In-memory-only

The in-memory-only store has no means of persistence, but offers the full log functionality. This is also used internally for fuzzing other implementations against.

Redis

The redis implementation is enableable via the redis-store feature. Under the hood, it uses redis streams. It supports subscriptions, but not ranges.

Sqlite

The sqlite implementation is enableable via the sqlite-store feature. It supports ranges, but not subscriptions.

Testing

Unit tests

Tests can be run via make test. This will also be run in CI.

Benchmarks

Benchmarks can be run via make bench.

Fuzzing

A fuzzer is available, ensuring the the sqlite and in-memory datastores operate identically. Run it via make fuzz.

Checks

Lint and formatting checks can be run via make check. Equivalent checks will also run in CI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

binlog-0.5.0.tar.gz (34.6 kB view details)

Uploaded Source

Built Distributions

binlog-0.5.0-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.5 MB view details)

Uploaded PyPy manylinux: glibc 2.5+ x86-64

binlog-0.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.5+ x86-64

binlog-0.5.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (2.0 MB view details)

Uploaded CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

binlog-0.5.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.5+ x86-64

binlog-0.5.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (2.0 MB view details)

Uploaded CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

binlog-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.5+ x86-64

binlog-0.5.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (2.0 MB view details)

Uploaded CPython 3.8 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

binlog-0.5.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.5+ x86-64

binlog-0.5.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (2.0 MB view details)

Uploaded CPython 3.7m macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

File details

Details for the file binlog-0.5.0.tar.gz.

File metadata

  • Download URL: binlog-0.5.0.tar.gz
  • Upload date:
  • Size: 34.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/0.12.15

File hashes

Hashes for binlog-0.5.0.tar.gz
Algorithm Hash digest
SHA256 9db3bff7d4cae803625c31e4922077eec8bde97750b512745de92831fcf57fc9
MD5 7ef6188aaad4a2c0000518b239666240
BLAKE2b-256 5ee9cec725214d5b6c029a9834675c67784db5cd4ebb4c1a84492e2a63931f5d

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-pp37-pypy37_pp73-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2bc464c76734673ed44ee50e9848f82cabe452dc6a1b5eedbf9507950fe05e4f
MD5 9d61ef66a7a02b1ee9041a9881a15fd3
BLAKE2b-256 09a7ff9276c46fa18a2dceeb07577d01b9a1010c855017ce3a82913bb2f30d0b

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 17a5133b668d99cbbbffe943fa07a72d585d5ae52d76f1aa001c740dd8acbc2a
MD5 dbe5bdc043ee0078e13bcb854843c060
BLAKE2b-256 4dbe9090a4a745c7a988b9825a80fd2ecdcaace17b9360aaf55f54fec168fc00

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 eaac765457a2a5a514234ee46d6b5e404c4e97bf24cfea4d1bb4770adadad6b4
MD5 4261a3b8237d1b35d5a84c176af8579f
BLAKE2b-256 2238e63bff731756044236c83b252787b17ca3a76303122ba3e6ad648e4fe525

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 124ae3802657c9d1ed0f99d8ee654d2fcbc9e8e286e708ee2838cb8e392e336a
MD5 164afd554e25ee1753f7a3042a5b2c3d
BLAKE2b-256 c4d2143adac2a1550603e910e65a9bd8180bc7c78b03a3699511b63b68ac3286

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 5b22f0c60574a30f6fd7ca5a4beb7f5b0a407801df84591600704b6b50f691d9
MD5 788e365395dc3ea03c8035ca2dd4781f
BLAKE2b-256 0f46074135c144210e94f02df3caefd9b13a73cbf6332ffb7f4382eb3ca18856

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 de8d875404b7cddb319bb7ed55d81f716242d9fc05c5741696b5118531f34e87
MD5 2536c1f1305193a53a1f47a17478a231
BLAKE2b-256 660db8f712476773944d3d819cfe4a23352e38f8c708b48578fcda72ff0afc06

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 c703b49873051ab5885658c8ddb41d21a214847478cd3e96d03c77971a5a4f38
MD5 ccaad6f03f3ba6c34e3fbe97cc17f12d
BLAKE2b-256 4f7b5746d843903bca4aa50445db86eefe09ef40d6eae7ae646e433015eeb935

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 72802d51eef32735a9e60e99209c6672f07345f9d20d7ecabf15c30ac96c21c2
MD5 3edd8e42eaf717a7b6754ac7eb93212a
BLAKE2b-256 846b8161b30fbaf62846e1b812dd781bbbcf05c53c56c62ed92d52abc81e9907

See more details on using hashes here.

File details

Details for the file binlog-0.5.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for binlog-0.5.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 0e2fbb7150010836f83fc979228fb9c7c723028c0387466ff7eb35b9426b5f6d
MD5 d4926ee138be8cbeb83e863e7e1f8054
BLAKE2b-256 bb9186f818e69a85f94587f882ad584c4473bd7a818d9c3205cec7b6d356d540

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page