Skip to main content

btsf (for Binary Time Series File) is a package to store your data in a condensed and fast yet flexible way.

Project description

btfs - Binary Time Series File

This is a module to store time series data in a custom binary file format (.btfs) with the following objectives:

  • small file size
  • appendable
  • fast to read and write
  • easy to adapt to new scenarios
  • ability to store metadata (units, annotations, ...)
  • targeted at Python (adding support for other languages should be rather straight-forward)
  • possibility to tune implementation details such as byte order or padding

Quick description of the file structure of a .btfs file:

File Signature (fixed)
(32 bytes)

at least one intro section
(16-byte header + variable length payload + variable length reserved zero-bytes)

end of intro section header
(16 bytes)

concatenated packed structured data
(structure of fixed length * N, thus appendable!)

A typical use case for btfs is:

  • Time series measurements of one or multiple variables (sampled at the same time).
  • Appending (=adding) new measurements to file is always possible, eg. after a restart of the acquisition.
  • The metadata of the file (stored in the intro sections of the file) can later be updated with additional information, such as data annotations, tags or similar.

What btfs is not suited for, at least not perfectly:

  • event-based data with varying number of values or varying value types.

Note: As the size of the introduction sections cannot be expanded, editing the metadata is only possible within the limits of reserved extra space for the intro section, specified when creating the file.

Installation

pip install --upgrade https://github.com/pklaus/btsf/archive/master.zip

Example Usage

from btsf import Metric, MetricType, BinaryTimeSeriesFile

metrics = [
    Metric('time', MetricType.Double),
    Metric('power', MetricType.Float),
    Metric('counter', MetricType.UInt64),
    Metric('flags', MetricType.UInt8),
]

with BinaryTimeSeriesFile.create('test.btsf', metrics) as f:
    f.append(1.1, 2.2, 0xf0f0f0f0f0f0f0f0, 1)
    f.append(3.3, 4.4, 0xffffffffffffffff, 2)
    f.append(5.0, 1.0, 0x123456789ABCDEF1, 3)

Checking what's in a .btsf file is easy with the supplied CLI:

btsf info test.btsf

resulting in the following output:

test.btsf - Number of entries: 3
Metrics:
time (MetricType.Double)   power (MetricType.Float)   counter (MetricType.UInt64)   flags (MetricType.UInt8)
entries:
(1.1, 2.200000047683716, 17361641481138401520, 1)
(3.3, 4.400000095367432, 18446744073709551615, 2)
(5.0, 1.0, 1311768467463790321, 3)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

btsf-1.0.1.tar.gz (11.5 kB view details)

Uploaded Source

Built Distribution

btsf-1.0.1-py2.py3-none-any.whl (9.3 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file btsf-1.0.1.tar.gz.

File metadata

  • Download URL: btsf-1.0.1.tar.gz
  • Upload date:
  • Size: 11.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.3

File hashes

Hashes for btsf-1.0.1.tar.gz
Algorithm Hash digest
SHA256 a0a098a9da58c59bd7319b7afbd1245bb549ddb20e21ff21a5b825690f8722e1
MD5 156471d91eac5a0878277b59fe844397
BLAKE2b-256 f3f1168d41ee2c3cf6267815397a9d7febfa860bb778cad812076198b1e1f9d5

See more details on using hashes here.

File details

Details for the file btsf-1.0.1-py2.py3-none-any.whl.

File metadata

  • Download URL: btsf-1.0.1-py2.py3-none-any.whl
  • Upload date:
  • Size: 9.3 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.3

File hashes

Hashes for btsf-1.0.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 71872d1550fe32c2dec2f1534ef5e6454bd88b7003d0529f913376fcd52ff4ea
MD5 bee9d0c78ec22f26ae677f7f6d8585b4
BLAKE2b-256 7a77d5a00a7d171b6ddc6f3f50dbe9278ee8e2f249c203fac7ebe997c2405687

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page