btsf (for Binary Time Series File) is a package to store your data in a condensed and fast yet flexible way.
Project description
btfs - Binary Time Series File
This is a module to store time series data in a custom binary file format (.btfs) with the following objectives:
- small file size
- appendable
- fast to read and write
- easy to adapt to new scenarios
- ability to store metadata (units, annotations, ...)
- targeted at Python (adding support for other languages should be rather straight-forward)
- possibility to tune implementation details such as byte order or padding
Quick description of the file structure of a .btfs file:
File Signature (fixed)
(32 bytes)
at least one intro section
(16-byte header + variable length payload + variable length reserved zero-bytes)
end of intro section header
(16 bytes)
concatenated packed structured data
(structure of fixed length * N, thus appendable!)
A typical use case for btfs is:
- Time series measurements of one or multiple variables (sampled at the same time).
- Appending (=adding) new measurements to file is always possible, eg. after a restart of the acquisition.
- The metadata of the file (stored in the intro sections of the file) can later be updated with additional information, such as data annotations, tags or similar.
What btfs is not suited for, at least not perfectly:
- event-based data with varying number of values or varying value types.
Note: As the size of the introduction sections cannot be expanded, editing the metadata is only possible within the limits of reserved extra space for the intro section, specified when creating the file.
Installation
pip install --upgrade https://github.com/pklaus/btsf/archive/master.zip
Example Usage
from btsf import Metric, MetricType, BinaryTimeSeriesFile
metrics = [
Metric('time', MetricType.Double),
Metric('power', MetricType.Float),
Metric('counter', MetricType.UInt64),
Metric('flags', MetricType.UInt8),
]
with BinaryTimeSeriesFile.create('test.btsf', metrics) as f:
f.append(1.1, 2.2, 0xf0f0f0f0f0f0f0f0, 1)
f.append(3.3, 4.4, 0xffffffffffffffff, 2)
f.append(5.0, 1.0, 0x123456789ABCDEF1, 3)
Checking what's in a .btsf file is easy with the supplied CLI:
btsf info test.btsf
resulting in the following output:
test.btsf - Number of entries: 3
Metrics:
time (MetricType.Double) power (MetricType.Float) counter (MetricType.UInt64) flags (MetricType.UInt8)
entries:
(1.1, 2.200000047683716, 17361641481138401520, 1)
(3.3, 4.400000095367432, 18446744073709551615, 2)
(5.0, 1.0, 1311768467463790321, 3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file btsf-1.0.1.tar.gz
.
File metadata
- Download URL: btsf-1.0.1.tar.gz
- Upload date:
- Size: 11.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a0a098a9da58c59bd7319b7afbd1245bb549ddb20e21ff21a5b825690f8722e1 |
|
MD5 | 156471d91eac5a0878277b59fe844397 |
|
BLAKE2b-256 | f3f1168d41ee2c3cf6267815397a9d7febfa860bb778cad812076198b1e1f9d5 |
File details
Details for the file btsf-1.0.1-py2.py3-none-any.whl
.
File metadata
- Download URL: btsf-1.0.1-py2.py3-none-any.whl
- Upload date:
- Size: 9.3 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 71872d1550fe32c2dec2f1534ef5e6454bd88b7003d0529f913376fcd52ff4ea |
|
MD5 | bee9d0c78ec22f26ae677f7f6d8585b4 |
|
BLAKE2b-256 | 7a77d5a00a7d171b6ddc6f3f50dbe9278ee8e2f249c203fac7ebe997c2405687 |