Python parser for binary log files written with the TiDaL C++ logging library
TiDaL is a time-series data logging utility for C++ applications. It consists of two parts:
- A header-only C++ library for logging numerical data (scalars or Eigen vectors/matrices), with an associated timestamp, to a file
- A Python module for parsing log-file data into numpy arrays for further processing and/or plotting
Table of Contents
- Log format
- C++ logging
- Python parsing
Each log file can contain multiple named "streams," where each stream is an independent time series. The time stamps need not be synchronized between streams. Each stream can be one of the following types:
- Scalar: More properly, a tuple of any combination of the supported scalar types (signed and unsigned integers, single- or double-precision floating point numbers, or boolean). Labels for each of the fields in the tuple can also be written to the log file.
- Vector: A one-dimensional vector of any of the supported scalar types
- Matrix: A two-dimensional matrix of any of the supported scalar types
Data is stored in a binary format in the log files, with minimal metadata overhead. As a result, logging and parsing are fast and the files are compact.
Note: Requires C++17 support.
Example usage can also be seen in the file
Each log file is managed by a
#include <tidal/tidal.h> tidal::Log log("filename.bin");
Streams are added by calling the appropriate
add_<type>_stream() method on the
Log object, which then returns a
std::shared_ptr<> to the
Stream object that provides a
log() function for actually writing data to that stream in the log file. Streams can be added at any time, even if data has already been logged to other streams.
Specific instructions for each stream type are given below.
A scalar stream can contain any number of elements that are of the supported scalar types.
The supported scalar types are
bool. Equivalent types such as
int instead of
int32_t are also supported.
For example, a scalar stream named "My Scalar Stream" and consisting of a
int, and a
bool would be added and then used as
auto scalar_stream = log.add_scalar_stream<double, int, bool>("My Scalar Stream"); uint64_t timestamp = 1234567; scalar_stream->log(timestamp, 23.5, 42, true);
The SFINAE implementation of the library ensures at compile time that only the correct number and types of values are passed to the
log() function. The type of
scalar_stream in this example is
Additionally, scalar streams support adding labels for each of the fields, such as
scalar_stream->set_labels("x", "y", "z");
where the correct number of labels is enforced at compile time.
A vector stream is used to log fixed-size Eigen column vectors, such as
Eigen::Matrix<double, 6, 1>. Vector streams are added by calling the
add_vector_stream<>() method, whose template arguments are the datatype and length of the Eigen vector.
For example, a vector stream named "My Vector Stream" for vectors of type
Eigen::Vector3d would be added and then used as
#include <eigen3/Eigen/Core> auto vector_stream = log.add_vector_stream<double, 3>("My Vector Stream"); uint64_t timestamp = 1234567; Eigen::Vector3d x; x << 1.0, 2.0, 3.0; vector_stream->log(timestamp, x);
The type of
vector_stream in this example is
A matrix stream is used to log fixed-size Eigen matrices, such as
Eigen::Matrix<double, 6, 4>. Matrix streams are added by calling the
add_matrix_stream<>() method, whose template arguments are the datatype, rows, and columns of the Eigen matrix.
For example, a matrix stream named "My Matrix Stream" for matrices of type
Eigen::Matrix3d would be added and then used as
#include <eigen3/Eigen/Core> auto matrix_stream = log.add_matrix_stream<double, 3, 3>("My Matrix Stream"); uint64_t timestamp = 1234567; Eigen::Matrix3d X = Eigen::Matrix3d::Identity(); matrix_stream->log(timestamp, X);
The type of
matrix_stream in this example is
Including in your project
The entire logging library is contained in the header file
tidal.h. The easiest way to include this library in your project is probably to add it as a submodule.
For example, with a CMake-based project using Git, I might add this library in the
lib/tidal directory with
git submodule add https://github.com/dpkoch/tidal.git lib/tidal
Then in my
CMakeLists.txt I may have something like
include( # other include directories lib/tidal/include )
After which I can include the header file with
The python parsing package is available from Python Package Index as the
tidal-parser package. This can be installed with pip:
python3 -m pip install --user tidal-parser
A log file is parsed by creating a
Parser object with the location of the log file as its argument:
from tidal_parser import Parser log = Parser('/path/to/my/log/file.bin')
The data is then accessed as numpy arrays through the
data members of the
Parser object. These members are Python dictionaries whose keys are the stream names specified by the C++
add_<type>_stream() methods described above.
The following examples show how the streams added in the C++ snippets above might be accessed:
from tidal_parser import Parser log = Parser("filename.bin") # scalar stream scalar_time = log.time['My Scalar Stream'] x = log.data['My Scalar Stream']['x'] y = log.data['My Scalar Stream']['y'] z = log.data['My Scalar Stream']['z'] # vector stream vector_time = log.time['My Vector Stream'] vector_data = log.data['My Vector Stream'] # returns nx3 array, where n is the number of timesteps v_0 = log.data['My Vector Stream'][:,0] # get the 1st element of the vector across all timesteps # matrix stream matrix_time = log.time['My Matrix Stream'] matrix_data = log.data['My Matrix Stream'] # returns nx3x3 array, where n is the number of timesteps m_1_2 = log.data['My Matrix Stream'][:,1,2] # get the (1,2) element of the matrix across all timesteps
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size tidal_parser-0.0.1.post1-py3-none-any.whl (7.1 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size tidal-parser-0.0.1.post1.tar.gz (6.4 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for tidal_parser-0.0.1.post1-py3-none-any.whl