Zeitgleich 0.1.4

Zeitgleich is a time series library for handling data and timestamps with origins. It provides a way to manage time series data, validate formats, and perform operations on time series data while maintaining origin information.

Zeitgleich

Zeitgleich is a time series library designed for handling timestamped data with origins, specifically developed for use within an MQTT ecosystem in combination with the SwampClient. It allows you to manage different timestamp formats, validate data schemas, and perform pandas operations on time series data while maintaining origin information.

Features

• Pandas Operations: Perform standard Pandas operations (arithmetic, indexing, aggregation) directly on time series data while maintaining origin information.
• Uniform Timestamp Handling: Convert between ISO, Unix (seconds/nanoseconds), and RFC3339 formats.
• Flexible Timestamp Conversion: Specify desired output timestamp formats, defaulting to RFC3339, with support for various input formats.
• Timestamp Normalization: Standardize timestamps to a consistent format, localized to a specified timezone.
• Origin Extraction: Extract the device and topic from the origin, typically representing the MQTT topic the data originated from.
• Data Validation: Validate that input data (dictionaries or DataFrames) contain the required timestamp and value fields, with configurable naming presets.
• Customization: Adjust timestamp formats, column names, and origin patterns using method parameters.
• Multi-Origin Time Series: Manage MQTT data from multiple topics simultaneously using the MultiOriginTimeSeries class.

Installation

pip install Zeitgleich

Quick Start

Zeitgleich provides several examples to help you get started. These examples demonstrate how to manage single and multiple origins, perform timestamp conversions, and validate data formats. Refer to the examples directory in the project repository for detailed implementations.

Managing Single Origin with Timestamp Conversion

Initialize a TimeSeriesData object with a single origin, convert timestamps from ISO format to Unix nanoseconds, and extract origin information.

Managing Multiple Origins with MultiOriginTimeSeries

Handle MQTT data from multiple topics using the MultiOriginTimeSeries class. Add data from different origins with varying input timestamp formats and perform aggregate operations.

Utilizing TimestampFormat for Various Conversions

Use the TimestampFormat class to convert timestamps for single values, lists, dictionaries, and DataFrames, including handling timezones.

Data Validation and Error Handling

Zeitgleich enforces data validation rules by handling incorrect origin formats and missing data columns.

For complete code implementations, refer to the examples directory in the project repository.

Data Formats

Timestamp Normalization

Zeitgleich normalizes timestamps by:

• Parsed into datetime Objects: Timestamps are converted into datetime objects for consistency and ease of manipulation.
• Timezone-Aware: Timestamps are localized to a specified timezone (default is UTC) and can be converted to other timezones if needed.
• Consistent Format: After conversion, all timestamps have a consistent format as specified by the output_timestamp_format parameter, facilitating time-based operations.

Flexible Timestamp Conversion

Zeitgleich allows for flexible conversion between various timestamp formats by specifying both input and output formats. Here's how you can customize them:

• Input Timestamp Format (input_timestamp_format): Specify the format of your input timestamps using the input_timestamp_format parameter. Supported formats include:

  • TimestampFormat.ISO (e.g., "2021-06-05T12:00:00.123456")
  • TimestampFormat.RFC3339 (e.g., "2021-06-05T12:00:00.123456Z")
  • TimestampFormat.UNIX (e.g., 1622894400)
  • TimestampFormat.EPOCH_S (e.g., 1622894400.123456)
  • TimestampFormat.EPOCH_NS (e.g., 1622894400123456000)

• Output Timestamp Format (output_timestamp_format): Specify the desired format for your output timestamps using the output_timestamp_format parameter. Default is TimestampFormat.RFC3339, but you can choose any supported format.

• Timestamp Column Name (timestamp_col): Use the timestamp_col parameter to specify the name of your timestamp column if it differs from the default 'timestamp'.

• Timezone (timezone_obj): Use the timezone_obj parameter to set the timezone for timestamp localization. This parameter accepts a datetime.timezone object.

Origin Format

By default, the origin is assumed to be in the format <device>/<topic>. This typically represents the MQTT topic the data originated from. You can adjust the regex patterns in the get_device() and get_topic() methods to match your specific origin format.

DataFrame Format

The DataFrame should contain a timestamp column and at least one data column (e.g., value). The timestamp column will be converted to the desired format and, by default, set as the index of the DataFrame.

Format Validation

Zeitgleich includes data validation to ensure that your time series data adheres to the expected formats for origins and data columns. This is achieved using the validate_format decorator from FormatChecks.py.

Origin and Column Checks

When initializing TimeSeriesData or adding data to MultiOriginTimeSeries, the following validations are performed:

1. Origin Format Check:

   • Ensures that the origin string matches the expected pattern (default: <device>/<topic>).
   • Raises a ValueError if the origin does not conform to the specified regex pattern.

2. Data Columns Check:

   • Confirms the presence of required data columns (timestamp and value by default).
   • Raises a ValueError if any required column is missing.

For detailed examples, refer to the examples directory.

Unit Tests

This project includes unit tests to ensure functionality. Tests cover:

• Timestamp Format Detection: Verifying that different timestamp formats are correctly identified.
• Timestamp Conversion: Ensuring accurate conversion between timestamp formats.
• DataFrame Validation: Checking that data conforms to expected formats.
• Origin Extraction: Testing extraction of device and topic from origin strings.
• Pandas Operations on Time Series Data: Validating that standard Pandas operations work as expected on the time series data.

To run the tests, use the following command:

make tests

Note: A Makefile is provided to simplify running all tests.

Future Developments

In the future, a complementary library will be developed to handle database interactions, allowing pushing of data received from different MQTT topics to a PostgreSQL time series database. This feature is currently under development.

License

Licensed under the MIT License.