zynamon 0.3.0

Generalized & powerful time-series class and related functions

zynamon

Generalized & powerful time-series class and related functions.

Synopsis

This library defines a powerful, generic time-series representation based on the popular pandas DataFrames. However, it adds meta-data as well as conveniently built-in filtering & alignment routines for both value and time dimensions. Moreover, time-series objects can be constructed from nearly any 1D data such that handling of various sources in a standardised way is made possible (e.g. regularly-sampled sensor data, event logs, audio streams etc).

Package Contents

The package is structured into the following modules:

zeit: Main class TimeSeries containing time-, value- and samples-related methods and algorithmic operators (+, -, *, /)

zeitspec: Class TimeSpec and conversion routine for a unified treatment of various timestamp representations as settings or from data (types and strings)

imex: Routines for the automatic bulk import, conversion & aggregation of data from (a lot of) CSV-like files, e.g. proper extraction of all individual time-series hidden in lengthy event logs

utils: Utilities working on TimeSeries objects such as helpers for general analysis (e.g. outliers, predictions, history etc) and operators on two arbitrary objects (incl. routines to make objects "time-coherent" by aggregation / re-sampling)

xutils: Special conversion utilities for CSV-files containing DFT spectra as generated by the "CMS X-Tools" real-time multi-channel analysis software (Siemens AG).

CLI Application

Besides the library core, the cli module provides convenient script routines for an automatic processing (e.g. bulk import and/or conversion from - a loooot of! - CSV-like files). After installation of the package, these routines can either be invoked via the Python module zynamon.cli or even simpler as standalone executables from the terminal:

zynamon ROUTINE

In total, the following tasks of ROUTINE are offered by the CLI application:

• importer
• compresser
• merger

and also a generic summarizer that will collate readable information on existing time-series collections within a given root folder. Note: By default this step is already performed by the above routines.

All of the routines come with a reasonable default behaviour such that only few arguments are actually required while missing details are "guessed in a smart way". Using the --help option will generate self-explanatory information for an intuitive usage on all routines. For more details refer to the background section (e.g. expected folder structures).

GUI Application

A special routine of the above interface is finally given by

zynamon explorer

as this provides a web-based GUI experience powererd by the popular streamlit rapid prototyping package. This enables to quickly visualise multiple time-series from previously stored collection files while additional cropping, filtering and aggregation options can be applied on-the-fly (i.e. in a non-destructive way).

Background

This section provides additional information for users.

Collection & summary files

TimeSeries objects can either be stored individually or in collection files, housing one or more objects at a time. For the latter, both dict or list arrangements are available in Python, allowing to choose the more suitable layout.

In general, the following file formats are supported:

• *.pk => Python's builtin pickle
• *.json => JSON / readable dict-type
• *.h5 => HDF5 hierachical storage format
• *.parquet => Apache parquet file (implemented by pyarrow package, only for single files!)

Since collections may include a large number of different time-series, it may be desirable to have means for a quick overview. To this end, so called summary files with extension *.tsinfo can be generated, offering details in readable text as follows:

• collection totals: # time-series, min/max lengths, common time interval (if any)
• per time-series details: name, number of samples, time interval covered

Folder structures for projects

For "smart guess" defaults to work, a certain folder hierarchy is expected and must be present below each project's root location. This expected structure is shown in the following:

• ---Project---
  • --Asset A1--
    • -Period P1-
      • subfolders "log-enum" (if any)
      • subfolders "log-real" (if any)
      • subfolders "streams" (if any)
      • subfolders "streams-xt" (if any)
    • -Period P2-
      • subfolders "log-enum" (if any)
      • subfolders "log-real" (if any)
      • subfolders "streams" (if any)
      • subfolders "streams-xt" (if any)
    • ...
  • --Asset A2--
    • -Period P1-
      • ...
    • -Period P2-
      • ...
  • ...

Note that all of the subfolder categories within a period are optional and, thus, zero or more items may be given. However, at least one item must be given for any category in every period, as otherwise no data will be processed!

Typical example calls

a) Importing all files acc. to a project configuration and saving all time-series separately:

zynamon importer . --cfg-file D:/MyData/cfg_project.json --save-series

b) Compressing collection files for given assets & time period sub-folders acc. desired settings:

zynamon compresser D:/MyProject [Asset1,Asset2] 2025-07 --agg-time [10,60,300,3600] --agg-mode [avg,min,max]

c) Traverse given folder and create summary files for all collections (even if different extensions may be found) and existing summary files may be overwritten:

zynamon summarizer D:/MyProject --ignore-ext --overwrite --verbosity 1

Note: This use case may occur if collection files are created at different points in time (after new data has become available) and/or with different storage formats and proper summarizing needs to be repeated.

Template for configuration files

To create a JSON configuration for a specific project (folder) layout, the following schema can be used. Obviously, not all of these fields are mandatory, but rather dependent on the actual contents of the CSV information. To keep the text short, a simplified schema documenting style is used here, where the expected types for values are denoted by /schema-type/ and *required or *required-if-category-used is appended if the field is essential:

{
    "name": /str/,
    "info": /str/,

    "root": /str/, *required*
    "path": /str/, *required*

    "assets":  /list[str]/,
    "periods": /list[str]/,

    "sub_logs_enum":  /list[str]/,
    "sub_logs_real":  /list[str]/,
    "sub_streams":    /list[str]/,
    "sub_streams_xt": /list[str]/,
    
    "def_logs_enum": /dict/ *required-if-category-used*
    {
        "name": /str/ | /list[str]/, *required*
        "time": /str/, *required*
        "data": /str/  *required*
    },
        
    "def_logs_real": /dict/ *required-if-category-used* 
    {
        "name": /str/ | /list[str]/, *required*
        "time": /str/, *required*
        "data": /str/  *required*
    },

   "def_streams": /dict/ *required-if-category-used*  
   {
        "time": /str/, *required*
        "data": /list[str]/ *required*
    },

    "def_streams_xt": /dict/ *required-if-category-used*  
    {
        "time": /str/, *required*
        "data": /str/ *required*
    },

    "_comment": /str/    
}

[ Dr. Marcus Zeller | [e-mail] | Erlangen, Germany | 2022-2026 ]