pipeline-telemetry 1.0.0

Measure your data pipelines with easy to use telemetry logic

======== Overview

.. start-badges

.. list-table:: :stub-columns: 1

* - docs
  - |docs|
* - tests
  - | |travis| |requires|
    | |codecov|
* - package
  - | |version| |wheel|
    | |supported-versions|
    | |supported-implementations|
    | |commits-since|

.. |docs| image:: https://readthedocs.org/projects/pipeline-telemetry/badge/?style=flat :target: https://pipeline-telemetry.readthedocs.io/ :alt: Documentation Status

.. |travis| image:: https://api.travis-ci.com/MaartendeRuyter/pipeline-telemetry.svg?branch=master :alt: Travis-CI Build Status :target: https://travis-ci.com/github/MaartendeRuyter/pipeline-telemetry

.. |requires| image:: https://requires.io/github/MaartendeRuyter/pipeline-telemetry/requirements.svg?branch=master :alt: Requirements Status :target: https://requires.io/github/MaartendeRuyter/pipeline-telemetry/requirements/?branch=master

.. |codecov| image:: https://codecov.io/gh/MaartendeRuyter/pipeline-telemetry/branch/master/graphs/badge.svg?branch=master :alt: Coverage Status :target: https://codecov.io/github/MaartendeRuyter/pipeline-telemetry

.. |version| image:: https://img.shields.io/pypi/v/pipeline-telemetry.svg :alt: PyPI Package latest release :target: https://pypi.org/project/pipeline-telemetry

.. |wheel| image:: https://img.shields.io/pypi/wheel/pipeline-telemetry.svg :alt: PyPI Wheel :target: https://pypi.org/project/pipeline-telemetry

.. |supported-versions| image:: https://img.shields.io/pypi/pyversions/pipeline-telemetry.svg :alt: Supported versions :target: https://pypi.org/project/pipeline-telemetry

.. |supported-implementations| image:: https://img.shields.io/pypi/implementation/pipeline-telemetry.svg :alt: Supported implementations :target: https://pypi.org/project/pipeline-telemetry

.. |commits-since| image:: https://img.shields.io/github/commits-since/MaartendeRuyter/pipeline-telemetry/v0.0.1.svg :alt: Commits since latest release :target: https://github.com/MaartendeRuyter/pipeline-telemetry/compare/v0.0.1...master

.. end-badges

Create and store data pipeline telemetry data

• Free software: GNU Lesser General Public License v3 or later (LGPLv3+)

Installing pipeline-telemetry

::

pip install pipeline-telemetry

You can also install the in-development version with::

pip install https://github.com/MaartendeRuyter/pipeline-telemetry/archive/master.zip

Documentation

https://pipeline-telemetry.readthedocs.io/

Testing

To run all the tests run::

tox

Note, to combine the coverage data from all the tox environments run:

.. list-table:: :widths: 10 90 :stub-columns: 1

- - Windows
  - ::

        set PYTEST_ADDOPTS=--cov-append
        tox

- - Other
  - ::

        PYTEST_ADDOPTS=--cov-append tox

Changelog

1.0.0 (2024-05-08)

• Added python 3.12 support

0.9.4 (2022-11-03)

• Added PartialToSingleMongoAggregator and PartialToSingleAggregator classes to allow automated aggregation of multiple partial telemetry objects to a single telemetry object.

0.9.3 (2022-10-26)

• Added support for python 3.11

0.9.2 (2022-10-11)

• Added automatic deletion of old aggegration telemetry

0.9.1 (2022-10-04)

• Fixed defect in MongoStorage.

0.9.0 (2022-10-03)

• Added DailyMongoAggregator and DailyAggregator classes to allow automated aggregation of multiple intraday telemetry objects to daily telemetry objects.

0.4.5 (2022-06-09)

• Added set_telemetry_source_name method to TelemetryMixin class

0.4.4 (2022-06-08)

• Added custom hash method to TelemetryCounter

0.4.3 (2022-06-01)

• Added set_increment method to TelemetryCounter class.

0.4.2 (2022-05-17)

• Added process_telemetry_counters_from_list method to TelemetryMixin class.

0.4.1 (2022-05-16)

• Added process_telemetry_counters_from_return_value method to TelemetryMixin class.
• Added documentation on TelemetryCounter

0.4.0 (2022-05-15)

• Added TelemetryMixin class to enable easy including of telemetry update methods in any datapipeline class.

0.3.1 (2022-05-13)

• Added process_telemetry_counters_in_return_value helper method to easily add retrieve and process TelemetryCounter objects from a return value object.
• Added process_return_value helper method to process both ErrorCode and TelemetryCounter objects from a return value object.
• Added more documentation

0.3.0 (2022-05-12)

• Added add_errors_from_return_value helper method to easily add the errors from a ReturnValueWithStatus object from error-manager package.
• Automatically initialize a missing sub_process when a telemetry counter is added to the telemetry instance.

0.2.18 (2022-03-2)

• Added add_single_usage_telemetry decorator to be used for single usage
• telemetry objects

0.2.16 (2022-02-01)

• Added telemetry_type and io_time_in_seconds attributes to Telemetry
• object including a method to increase io_time_in_seconds.

0.2.12 (2022-01-19)

• Added traffic_light attribute to Telemetry object indicating the success state of the datapipeline process that is reporting on
• Some minor refactoring

0.2.11 (2022-01-18)

• Added created_at field in mongo storage class for better date selection
• of the telemetry objects

0.2.10 (2022-01-18)

• Added indexes to mongo storage class

0.2.8 (2022-01-13)

• Fix in storage classes to ensure category and sub_category to be stored in the toplevel of the telemetry object

0.2.7 (2022-01-13)

• Added fields category and sub_category to the telemetry object to allow for better distinction between telemetry sources
• Rename telemetry field 'process_name' to 'soure_name' to be more clear about the data source in scope of the telemetry object

0.2.6 (2021-12-08)

• Added default increment value = 1 to methods increase_base_count and increase_fail_count

0.2.5 (2021-11-11)

• Implemented increase_base_count and increase_fail_count method. They can be used to make your code more readable when updating the telemetry

0.2.4 (2021-11-11)

• Implemented add_to method in TelemetryCounter. Can be used to add a TelemetryCounter instance to an object with a telemetry instance attached to it. This will make the code more readable.

0.2.3 (2021-11-10)

• Added attribute process_types to TelemetryCounter dataclass next to existing process_type attribute. This allows you to choose between a list of process_types or just a single process_type to be in scope of the ``TelemetryCounter``` instance

0.2.2 (2021-11-10)

• added add_mongo_telemetry decorator that uses mongo storage class

0.2.1 (2021-11-09)

• added is_telemetry_counter method in helper module
• moved add_telemetry decorator to helper module
• both methods can be directly imported from pipeline_telemetry module

0.2.0 (2021-11-09)

• added error field to TelemetryCounter so that default Errorcodes can be used in a TelemetryCounter instance to keep an error counter in the telemetry
• Updated add_telemetry_counter so that it can handle TelemetryCounter instances with either an Errorcode of a custom counter. In case of ErrorCode the actual error_code will serve as the customer counter.

0.1.0 (2021-11-05)

• added Telemetry method add_telemetry_counter

• Added TelemetryCounter class that can be used to make prefedined objects that can be added to the Telemetry instance with the add_telemetry_counter method. This will improve readibilty of your code::

  from pipeline_telemetry import TelemetryCounter

  YOUR_PREDFINED_COUNTER = ( process_type=ProcessTypes.CREATE_DATA_FROM_API, sub_process='RETRIEVE_RAW_DATA', counter_name='my_custom_counter', )

  telemetry.add_telemetry_counter(YOUR_PREDFINED_COUNTER)

0.0.4 (2021-10-31)

• Seperated the storage module
• Added MongoDb storage class
• Started with documentation

0.0.1 (2021-10-06)

• First release on PyPI.