Measure your data pipelines with easy to use telemetry logic
Project description
======== Overview
.. start-badges
.. list-table:: :stub-columns: 1
* - docs
- |docs|
* - tests
- | |travis| |requires|
| |codecov|
* - package
- | |version| |wheel|
| |supported-versions|
| |supported-implementations|
| |commits-since|
.. |docs| image:: https://readthedocs.org/projects/pipeline-telemetry/badge/?style=flat :target: https://pipeline-telemetry.readthedocs.io/ :alt: Documentation Status
.. |travis| image:: https://api.travis-ci.com/MaartendeRuyter/pipeline-telemetry.svg?branch=master :alt: Travis-CI Build Status :target: https://travis-ci.com/github/MaartendeRuyter/pipeline-telemetry
.. |requires| image:: https://requires.io/github/MaartendeRuyter/pipeline-telemetry/requirements.svg?branch=master :alt: Requirements Status :target: https://requires.io/github/MaartendeRuyter/pipeline-telemetry/requirements/?branch=master
.. |codecov| image:: https://codecov.io/gh/MaartendeRuyter/pipeline-telemetry/branch/master/graphs/badge.svg?branch=master :alt: Coverage Status :target: https://codecov.io/github/MaartendeRuyter/pipeline-telemetry
.. |version| image:: https://img.shields.io/pypi/v/pipeline-telemetry.svg :alt: PyPI Package latest release :target: https://pypi.org/project/pipeline-telemetry
.. |wheel| image:: https://img.shields.io/pypi/wheel/pipeline-telemetry.svg :alt: PyPI Wheel :target: https://pypi.org/project/pipeline-telemetry
.. |supported-versions| image:: https://img.shields.io/pypi/pyversions/pipeline-telemetry.svg :alt: Supported versions :target: https://pypi.org/project/pipeline-telemetry
.. |supported-implementations| image:: https://img.shields.io/pypi/implementation/pipeline-telemetry.svg :alt: Supported implementations :target: https://pypi.org/project/pipeline-telemetry
.. |commits-since| image:: https://img.shields.io/github/commits-since/MaartendeRuyter/pipeline-telemetry/v0.0.1.svg :alt: Commits since latest release :target: https://github.com/MaartendeRuyter/pipeline-telemetry/compare/v0.0.1...master
.. end-badges
Create and store data pipeline telemetry data
- Free software: GNU Lesser General Public License v3 or later (LGPLv3+)
Installing pipeline-telemetry
::
pip install pipeline-telemetry
You can also install the in-development version with::
pip install https://github.com/MaartendeRuyter/pipeline-telemetry/archive/master.zip
Documentation
https://pipeline-telemetry.readthedocs.io/
Testing
To run all the tests run::
tox
Note, to combine the coverage data from all the tox environments run:
.. list-table:: :widths: 10 90 :stub-columns: 1
- - Windows
- ::
set PYTEST_ADDOPTS=--cov-append
tox
- - Other
- ::
PYTEST_ADDOPTS=--cov-append tox
Changelog
1.0.0 (2024-05-08)
- Added python 3.12 support
0.9.4 (2022-11-03)
- Added
PartialToSingleMongoAggregator
andPartialToSingleAggregator
classes to allow automated aggregation of multiple partial telemetry objects to a single telemetry object.
0.9.3 (2022-10-26)
- Added support for python 3.11
0.9.2 (2022-10-11)
- Added automatic deletion of old aggegration telemetry
0.9.1 (2022-10-04)
- Fixed defect in MongoStorage.
0.9.0 (2022-10-03)
- Added
DailyMongoAggregator
andDailyAggregator
classes to allow automated aggregation of multiple intraday telemetry objects to daily telemetry objects.
0.4.5 (2022-06-09)
- Added
set_telemetry_source_name
method toTelemetryMixin
class
0.4.4 (2022-06-08)
- Added custom hash method to
TelemetryCounter
0.4.3 (2022-06-01)
- Added
set_increment
method toTelemetryCounter
class.
0.4.2 (2022-05-17)
- Added
process_telemetry_counters_from_list
method toTelemetryMixin
class.
0.4.1 (2022-05-16)
- Added
process_telemetry_counters_from_return_value
method toTelemetryMixin
class. - Added documentation on TelemetryCounter
0.4.0 (2022-05-15)
- Added
TelemetryMixin
class to enable easy including of telemetry update methods in any datapipeline class.
0.3.1 (2022-05-13)
- Added
process_telemetry_counters_in_return_value
helper method to easily add retrieve and process TelemetryCounter objects from a return value object. - Added
process_return_value
helper method to process both ErrorCode and TelemetryCounter objects from a return value object. - Added more documentation
0.3.0 (2022-05-12)
- Added
add_errors_from_return_value
helper method to easily add the errors from aReturnValueWithStatus
object fromerror-manager
package. - Automatically initialize a missing sub_process when a telemetry counter is added to the telemetry instance.
0.2.18 (2022-03-2)
- Added
add_single_usage_telemetry
decorator to be used for single usage - telemetry objects
0.2.16 (2022-02-01)
- Added
telemetry_type
andio_time_in_seconds
attributes to Telemetry - object including a method to increase
io_time_in_seconds
.
0.2.12 (2022-01-19)
- Added
traffic_light
attribute to Telemetry object indicating the success state of the datapipeline process that is reporting on - Some minor refactoring
0.2.11 (2022-01-18)
- Added
created_at
field in mongo storage class for better date selection - of the telemetry objects
0.2.10 (2022-01-18)
- Added indexes to mongo storage class
0.2.8 (2022-01-13)
- Fix in storage classes to ensure
category
andsub_category
to be stored in the toplevel of the telemetry object
0.2.7 (2022-01-13)
- Added fields
category
andsub_category
to the telemetry object to allow for better distinction between telemetry sources - Rename telemetry field 'process_name' to 'soure_name' to be more clear about the data source in scope of the telemetry object
0.2.6 (2021-12-08)
- Added default increment value = 1 to methods
increase_base_count
andincrease_fail_count
0.2.5 (2021-11-11)
- Implemented
increase_base_count
andincrease_fail_count
method. They can be used to make your code more readable when updating the telemetry
0.2.4 (2021-11-11)
- Implemented
add_to
method inTelemetryCounter
. Can be used to add aTelemetryCounter
instance to an object with a telemetry instance attached to it. This will make the code more readable.
0.2.3 (2021-11-10)
- Added attribute
process_types
toTelemetryCounter
dataclass next to existingprocess_type
attribute. This allows you to choose between a list of process_types or just a single process_type to be in scope of the ``TelemetryCounter``` instance
0.2.2 (2021-11-10)
- added
add_mongo_telemetry
decorator that uses mongo storage class
0.2.1 (2021-11-09)
- added
is_telemetry_counter
method in helper module - moved
add_telemetry
decorator to helper module - both methods can be directly imported from
pipeline_telemetry
module
0.2.0 (2021-11-09)
- added error field to
TelemetryCounter
so that default Errorcodes can be used in aTelemetryCounter
instance to keep an error counter in the telemetry - Updated add_telemetry_counter so that it can handle TelemetryCounter instances with either an Errorcode of a custom counter. In case of ErrorCode the actual error_code will serve as the customer counter.
0.1.0 (2021-11-05)
-
added Telemetry method add_telemetry_counter
-
Added
TelemetryCounter
class that can be used to make prefedined objects that can be added to the Telemetry instance with the add_telemetry_counter method. This will improve readibilty of your code::from pipeline_telemetry import TelemetryCounter
YOUR_PREDFINED_COUNTER = ( process_type=ProcessTypes.CREATE_DATA_FROM_API, sub_process='RETRIEVE_RAW_DATA', counter_name='my_custom_counter', )
telemetry.add_telemetry_counter(YOUR_PREDFINED_COUNTER)
0.0.4 (2021-10-31)
- Seperated the storage module
- Added MongoDb storage class
- Started with documentation
0.0.1 (2021-10-06)
- First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pipeline_telemetry-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17e7e9afa38471b6a07db6c62a1ebdf3577bf3f7f7ca52cd56296675fd62e359 |
|
MD5 | 7c8a698afe0f712245b2c0d1d6364a57 |
|
BLAKE2b-256 | 6669961bb4f78417ea5c4fe2e3d56e7c730062516778b04424755399dd962867 |