Measure your data pipelines with easy to use telemetry logic
Project description
Pipeline Telemetry
Pipeline Telemetry makes it easy for project to generate store telemetry data from within your data pipelines.
from counters import INVALID_DATA, PROCESSED_DATA_POINTS
def my_data_pipeline():
telemetry = Telemetry(**TELEMETRY_PARAMS)
from url in data_urls:
data = get_data_from_url(url)
telemetry.add_telemetry_counter(telemetry_counter=URL_RETRIEVALS) # increase counter for retrieved URLS
if data.invalid:
telemetry.add_telemetry_counter(telemetry_counter=INVALID_DATA) # increase counter for failed retrievals
continue
processed_data_points = process_and_store_data(data)
telemetry.add_telemetry_counter( # increase counter number of datapoints retrieved
telemetry_counter=PROCESSED_DATA_POINTS,
increment=len(processed_data_points))
telemetry.save_and_close()
In this example data is retrieved for a list of urls. Total retrievals, insuccesfull retrievals as well as number of processed datapoints are stored in the telemetry object. This allows you to closely monitor the quality and behavior of your datapipeline. For example all retrievals might be OK but if the number of processed datapoints suddenly drops there might be an issue with some the endpoints.
Changelog
1.0.0 (2024-05-08)
- Added python 3.12 support
0.9.4 (2022-11-03)
- Added
PartialToSingleMongoAggregator
andPartialToSingleAggregator
classes to allow automated aggregation of multiple partial telemetry objects to a single telemetry object.
0.9.3 (2022-10-26)
- Added support for python 3.11
0.9.2 (2022-10-11)
- Added automatic deletion of old aggegration telemetry
0.9.1 (2022-10-04)
- Fixed defect in MongoStorage.
0.9.0 (2022-10-03)
- Added
DailyMongoAggregator
andDailyAggregator
classes to allow automated aggregation of multiple intraday telemetry objects to daily telemetry objects.
0.4.5 (2022-06-09)
- Added
set_telemetry_source_name
method toTelemetryMixin
class
0.4.4 (2022-06-08)
- Added custom hash method to
TelemetryCounter
0.4.3 (2022-06-01)
- Added
set_increment
method toTelemetryCounter
class.
0.4.2 (2022-05-17)
- Added
process_telemetry_counters_from_list
method toTelemetryMixin
class.
0.4.1 (2022-05-16)
- Added
process_telemetry_counters_from_return_value
method toTelemetryMixin
class. - Added documentation on TelemetryCounter
0.4.0 (2022-05-15)
- Added
TelemetryMixin
class to enable easy including of telemetry update methods in any datapipeline class.
0.3.1 (2022-05-13)
- Added
process_telemetry_counters_in_return_value
helper method to easily add retrieve and process TelemetryCounter objects from a return value object. - Added
process_return_value
helper method to process both ErrorCode and TelemetryCounter objects from a return value object. - Added more documentation
0.3.0 (2022-05-12)
- Added
add_errors_from_return_value
helper method to easily add the errors from aReturnValueWithStatus
object fromerror-manager
package. - Automatically initialize a missing sub_process when a telemetry counter is added to the telemetry instance.
0.2.18 (2022-03-2)
- Added
add_single_usage_telemetry
decorator to be used for single usage - telemetry objects
0.2.16 (2022-02-01)
- Added
telemetry_type
andio_time_in_seconds
attributes to Telemetry - object including a method to increase
io_time_in_seconds
.
0.2.12 (2022-01-19)
- Added
traffic_light
attribute to Telemetry object indicating the success state of the datapipeline process that is reporting on - Some minor refactoring
0.2.11 (2022-01-18)
- Added
created_at
field in mongo storage class for better date selection - of the telemetry objects
0.2.10 (2022-01-18)
- Added indexes to mongo storage class
0.2.8 (2022-01-13)
- Fix in storage classes to ensure
category
andsub_category
to be stored in the toplevel of the telemetry object
0.2.7 (2022-01-13)
- Added fields
category
andsub_category
to the telemetry object to allow for better distinction between telemetry sources - Rename telemetry field 'process_name' to 'soure_name' to be more clear about the data source in scope of the telemetry object
0.2.6 (2021-12-08)
- Added default increment value = 1 to methods
increase_base_count
andincrease_fail_count
0.2.5 (2021-11-11)
- Implemented
increase_base_count
andincrease_fail_count
method. They can be used to make your code more readable when updating the telemetry
0.2.4 (2021-11-11)
- Implemented
add_to
method inTelemetryCounter
. Can be used to add aTelemetryCounter
instance to an object with a telemetry instance attached to it. This will make the code more readable.
0.2.3 (2021-11-10)
- Added attribute
process_types
toTelemetryCounter
dataclass next to existingprocess_type
attribute. This allows you to choose between a list of process_types or just a single process_type to be in scope of the ``TelemetryCounter``` instance
0.2.2 (2021-11-10)
- added
add_mongo_telemetry
decorator that uses mongo storage class
0.2.1 (2021-11-09)
- added
is_telemetry_counter
method in helper module - moved
add_telemetry
decorator to helper module - both methods can be directly imported from
pipeline_telemetry
module
0.2.0 (2021-11-09)
- added error field to
TelemetryCounter
so that default Errorcodes can be used in aTelemetryCounter
instance to keep an error counter in the telemetry - Updated add_telemetry_counter so that it can handle TelemetryCounter instances with either an Errorcode of a custom counter. In case of ErrorCode the actual error_code will serve as the customer counter.
0.1.0 (2021-11-05)
-
added Telemetry method add_telemetry_counter
-
Added
TelemetryCounter
class that can be used to make prefedined objects that can be added to the Telemetry instance with the add_telemetry_counter method. This will improve readibilty of your code::from pipeline_telemetry import TelemetryCounter
YOUR_PREDFINED_COUNTER = ( process_type=ProcessTypes.CREATE_DATA_FROM_API, sub_process='RETRIEVE_RAW_DATA', counter_name='my_custom_counter', )
telemetry.add_telemetry_counter(YOUR_PREDFINED_COUNTER)
0.0.4 (2021-10-31)
- Seperated the storage module
- Added MongoDb storage class
- Started with documentation
0.0.1 (2021-10-06)
- First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pipeline_telemetry-1.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b528a10e814726725b7c34326dcaa07516fc792315ec39f09cc92dda20b16e43 |
|
MD5 | 77363a609b50dc900d5ca27a4779de8b |
|
BLAKE2b-256 | 347f118a20a0dd66986bf8b9e803312d6e3a18777d6637995c3fa39f7c8433ea |