Quality of Analysis for Machine Learning
Project description
QoA4ML - Quality of Analytics for ML
Source code
https://github.com/rdsea/QoA4ML
Probes
- QoA4ML Probes: libraries and lightweight modules capturing metrics. They are integrated into suitable ML serving frameworks and ML code
- Probe properties:
- Can be written in different languages (Python, GoLang)
- Can have different communications to monitoring systems (depending on probes and its ML support)
- Capture metrics with a clear definition/scope
- e.g., Response time for an ML stage (training) or a service call (of ML APIs)
- Thus output of probes must be correlated to objects to be monitored and the tenant
- Support high or low-level metrics/attributes
- depending on probes implementation
- Can be instrumented into source code or standlone
Provide some metric classes for collecting different types of metric: Counter, Gauge, Summary, Histogram
Metric
: an original class providing some common functions on an metric object.- Attribute:
metric_name
description
value
- Function:
__init__
: let user define the metric name, description and default value.set
: set itsvalue
to a specific valueget_val
: get current valueget_name
: return metric nameget_des
: return metric description__str__
: return information about the metric in form of stringto_dict
: return information about the metric in form of dictionary
- Attribute:
Counter
- Attribute: same as
Metric
& on further developing - Function:
inc
: increase the value of the metric by the given number/by 1 by default.reset
: set the value back to zero.
- Attribute: same as
Gauge
- Attribute: same as
Metric
& on further developing - Function:
inc
: increase the value of the metric by a given number/by 1 by default.dec
: decrease the value of the metric by a given number/by 1 by default.set
: set the value to a given number.
- Attribute: same as
Summary
- Attribute: same as
Metric
& on further developing - Function:
inc
: increase the value of the metric by a given number/by 1 by default.dec
: decrease the value of the metric by a given number/by 1 by default.set
: set the value to a given number.
- Attribute: same as
Histogram
- Attribute: same as
Metric
& on further developing - Function:
inc
: increase the value of the metric by a given number/by 1 by default.dec
: decrease the value of the metric by a given number/by 1 by default.set
: set the value to a given number.
- Attribute: same as
QoA4ML Reports
This module defines QoA_Client
, an object tha, provide functions to get/set metric values, create report one by one or all in one, etc
- Attribute:
config
: store the client configuration as a dictionarymetrics
: a dictionary of all metrics that client monitoringconnector
: a dictionary of connectors to send out monitoring data
- Function:
__init__
: init the client from user configuration.init_connector
: init the connnectors before sending monitoring data.init_metric
: create metric instances for the list of metrics it monitors.add_metric
: add metric to the clientget
: get the client configurationget_metric
: return all metrics as a dictionaryset
: set the configuration.generate_report
: generate report for given one/list/all metrics.asyn_report
: send out report in another threadreport
: start new thread to sendasyn_report
__str__
: return the client configuration and connector as string.
Examples
https://github.com/rdsea/QoA4ML/tree/main/example
Overview
Probes will be integrated to client program or system service to collect metrics at the edge Probes will generate reports and sent to message broker using different connector. Coresponding collector should be used to acquire the metrics.
Collector
The manager/orchestrator have to integrate collector to collect metric using different protocols for further analysis.
- Attribute:
- Function:
-
__init__
: take a configuration as adict
containing information about the data source, eg. broker, channel, queue, etc. It can take anobject
as an attributehost
to return the message for further processing. -
If the collector is initiated by an object inherited class, this class must implement
message_processing
function to process the message returned by the collector. Otherwise, the collector will print the message to the console. -
on_request
: handle message from data source (message broker,...) -
start
&stop
: start and stop consuming message -
get_queue
: return the queue name.
-
Connector
Connectors are implement with different protocols for sending report. Example: sending report to message broker - AMQP/MQTT
- Attribute:
- Function:
-
__init__
: take a configuration as adict
containing information about the data sink, eg. broker, channel, queue, etc. It can take abool
parameterlog
for logging messages for further processing. -
send_data
: a function to send data to specifiedrouting_key
/queue
with a corresponding keycorr_id
to trace back message.
-
Utilities
A module provide some frequently used functions and some function to directly collect system metrics.
Change Log
0.0.13 (18/04/2022)
First Release
0.0.18 (10/05/2022)
Update system metric
0.0.19 (31/05/2022)
Update process metric
0.0.54 (20/09/2022)
Update monitoring system/process/docker
0.0.62 (20/09/2022)
Add metric and modify format of system/process reports
0.0.64 (15/03/2022)
Add metric and modify format of system/process reports
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.