Quality of Analysis for Machine Learning
Project description
QoA4ML - Quality of Analytics for ML
Language
The design of QoA4ML specification is in language
Probes
- QoA4ML Probes: libraries and lightweight modules capturing metrics. They are integrated into suitable ML serving frameworks and ML code
- Probe properties:
- Can be written in different languages (Python, GoLang)
- Can have different communications to monitoring systems (depending on probes and its ML support)
- Capture metrics with a clear definition/scope
- e.g., Response time for an ML stage (training) or a service call (of ML APIs)
- Thus output of probes must be correlated to objects to be monitored and the tenant
- Support high or low-level metrics/attributes
- depending on probes implementation
- Can be instrumented into source code or standlone
QoA4ML Handler
- The module will run on server-side that handles all comming metrices with random id generated by client application. Metrices may come from message brokers or Rest API (developing). After receiving data, the handler will generate report in the right form and submit them to Observability service. The response will be return to client with the same ID.
QoA4ML Reports
This module defines QoA_Client
, an object that will gather metrics from probes and client information, create simple reports and send it to handler
Examples
Examples are in examples.
QoA4ML Observability
The code is in observability
QoA4ML Monitor is a component monitoring qoa for a ML model which is deployed in a serving platform.
- Monitoring Service: third party monitoring service used for managing monitoring data.
- We use Prometheus and other services: provide information on how to configure them.
- QoA4MLObservabilityService: a service reads QoA4ML specifications and real time monitoring data and detect if any violation occurs
Authors
- Linh Truong
- Minh-Tri Nguyen
References
- Hong-Linh Truong, Minh-Tri Nguyen, "QoA4ML -A Framework for Supporting Contracts in Machine Learning Services", Dec 2020.
- Minh-Tri Nguyen, Hong-Linh Truong Demonstration Paper: Monitoring Machine Learning Contracts with QoA4ML, Companion of the 2021 ACM/SPEC International Conference on Performance Engineering (ICPE'21), Apr. 19-23, 2021
- https://www.researchgate.net/publication/341762862_R3E_-An_Approach_to_Robustness_Reliability_Resilience_and_Elasticity_Engineering_for_End-to-End_Machine_Learning_Systems
Change Log
0.0.13 (18/04/2022)
First Release
0.0.17 (10/05/2022)
Update system metric
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
qoa4ml-0.0.17.tar.gz
(10.9 kB
view hashes)