qoa4ml 0.0.15

Quality of Analysis for Machine Learning

QoA4ML - Quality of Analytics for ML

Language

The design of QoA4ML specification is in language

Probes

• QoA4ML Probes: libraries and lightweight modules capturing metrics. They are integrated into suitable ML serving frameworks and ML code
• Probe properties:
  • Can be written in different languages (Python, GoLang)
  • Can have different communications to monitoring systems (depending on probes and its ML support)
  • Capture metrics with a clear definition/scope
    • e.g., Response time for an ML stage (training) or a service call (of ML APIs)
    • Thus output of probes must be correlated to objects to be monitored and the tenant
  • Support high or low-level metrics/attributes
    • depending on probes implementation
  • Can be instrumented into source code or standlone

QoA4ML Handler

• The module will run on server-side that handles all comming metrices with random id generated by client application. Metrices may come from message brokers or Rest API (developing). After receiving data, the handler will generate report in the right form and submit them to Observability service. The response will be return to client with the same ID.

QoA4ML Reports

This module defines QoA_Client, an object that will gather metrics from probes and client information, create simple reports and send it to handler

Examples

Examples are in examples.

QoA4ML Observability

The code is in observability

QoA4ML Monitor is a component monitoring qoa for a ML model which is deployed in a serving platform.

• Monitoring Service: third party monitoring service used for managing monitoring data.
  • We use Prometheus and other services: provide information on how to configure them.
• QoA4MLObservabilityService: a service reads QoA4ML specifications and real time monitoring data and detect if any violation occurs

Authors

• Linh Truong
• Minh-Tri Nguyen

References

• Hong-Linh Truong, Minh-Tri Nguyen, "QoA4ML -A Framework for Supporting Contracts in Machine Learning Services", Dec 2020.
• Minh-Tri Nguyen, Hong-Linh Truong Demonstration Paper: Monitoring Machine Learning Contracts with QoA4ML, Companion of the 2021 ACM/SPEC International Conference on Performance Engineering (ICPE'21), Apr. 19-23, 2021
• https://www.researchgate.net/publication/341762862_R3E_-An_Approach_to_Robustness_Reliability_Resilience_and_Elasticity_Engineering_for_End-to-End_Machine_Learning_Systems

Change Log

0.0.13 (18/04/2022)

First Release

0.0.15 (10/05/2022)

Update system metric