Modeled Metrics Monitoring Library

A Python library for monitoring modeled metrics with Google Cloud Monitoring.

Overview

This library provides a Python interface for working with Google Cloud Monitoring metric descriptors and writing metrics. It queries the Google Cloud Monitoring API to retrieve metric descriptors.

Key Features

• Direct API Integration: Queries Google Cloud Monitoring API for metric descriptors
• Type Safety: Uses Google's protobuf MetricDescriptor objects
• Flexible Metric Writing: Supports all metric value types (BOOL, INT64, DOUBLE, STRING, DISTRIBUTION)
• Error Handling: Comprehensive exception handling for Google Cloud API errors

Usage

Development

# Install in development mode
pip install -e .

# Run the example
python -m modeled_metrics_monitoring.run

Building and Distribution

# Build the package
./build.sh

# Install the built package
pip install dist/*.whl

Using the Library

from modeled_metrics_monitoring import get_metric_descriptor_by_type, write_metric

# Get a metric descriptor by type
descriptor = get_metric_descriptor_by_type(
    "custom.googleapis.com/contextual-data-monitoring/modeled-metrics-ml-ops/vertex_pipeline/foot_traffic/feature_null_ratio"
)

# Write a metric
write_metric(
    descriptor,
    0.1,
    metric_labels={
        "feature_group_id": "temporal",
        "feature_group_revision": "r0_1",
        "feature_id": "is_weekend"
    }
)

# Or write a metric using the type string directly
write_metric(
    "custom.googleapis.com/contextual-data-monitoring/modeled-metrics-ml-ops/vertex_pipeline/foot_traffic/feature_null_ratio",
    0.1,
    metric_labels={
        "feature_group_id": "temporal",
        "feature_group_revision": "r0_1",
        "feature_id": "is_weekend"
    }
)

Configuration

The library can be configured using environment variables. All configuration values are defined in modeled-metrics-monitoring/src/modeled_metrics_monitoring/config.py.

Environment Variables

MONITORING_ENABLED

• Description: Whether to enable monitoring functionality.
• Type: Boolean (via environment variable)
• Default: True
• Accepted Values: 'true', '1', 'yes', 'on' (case-insensitive). Any other value disables monitoring.
• Usage: Set to False to disable all monitoring operations without modifying code.

export MONITORING_ENABLED=False

MONITORING_CHECK_WRITE_ACCESS

• Description: When enabled, monitoring is only considered active if the current principal can write metrics to Google Cloud Monitoring. When disabled, MONITORING_ENABLED alone controls whether monitoring runs; write access is not checked at import time.
• Type: Boolean (via environment variable)
• Default: False
• Accepted Values: 'true', '1', 'yes', 'on' (case-insensitive). Any other value disables the write-access check.
• Usage: Set to True in environments where you want monitoring to turn itself off if IAM does not allow metric writes (for example, local development or read-only credentials).

export MONITORING_CHECK_WRITE_ACCESS=True

MONITORING_INIT_FAIL_SHOULD_RAISE_EXCEPTION

• Description: Whether to raise an exception when monitoring initialization fails (e.g., when the principal lacks required IAM permissions).
• Type: Boolean (via environment variable)
• Default: False
• Accepted Values: 'true', '1', 'yes', 'on' (case-insensitive). Any other value disables exception raising.
• Usage: Set to True to enable strict error handling. When False, initialization failures result in warnings and monitoring is disabled gracefully.

export MONITORING_INIT_FAIL_SHOULD_RAISE_EXCEPTION=True

MONITORING_PUSH_FAIL_SHOULD_RAISE_EXCEPTION

• Description: Whether to re-raise exceptions when sending a metric to Google Cloud Monitoring fails (for example, API errors or network issues after write_metric builds the time series).
• Type: Boolean (via environment variable)
• Default: False
• Accepted Values: 'true', '1', 'yes', 'on' (case-insensitive). Any other value keeps the default behavior (log and swallow).
• Usage: Set to True so callers see push failures; when False, failures are logged at error level and the call returns without raising.

export MONITORING_PUSH_FAIL_SHOULD_RAISE_EXCEPTION=True

METRIC_WRITER_IAM_ROLE

• Description: The IAM role name required for writing monitoring metrics.
• Type: String
• Default: "roles/monitoring.metricWriter"
• Usage: Override if using a custom IAM role for metric writing permissions.

export METRIC_WRITER_IAM_ROLE=roles/monitoring.metricWriter

MONITORING_TARGET_GCP_PROJECT_ID

• Description: The Google Cloud project ID where metric descriptors are stored and where the service account has monitoring permissions.
• Type: String
• Default: "uc-contextual-data-monitoring"
• Usage: Set to the target GCP project ID where your metric descriptors are managed.

export MONITORING_TARGET_GCP_PROJECT_ID=your-project-id

Internal Configuration

METRIC_DESCRIPTOR_TYPE_PREFIX

• Description: The prefix for all metric descriptor types managed by this project.
• Type: String
• Default: "custom.googleapis.com/contextual-data-monitoring/"
• Warning: DO NOT CHANGE THIS VALUE. Changing this will cause all existing metric descriptors in Google Cloud Monitoring to become unsupported.
• Note: This is an internal constant and should not be modified.

Architecture

• Terraform: Uses YAML files from monitoring-metrics-definitions/metric-descriptors/*.yaml to create metric descriptors in Google Cloud Monitoring
• Python Library: Queries Google Cloud Monitoring API directly to retrieve metric descriptors
• Separation of Concerns: Terraform handles infrastructure (creating metric descriptors), Python library handles runtime operations (querying and writing metrics)

This approach ensures that the Python library is always working with the current state of metric descriptors in Google Cloud Monitoring, while Terraform manages the infrastructure definitions.