Unified OpenTelemetry bootstrap for internal Python services. The objective is to provide every service with a consistent operational view without requiring teams to build dashboards from scratch.

Usage

from fastapi import FastAPI
from aice_observability import Observability

app = FastAPI()

obs = (
    Observability()
    .service("data-quality-service", version="2.0.0")
    .start()                        # before the library instrument_*() calls
    .instrument_fastapi(app)
    .instrument_sqlalchemy(engine)
)

.start() must be called before the library integrations; they raise if it has not run. Each one skips quietly when its library is not installed:

Method	Covers
instrument_redis()	redis-py
instrument_sqlalchemy(engine)	one SQLAlchemy engine
instrument_psycopg()	Psycopg 3, psycopg_pool included
instrument_psycopg2()	Psycopg 2
instrument_asyncpg()	asyncpg pools and connections
instrument_botocore()	boto3 / botocore: S3 and S3-compatible stores (MinIO)
instrument_celery()	Celery producers and workers; call it in the publishing process and in each worker child (worker_process_init)
instrument_django()	Django

Database integrations emit spans, not metrics; the collector's spanmetrics connector derives the database dashboard's series from them.

instrument_fastapi() and instrument_flask() do not need .start(), but they must run before the app serves its first request. Starlette builds its middleware stack on the first ASGI call, and the lifespan startup event is that call, so instrumenting FastAPI from inside a lifespan silently records nothing. If your service starts telemetry in its lifespan, instrument the app at import and start telemetry later; the instrumentor binds to OpenTelemetry's global proxy providers, which forward to the real ones once .start() installs them:

from aice_observability.frameworks.fastapi import instrument_fastapi

app = FastAPI(lifespan=lifespan)   # lifespan calls Observability()...start()
instrument_fastapi(app)

httpx is instrumented automatically inside .start(), since they need no app-specific object.

Traces across your own queues

Celery carries the trace in its message headers. A job stored in a Postgres table, a pgmq payload or any other hand-rolled queue does not, so the worker starts a new trace unless you carry it:

from aice_observability.propagation import continue_trace, inject_context

# producer, inside the request's span: a dict of W3C headers (IDs only)
enqueue({"document_id": doc_id, "trace": inject_context()})

# consumer
with continue_trace(job.get("trace"), "ingest.document"):
    process(job)

A missing or malformed carrier starts a new trace instead of failing the job.

Service name

OTEL_SERVICE_NAME, when set, wins over .service(name), so one image run as several processes (an API and its worker) reports each under its own name.

Configuration (Environment Variables)

Variable	Required	Default	Description
OTEL_EXPORTER_OTLP_ENDPOINT	Yes, unless all telemetry is disabled	none	OTLP HTTP endpoint used to export traces, metrics, and logs. Examples: http://192.168.100.254:4318 (Collector), https://otlp-gateway-prod-us-central-0.grafana.net/otlp (Grafana Cloud).
OTEL_EXPORTER_OTLP_HEADERS	No	none	Optional authentication headers in key=value,key=value format. Required by many hosted OTLP providers.
OTEL_SERVICE_NAME	Yes	none	Logical service name reported to the observability backend (for example dq-service).
OTEL_ENVIRONMENT	No	development	Deployment environment such as development, staging, or production.
OTEL_TRACES_ENABLED	No	true	Enable or disable distributed tracing.
OTEL_METRICS_ENABLED	No	true	Enable or disable metrics collection.
OTEL_LOGS_ENABLED	No	true	Enable or disable log export.
OTEL_LOG_LEVEL	No	INFO	Minimum application log level captured by the logging pipeline.

Example: Self-hosted OpenTelemetry Collector

OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.100.254:4318
OTEL_SERVICE_NAME=dq-service
OTEL_ENVIRONMENT=development
OTEL_TRACES_ENABLED=true
OTEL_METRICS_ENABLED=true
OTEL_LOGS_ENABLED=true

Standard Observability Dashboards

The dashboards are designed around operational questions rather than telemetry types. Instead of exposing every metric collected by OpenTelemetry, each dashboard focuses on answering a specific set of questions that engineers, tech leads, and platform teams routinely ask.

Version 1 of the observability platform includes the following dashboards:

1. Service Overview
2. Infrastructure
3. HTTP/API
4. Database (conditional)
5. External Services (HTTP Client)
6. Logs
7. Traces

---

1. Service Overview Dashboard

Purpose

The Service Overview dashboard is the primary landing page for every service.

It provides a high level summary of service health, traffic, latency, failures, and operational status.

This dashboard should answer the following questions within a few seconds:

• Is the service healthy?
• Is it currently receiving traffic?
• Has latency increased?
• Are users experiencing failures?
• Should I investigate further?

Target Audience

• Tech Leads
• Backend Engineers
• DevOps Engineers
• SREs
• Engineering Managers

Panels

Availability

Displays service uptime and overall availability.

Purpose:

Quickly determine whether the service is operational.

---

Request Rate (Requests Per Second)

Displays incoming request volume.

Purpose:

Identify traffic spikes, traffic drops, and overall load.

---

Active Requests

Shows the number of requests currently being processed.

Purpose:

Detect request buildup and possible bottlenecks.

---

Error Rate

Percentage and total number of failed requests.

Purpose:

Quickly identify service degradation.

---

Average Response Time

Average latency across all requests.

Purpose:

Monitor overall responsiveness.

---

P95 Latency

95% of requests complete faster than this value.

Purpose:

Represents the experience of almost all users.

---

P99 Latency

99% of requests complete faster than this value.

Purpose:

Highlights long-tail latency issues.

---

HTTP Status Code Distribution

Breakdown of

• 2xx
• 3xx
• 4xx
• 5xx

Purpose:

Identify client errors versus server failures.

---

Top Failing Endpoints

Endpoints with the highest error count.

Purpose:

Quickly identify problematic APIs.

---

Slowest Endpoints

Endpoints ranked by average latency.

Purpose:

Locate performance bottlenecks.

---

Service Metadata

Displays

• Service Name
• Version
• Environment
• Instance Count
• Last Restart Time

Purpose:

Provide deployment context during incident investigations.

---

Quick Navigation

Links directly to

• Logs Dashboard
• Traces Dashboard
• Infrastructure Dashboard

Purpose:

Reduce investigation time.

---

2. Infrastructure Dashboard

Purpose

The Infrastructure dashboard monitors resource utilization of the application process and host.

It helps determine whether performance issues originate from application logic or infrastructure limitations.

Target Audience

• Platform Engineers
• DevOps Engineers
• SREs

Panels

CPU Usage

CPU utilization over time.

---

Memory Usage

Current memory consumption.

---

Memory Utilization

Memory usage as a percentage of available memory.

---

Disk Usage

Disk space utilization.

---

Disk I/O

Read and write throughput.

---

Network Throughput

Incoming and outgoing traffic.

---

File Descriptors

Number of open file descriptors.

---

Thread Count

Active application threads.

---

Garbage Collection Activity

Displays

• Collection frequency
• Objects collected
• GC pause duration

Purpose:

Identify memory pressure.

---

Process Restarts

Detect unexpected application restarts.

---

3. HTTP/API Dashboard

Purpose

Provides endpoint-level visibility into API behaviour.

Rather than viewing the service as a whole, this dashboard focuses on individual routes.

Target Audience

• Backend Engineers
• API Owners

Panels

Requests Per Second by Endpoint

---

Average Latency by Endpoint

---

P95 Latency by Endpoint

---

P99 Latency by Endpoint

---

Error Rate by Endpoint

---

Slowest Endpoints

---

Largest Responses

---

Largest Requests

---

Request Duration Heatmap

Visualizes latency distribution across all requests.

---

Endpoint Ranking

Ranks endpoints by

• Traffic
• Errors
• Latency

---

4. Database Dashboard

Purpose

Provides visibility into database interactions.

This dashboard is displayed only when database instrumentation is enabled.

Supported instrumentations include

• SQLAlchemy
• psycopg2

Target Audience

• Backend Engineers
• Database Administrators

Panels

Queries Per Second

---

Average Query Duration

---

Slowest Queries

---

Database Error Rate

---

Connection Pool Utilization

---

Connection Pool Wait Time

---

Pool Saturation

---

Failed Queries

---

Transaction Duration

---

Database Retries

---

Conditional Display

Visible only if one of the following instrumentations is enabled.

• SQLAlchemy
• psycopg2

---

5. External Services Dashboard

Purpose

Provides visibility into outbound HTTP requests.

This dashboard helps identify slow or failing third-party services.

The dashboard is powered by HTTPX instrumentation.

Target Audience

• Backend Engineers
• Tech Leads

Panels

Outbound Requests Per Second

---

Average Outbound Latency

---

P95 Outbound Latency

---

Error Rate by Remote Service

---

Timeout Count

---

Retry Count

---

Top Slowest Remote Hosts

---

Request Distribution by Host

---

External Dependency Health

Summarizes

• Success Rate
• Failure Rate
• Average Latency

---

6. Logs Dashboard

Purpose

Provides centralized log exploration through Grafana and Loki.

The dashboard is intended for investigation after metrics indicate abnormal behaviour.

Target Audience

• All Engineers

Panels

Live Logs

---

Error Logs

---

Warning Logs

---

Log Volume Over Time

---

Log Levels

Distribution of

• DEBUG
• INFO
• WARNING
• ERROR
• CRITICAL

---

Service Filter

---

Search

Supports free text search.

---

Correlation Links

Jump directly to related traces.

---

7. Traces Dashboard

Purpose

Provides distributed tracing through Jaeger.

The dashboard allows engineers to follow requests across services.

Target Audience

• Backend Engineers
• Platform Engineers
• SREs

Panels

Recent Traces

---

Slowest Traces

---

Error Traces

---

Trace Duration Distribution

---

Service Dependency Graph

---

Span Timeline

---

Failed Spans

---

Trace Search

Supports filtering by

• Service
• Endpoint
• Duration
• Status

---

Correlated Logs

Jump directly to matching log entries.

---

Dashboard Design Principles

Every dashboard should follow these principles.

Minimal by Default

Dashboards should expose only the information required for operational decisions.

Avoid displaying every metric collected by OpenTelemetry.

---

Progressive Investigation

Dashboards should naturally lead the user from high level health to detailed diagnostics.

Typical investigation flow:

Service Overview

↓

HTTP/API

↓

Database or External Services

↓

Logs

↓

Traces

---

Consistency

All services should expose the same dashboard structure.

This reduces cognitive load when switching between projects.

---

Conditional Sections

Dashboards should only display panels relevant to the service.

Examples:

A service without Redis should not display Redis metrics.

A service without SQLAlchemy should not display SQL metrics.

A service without HTTP clients should not display outbound request metrics.

---

Standard Labels

All dashboards should support filtering using common resource attributes.

Examples include:

• Service Name
• Environment
• Version
• Instance
• Endpoint
• HTTP Method
• Database System
• Remote Host

---

Future Dashboard Roadmap

The following dashboards are planned for future releases.

• Redis Dashboard
• Background Jobs Dashboard
• AI/LLM Dashboard
• Cache Dashboard
• Storage Dashboard
• Security Dashboard
• Business Metrics Dashboard