observability-python

Repository: from021-io/observability-python. Python distribution: 021-python-observability; import: observability_021. Tags up to v0.2.0 use the previous distribution name, 021-observability.

Common API observability for Python 3.12–3.14: FastAPI request metrics and traces, JSON stdout logs, explicit HTTPX/SQLAlchemy instrumentation, process signals, health/readiness and domain operation metrics. Uses OpenTelemetry over HTTP/protobuf.

The package is under development. Consumer migrations are being prepared; live Vercel/Grafana verification follows deployment. No paid registry is required to build it.

Versions

Every merge to main updates a release PR, chore(release): X.Y.Z, built by release-please from the PR titles (Conventional Commits). Merging it bumps the version, writes CHANGELOG.md and tags vX.Y.Z. Consumers depend on a tag:

[tool.uv.sources]
021-python-observability = { git = "https://github.com/from021-io/observability-python.git", tag = "v0.3.0" } # x-release-please-version

Once releases are published to PyPI, pin the exact version instead; no Git credential is needed to install it:

uv add "021-python-observability==0.3.0" # x-release-please-version

Publishing to PyPI

.github/workflows/release.yml publishes a release with PyPI Trusted Publishing (OIDC, no API token). It stays off until a maintainer sets the repository variable PYPI_PUBLISH to true; then every release tag is built, checked with twine check --strict, smoke-tested and uploaded through the pypi environment. A manual run of the workflow publishes an existing tag, to TestPyPI (testpypi environment) by default. The build refuses a tag without a LICENSE file or whose pyproject.toml version differs from the tag.

Install locally

uv sync --all-extras
uv build
# In a consumer's environment, use the actual wheel path:
python -m pip install /path/to/021_python_observability-0.2.0-py3-none-any.whl

The vercel extra supplies the Vercel Python SDK. Install that extra in Vercel consumers; normal Python servers do not need it. Selecting Git, a package index or a wheel URL later does not change this integration API.

Configure the application once

from fastapi import FastAPI
from observability_021 import setup_observability

app = FastAPI()  # Existing routers, exception handlers, auth and lifespan remain here.
observability = setup_observability(app, service_name="021-api")

@app.get("/items/{item_id}")
async def get_item(item_id: str):
    return {"id": item_id}

Call setup while constructing the app, before serving requests. The same app object is returned to the server: Vercel still sees a FastAPI application. Setup returns the shared observability object, also available at app.state.observability. Repeated setup on the same app returns that object without stacking providers, middleware or log handlers.

The integration wraps FastAPI's fully built ASGI middleware stack, including its error handler. It observes the response actually sent, including early auth/rate-limit rejections and transformed errors. It does not decorate individual handlers. Existing lifespan startup/state/shutdown is preserved; providers flush and stop after application teardown.

Where telemetry goes

Example deployment variables (URLs and credentials below are placeholders):

OTEL_SERVICE_NAME=021-api
OTEL_DEPLOYMENT_ENVIRONMENT=staging
OTEL_SERVICE_VERSION=0.1.0
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20REPLACE_WITH_CREDENTIAL

Use the ingestion endpoint of the OTLP gateway/collector, not the Grafana dashboard URL. With that base, traces go to /otlp/v1/traces and metrics to /otlp/v1/metrics. The header value follows the OTLP comma-separated key=value format with percent encoding. Put real credentials in deployment secrets, never in the source repository or dependency URL.

Both exporters serialize binary protobuf and default to Content-Type: application/x-protobuf. Custom headers can override that header; keep its default value. A different Content-Type does not change the encoding. Use custom headers for gateway authentication and verify receipt of both signals against the configured gateway during rollout.

Separate destinations are supported and take precedence over the shared base:

OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://metrics.example.com/v1/metrics
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://traces.example.com/v1/traces
OTEL_EXPORTER_OTLP_METRICS_HEADERS=Authorization=Bearer%20METRICS_CREDENTIAL
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer%20TRACES_CREDENTIAL

Signal-specific endpoints are used exactly as provided, including their paths. Omit all endpoints for local development without remote export. Logs remain JSON on stdout in every environment, so Vercel collects them; the package does not export logs over OTLP.

A service instance gets a unique service.instance.id. Code can alternatively pass an explicit Settings object; that object does not implicitly load environment variables.

In persistent processes, SDK batching/periodic export and normal shutdown deliver signals. On Vercel, the request span closes before a post-response wait_until flush is registered. Flushes run off the event loop; failures produce sanitized diagnostics and do not change business responses. The async wait budget does not kill Python worker threads: the HTTP exporter also has its own transport timeout. Forced process termination can still lose telemetry. Live delivery on Vercel remains a deployment validation, not a unit-test claim.

Dependencies and business operations

HTTPX clients opt in at construction, using a fixed dependency name. Use the sync equivalent observability.httpx_transport for a synchronous httpx.Client.

import httpx

client = httpx.AsyncClient(
    transport=observability.async_httpx_transport(dependency="company-brain"),
    base_url=brain_url,
)
# Close the client using the service's existing lifespan.
observability.instrument_sqlalchemy(engine, dependency="database")

Pass an existing HTTPX transport through transport= if it carries custom TLS, proxy, retry or test behavior. Explicit per-host mounts also need their own observed transport. SQLAlchemy supports both Engine and AsyncEngine. Neither adapter records URLs, hostnames, SQL statements, SQL parameters, exception messages or response bodies. Client spans inject W3C trace context; HTTP requests also propagate the validated request ID.

Readiness checks remain owned by each service:

from sqlalchemy import text

async def database_ready():
    async with engine.connect() as connection:
        await connection.execute(text("SELECT 1"))

observability.add_health_routes(app, readiness={"database": database_ready})

This explicitly registers /health (process alive) and /ready (all declared checks succeed). Checks must be async and cancellation-cooperative; False means unavailable, None/other successful returns mean available. A failure or timeout returns sanitized 503. Existing routes are never silently replaced. Migrate monitors and auth exemptions first. With no declared checks, readiness represents the process only. The JSON contract is status: ok|unavailable plus checks, not status: ready; see the exact probe contract.

Generic domain instrumentation is explicit and works around synchronous code or await:

with observability.operation("brain.ingest"):
    await ingestion_service.run()

observability.increment("app.documents.count", operation="ingest", status="ok")
observability.observe("app.ingest.duration", 12.5, operation="ingest", status="ok")

Use static metric/operation/dependency names from code, never IDs or user-provided text. Operation/result remain separate dimensions. operation() reports ok, error or cancelled; explicit helpers also preserve static domain outcomes.

Logging

get_logger(__name__) provides structlog logging. Use static event names and approved structured fields (operation, op, action, dependency, status, reason, error_type). Do not interpolate messages or pass business payloads into approved fields. Unknown extra fields and raw exception objects/messages are not serialized. Foreign stdlib messages are replaced with a safe event and logger name; exporter failures get telemetry.export_failed.

Logs also retain UUID correlation fields: brain_id, job_id, file_id, file_version_id, entity_id, event_id and correlation_id. Pass a UUID object or UUID string; valid values are normalized and included in JSON and searchable message. Invalid values and unknown fields are omitted. These fields are never metric labels.

Use logger.exception("ingestion.failed", brain_id=brain_id, job_id=job_id) inside an exception handler, or logger.error(..., exc_info=error) for a captured exception. Stdlib logging.exception / logging.error(..., exc_info=True, extra={...}) and Uvicorn errors use the same formatter. The actual exception supplies error_type and an exception array, ordered from the current error through its cause/context chain:

{"type":"ValueError","frames":[{"file":"worker.py","function":"ingest","line":42}],"frames_truncated":false}

Each entry contains only the type and frame locations, never exception messages, source lines, locals or absolute directories. At most 8 chained exceptions and the 32 frames nearest each failure are retained; exception_truncated and each entry's frames_truncated flag indicate omitted diagnostics. Explicit raise ... from None suppresses the previous context. Exception-group members are not expanded; their group type and its own frames are retained. Type/function names are capped at 128 characters and filenames at 256. Diagnostic fields are generated from the exception, not accepted as caller-supplied JSON. Unhandled ASGI failures include these diagnostics in the single request.completed log; response codes, exception propagation and span data are unchanged.

Each HTTP request emits one request.completed line with service identity, a validated UUID request ID, trace/span IDs, method, route template, status and duration. Invalid inbound request IDs are replaced. IDs appear in logs, not metric labels. Setup owns root logging configuration: remove the consumer's previous formatter/configuration during migration.

Contract and checks

See CONTRACT.md for signal names, units, buckets and measurement boundaries.

make check
uv build

Tests exercise actual ASGI responses, a local OTLP HTTP/protobuf receiver, a distributed API-to-API call, SQLAlchemy sync/async operations, exporter failures and package installation. No test needs Grafana credentials or calls a deployed service.

Existing domain ports and scrape endpoints

For consumers retaining /metrics, install [prometheus], pass prometheus=True at setup and return obs.prometheus_latest() from the endpoint. Metrics are read from the same OTel provider used by OTLP. Existing dashboards still need the explicit name/label migration in CONTRACT.md; the reader does not recreate legacy instrument names.

Domain helpers accept static attributes from the documented allowlist. Scalar histograms can use unit="1" (or "USD" for already-existing cost observations); duration histograms stay in seconds. This preserves consumer domain ports without reimplementing exporters or instrument caches.

Grafana synchronization tool

tools/grafana/sync.py is a standalone Python 3.12–3.14 CLI using only the standard library. Consumers check it out at a release tag (ref: vX.Y.Z), the same tag their runtime dependency pins. The package does not import or run it. Services own dashboard JSON, queries, inventories, secrets and workflow triggers; this repository owns the synchronization mechanism and tests.

Pass --manifest /path/to/service/grafana/dashboards.json: a nonempty JSON array of paths relative to that manifest, for example ["overview.json", "usage/funnel.json"]. Paths must stay inside its directory, refer to JSON files and have unique dashboard UIDs. No automatic file discovery or remote dashboard deletion is performed.

python3 tools/grafana/sync.py --manifest /path/to/service/grafana/dashboards.json --validate
python3 -m unittest discover -s tools/grafana -p 'test_sync.py' -v
# With the service's Actions configuration, authenticated GETs only:
python3 tools/grafana/sync.py --manifest /path/to/service/grafana/dashboards.json --dry-run

Publication requires GRAFANA_URL, GRAFANA_SERVICE_ACCOUNT_TOKEN and GRAFANA_COMMIT (the consumer checkout SHA). Grafana Cloud additionally needs GRAFANA_NAMESPACE=stacks-<instance ID>; OSS defaults to default. GRAFANA_FOLDER_UID is required for creation; existing folders are preserved. GRAFANA_DATASOURCES is an optional JSON mapping of variable names to datasource UIDs, required when a new datasource variable has no saved selection. Keep these values in consumer deployment configuration. The tool preserves saved variable selections, refresh and time range; preflights the entire inventory before writing; rejects redirects; sanitizes errors; and verifies every write by reading it back. --dry-run reports preflight_verified and never checks write permission or panel queries.

The existing make check and package CI matrix run the common tests. Consumer workflows validate their own manifests and fetch the same reviewed tool revision using their existing read-only OBSERVABILITY_GITHUB_TOKEN, with persist-credentials: false. Move the CI tool pin to a new tag through a PR after its checks pass. A tool-only release does not require moving the runtime dependency, but keeping both on the same tag is the default.

Consumer rollout checklist

• Moving from 021-observability to 021-python-observability: rename the dependency (with its extras) in [project] dependencies, the [tool.uv.sources] key or the requirements.txt line, and regenerate the lock. Imports (observability_021) and every signal name stay the same.
• Merge the release PR that release-please opens (chore(release): X.Y.Z), then pin both consumers to its tag vX.Y.Z (tag = "vX.Y.Z" in [tool.uv.sources], ref: vX.Y.Z for the Grafana tool); regenerate API lock/export and rerun both consumer gates.
• Update readiness JSON assertions and dependency monitors together. In Brain, old dependency checks moved from /health to /ready; liveness alone cannot prove readiness.
• Migrate existing metric queries, labels, duration units and alert thresholds together; if none exist, record that inventory explicitly before rollout.
• Replace log searches by email/raw path with the response's UUID X-Request-ID and trace_id. Confirm callers accept a replacement ID when they send a non-UUID value.
• Retain the vercel extra in Vercel consumers: wait_until keeps export work running after the response. While consuming from Git, configure the package PAT separately for each deployment environment; a PyPI pin needs none.

License

Apache License 2.0. See LICENSE.