polars-telemetry

OpenTelemetry instrumentation for Polars query execution.

One span per query carrying the plan, and per-node counters as metrics, to any OTLP collector — or a profile file you open in your browser, with no collector at all.

Install

pip install polars-telemetry          # API only; bring your own OTel SDK
pip install 'polars-telemetry[otlp]'  # with SDK and OTLP exporter

Python 3.10+.

Use

import polars_telemetry

polars_telemetry.install()

install() enables polars' query monitoring, which sets the engine affinity to "streaming" and therefore changes how your queries execute — so it never happens on import. uninstall() reverses it.

What you get

A polars.collect span per query, on whatever trace context was active:

• the plan — scan sources, pushed-down predicates, join types and keys, group-by keys
• polars.cpu_ms, polars.parallelism, result rows
• the hottest node and its share of total CPU
• diagnostics — parallel efficiency, filter selectivity, join amplification, projection efficiency, morsel skew, predicate pushdown, row-group skipping

Per-node counters — rows, morsels, polls, work-stealing, poll latency, state updates, IO time and bytes — as 15 metric instruments dimensioned by node kind.

Every name is listed in the attribute reference.

Profiles without a collector

from polars_telemetry.export.file import FileExporter

polars_telemetry.install(exporter=FileExporter("profiles/session.jsonl"))

One self-contained JSON document per query: both plans with every node property, all 19 per-node counters, the diagnostics, and a fingerprint of the plan shape.

Drop the file on the profile viewer to read both plans, per-node counters, and a diff between two runs of the same shape. It runs entirely in your browser; nothing is uploaded.

Configure

from polars_telemetry import Config

polars_telemetry.install(Config(node_metrics=False))

Your data

Spans carry plan detail: scan paths, column names, join keys and literal predicate values — col("email") == "..." arrives verbatim, because knowing which predicate was slow is usually the point.

• Config(redact_literals=True) masks literal values.
• Literals are never used as metric attributes, at any setting.
• Attributes that can carry user data are listed in polars_telemetry.export.semconv.CARRIES_USER_DATA.

Polars Cloud

If polars-cloud is installed, its observer is wrapped and forwarded to rather than replaced. Both work at once.

Links

• Documentation
• Contributing — development, testing, releasing
• Changelog

License

Apache-2.0. See LICENSE and NOTICE.