polars-telemetry

OpenTelemetry instrumentation for Polars query execution.

One span per query carrying the plan, and per-node counters as metrics, to any OTLP collector — or a profile file you open in your browser, with no collector at all.

Install

pip install polars-telemetry          # API only; bring your own OTel SDK
pip install 'polars-telemetry[otlp]'  # with SDK and OTLP exporter
pip install 'polars-telemetry[datadog]'  # for the DogStatsD exporter

Python 3.10+.

Use

import polars_telemetry

polars_telemetry.install()

install() enables polars' query monitoring, which sets the engine affinity to "streaming". On polars 1.44 that changes how your queries execute, so it never happens on import. polars 2 runs lazy queries on the streaming engine anyway. uninstall() turns monitoring off and puts the previous affinity back.

What you get

A polars.collect span per query, on whatever trace context was active:

• the plan — scan sources, pushed-down predicates, join types and keys, group-by keys
• polars.cpu_ms, polars.parallelism, result rows
• the hottest node and its share of total CPU
• diagnostics — parallel efficiency, join growth, projection efficiency, morsel skew, predicate pushdown, row-group skipping
• the file, line and function that ran the query, as OpenTelemetry's code.* attributes

Per-node counters — rows, morsels, polls, work-stealing, poll latency, state updates, IO time and bytes — as 15 metric instruments dimensioned by node kind.

Every name is listed in the attribute reference.

Where it goes

from datadog import DogStatsd
from polars_telemetry.export.dogstatsd import DogStatsdExporter

statsd = DogStatsd(disable_buffering=False, disable_background_sender=False)
polars_telemetry.install(exporter=DogStatsdExporter(statsd))

exporter takes a list, so several can run at once. Each has a page in the docs, with its options and what it costs.

Profiles without a collector

from polars_telemetry.export.file import FileExporter

polars_telemetry.install(exporter=FileExporter("profiles/session.jsonl"))

One self-contained JSON document per query: both plans with every node property, all 19 per-node counters, the diagnostics, and a fingerprint of the plan shape.

Drop the file on the profile viewer to read both plans, per-node counters, and a diff between two runs of the same shape. It runs entirely in your browser; nothing is uploaded. To try it without a workload of your own, open the viewer and load a TPC-H example with one click: the 22 queries at scale factor 1 or 10, three runs each. The same sessions are in examples/.

Label what runs

with polars_telemetry.label("revenue_by_region"):
    report.collect()

The label is on the span and in the profile; nested labels join with /.

Profile a block of code

from polars_telemetry import profile

with profile() as session:
    report = build_report()

session.slowest.call_site  # where the slow one was run
session.write("report.jsonl")  # open in the viewer

Installs instrumentation only if nothing was installed. With an application already instrumented it collects alongside the existing exporter. polars 2 removed LazyFrame.profile(); this covers the same ground, for every query a block runs.

Configure

from polars_telemetry import Config

polars_telemetry.install(Config(node_metrics=False))

Your data

Spans carry plan detail: scan paths, column names, join keys and literal predicate values — col("email") == "..." arrives verbatim, because knowing which predicate was slow is usually the point.

• Config(redaction=Redaction()) masks literal values: text, numbers, dates and times. Redaction(paths=True, call_site=True, labels=True) masks more.
• A URL's query string, which for a presigned or SAS URL is a credential, becomes ?<query> for every exporter, unless a redaction sets url_queries=False.
• redacted(exporter, ...) gives one exporter its own setting, so a shared backend can get a masked copy while a local file keeps full detail.
• Literals are never used as metric attributes, at any setting.
• Attributes that can carry user data are listed in polars_telemetry.export.semconv.CARRIES_USER_DATA.

Polars Cloud

If polars-cloud is installed, its observer is wrapped and forwarded to rather than replaced. Both work at once.

Links

• Documentation
• Contributing — development, testing, releasing
• Changelog

License

Apache-2.0. See LICENSE and NOTICE.