Skip to main content

sentinel-probes: Dynamic Instrumentation for Python

Metric, span and span-tag probes on a running Python service, added, changed and removed from Sentinel without a deploy. Standard library only.

Start it

With no code change, the way ddtrace-run and opentelemetry-instrument work. Every Python process the command starts runs the client, including pre-forked workers:

pip install sentinel-probes

SENTINEL_PROBES_URL=https://ingest.example.com/otlp/v1/probes \
SENTINEL_API_KEY=<ingest key> \
OTEL_SERVICE_NAME=checkout \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.version=1.4.2 \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
sentinel-probes-run gunicorn app:app

SENTINEL_SERVICE, SENTINEL_ENV, SENTINEL_VERSION and SENTINEL_OTLP_ENDPOINT take precedence over the OpenTelemetry variables. Without SENTINEL_PROBES_URL the client stays off. With the URL set but the rest of the configuration incomplete, it logs one warning and stays off.

Or start it from code:

import sentinel_probes

sentinel_probes.start_probes(
    service="checkout",
    env="production",
    api_key="<ingest key>",
    version="1.4.2",
    probes_url="https://ingest.example.com/otlp/v1/probes",
    otlp_endpoint="http://otel-collector:4318",
)

How it stays cheap

  • Python 3.12+: probes are placed with sys.monitoring local events, only on the targeted code objects. Every other function runs untouched.
  • Rate limit: each probe runs at most 50 times a second. Once a probe has used up that allowance, its code object's events are switched off for the rest of the second, so a hot method runs at full speed between its sampled calls.
  • Before 3.12: a method probe wraps the function. The signature is computed once, the rate limit is checked before arguments are bound, and once the allowance is used up the wrapper skips the lock entirely.
  • CPU budget: probes may use at most 2% of a second's CPU time. Beyond that, they're refused until the next second.
  • Off the application's threads: a daemon thread polls, reports and exports. It restarts in forked children with a new runtime id.
  • Raw-socket HTTP: the client talks over a plain socket, not urllib, http.client or requests, so OpenTelemetry's and ddtrace's instrumentation never see its traffic. Each call is time-boxed and responses are capped at 1 MiB.
  • Capture runs no application code:
    • attributes are read through descriptors, never via properties, __getattr__ or __str__;
    • an exception's message is read only when its class keeps BaseException.__str__;
    • only exact standard-library types (datetime, UUID, Decimal, …) are turned into text.
  • Bounded and redacted: capture is bounded (depth 2, 20 entries, 256 characters, 30 variables) and redacted by name (password, token, session, …, plus the probe's keywords).
  • Never into the application: a failure becomes the probe's error.

Per call of a probed method (tests/bench.py). The probe is a gauge of the return value; most calls are past the 50-a-second limit:

Python plain call with the probe added
3.13 (sys.monitoring) 0.10 µs 0.10 µs +0.01 µs
3.11 (function wrapper) 0.09 µs 0.51 µs +0.43 µs

Continuous profiling

Set SENTINEL_PROFILING_ENABLED=true and the client also profiles CPU, with or without probes. Profiles go to Sentinel once a minute and appear on the service's Profiling tab.

Setting Default Meaning
SENTINEL_PROFILING_URL beside SENTINEL_PROBES_URL …/otlp/v1/profiles/pprof
SENTINEL_PROFILING_INTERVAL_MS 10 how often threads are sampled
SENTINEL_PROFILING_MAX_OVERHEAD 0.01 the most of one core the profiler may use

How it stays cheap:

  • CPU, not wall time. A daemon thread reads every thread's stack and its CPU clock. Only CPU used since the last sample is counted, so a thread waiting on I/O or a lock costs nothing and shows nothing.
  • A budget it enforces on itself. The sampler measures its own CPU. Over 1% of one core, it samples less often, down to every 100 ms. If it is still over, it stops and logs why, once.
  • Bounded memory. At most 10,000 distinct stacks a minute (the rest are counted as "other stacks"), 128 frames deep, and 5 minutes of profiles queued while Sentinel is unreachable.
  • Off the application's threads. Encoding and upload run on their own threads. A 402 (not in the plan) pauses uploads for an hour; 429 and 5xx back off. It restarts in forked children.

Measured with tests/bench_profiling.py (a threaded HTTP server under load): the sampler used 0.49% of the server's CPU at the default 10 ms interval.

What runs where

Python Method probes Line probes
3.12+ yes yes
3.8–3.11 yes no: the only way is sys.settrace, which taxes every call in every thread. A line probe reports an error that says so.

The client advertises what it can place (locations=line,method or method).

Before 3.12, the wrapper replaces the function where it is defined. Code that imported it by name before the probe was placed (from orders import total) keeps calling the original, and a nested function can't be reached at all; the probe reports which.

Tests

python -m pytest -q                            # unit, conformance (79 cases, 3,030 patterns), hardening
docker run --rm -v "$PWD":/w/python:ro -w /w/python/tests/e2e python:3.11-slim sh run.sh
docker run --rm -v "$PWD":/w/python:ro -w /w/python/tests/e2e python:3.13-slim sh run.sh
docker run --rm -v "$PWD":/w/python:ro -w /w/python python:3.13-slim python tests/bench.py

The end-to-end run starts an unmodified application with python -m sentinel_probes.run against a fake backend. It checks the poll, statuses (including the line-probe refusal on 3.11), a redacted sample and the OTLP metrics.

Metadata

Release files for sentinel-probes 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sentinel-probes 0.2.0
File Size Uploaded Provenance Actions
sentinel_probes-0.2.0.tar.gz 57.4 kB Without provenance Details for sentinel_probes-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sentinel-probes 0.2.0
File Interpreter ABI Platform Actions
sentinel_probes-0.2.0-py3-none-any.whl Python 3 none any Details for sentinel_probes-0.2.0-py3-none-any.whl

Total release size: 101.9 kB

Release files / sentinel_probes-0.2.0.tar.gz

Download URL sentinel_probes-0.2.0.tar.gz
Size 57.4 kB
Tags Source
SHA-256 checksum
How to use checksums
8751b7db7712a5a858fe681bf3b872e6e51482a65b4a5a8e30f93a4fa049c175
BLAKE2b-256 checksum
How to use checksums
a4db052fd1cc9a91c5bb439e9ce8919ae5e9f9c3a354a3dd6afff92005d0d6c6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Release files / sentinel_probes-0.2.0-py3-none-any.whl

Download URL sentinel_probes-0.2.0-py3-none-any.whl
Size 44.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
76ab99023c0ef3d21f3bdda1076fa16e567c5b17f530ec8d543e493dc8330ae5
BLAKE2b-256 checksum
How to use checksums
2b39d29075a2d4221aa02400508a07fbca2ad8b3baf0a4ad7bac43a4bcd6bb41
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Security for sentinel-probes 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page