Skip to main content

sentinel-probes: Dynamic Instrumentation for Python

Metric, span and span-tag probes on a running Python service, added, changed and removed from Sentinel without a deploy. Standard library only.

Start it

With no code change, the way ddtrace-run and opentelemetry-instrument work. Every Python process the command starts runs the client, including pre-forked workers:

pip install sentinel-probes

SENTINEL_PROBES_URL=https://ingest.example.com/otlp/v1/probes \
SENTINEL_API_KEY=<ingest key> \
OTEL_SERVICE_NAME=checkout \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.version=1.4.2 \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
sentinel-probes-run gunicorn app:app

SENTINEL_SERVICE, SENTINEL_ENV, SENTINEL_VERSION and SENTINEL_OTLP_ENDPOINT take precedence over the OpenTelemetry variables. Without SENTINEL_PROBES_URL the client stays off. With the URL set but the rest of the configuration incomplete, it logs one warning and stays off.

Or start it from code:

import sentinel_probes

sentinel_probes.start_probes(
    service="checkout",
    env="production",
    api_key="<ingest key>",
    version="1.4.2",
    probes_url="https://ingest.example.com/otlp/v1/probes",
    otlp_endpoint="http://otel-collector:4318",
)

How it stays cheap

  • Python 3.12+: probes are placed with sys.monitoring local events, only on the targeted code objects. Every other function runs untouched.
  • Rate limit: each probe runs at most 50 times a second. Once a probe has used up that allowance, its code object's events are switched off for the rest of the second, so a hot method runs at full speed between its sampled calls.
  • Before 3.12: a method probe wraps the function. The signature is computed once, the rate limit is checked before arguments are bound, and once the allowance is used up the wrapper skips the lock entirely.
  • CPU budget: probes may use at most 2% of a second's CPU time. Beyond that, they're refused until the next second.
  • Off the application's threads: a daemon thread polls, reports and exports. It restarts in forked children with a new runtime id.
  • Raw-socket HTTP: the client talks over a plain socket, not urllib, http.client or requests, so OpenTelemetry's and ddtrace's instrumentation never see its traffic. Each call is time-boxed and responses are capped at 1 MiB.
  • Capture runs no application code:
    • attributes are read through descriptors, never via properties, __getattr__ or __str__;
    • an exception's message is read only when its class keeps BaseException.__str__;
    • only exact standard-library types (datetime, UUID, Decimal, …) are turned into text.
  • Bounded and redacted: capture is bounded (depth 2, 20 entries, 256 characters, 30 variables) and redacted by name (password, token, session, …, plus the probe's keywords).
  • Never into the application: a failure becomes the probe's error.

Per call of a probed method (tests/bench.py). The probe is a gauge of the return value; most calls are past the 50-a-second limit:

Python plain call with the probe added
3.13 (sys.monitoring) 0.10 µs 0.10 µs +0.01 µs
3.11 (function wrapper) 0.09 µs 0.51 µs +0.43 µs

What runs where

Python Method probes Line probes
3.12+ yes yes
3.8–3.11 yes no: the only way is sys.settrace, which taxes every call in every thread. A line probe reports an error that says so.

The client advertises what it can place (locations=line,method or method).

Before 3.12, the wrapper replaces the function where it is defined. Code that imported it by name before the probe was placed (from orders import total) keeps calling the original, and a nested function can't be reached at all; the probe reports which.

Tests

python -m pytest -q                            # unit, conformance (79 cases, 3,030 patterns), hardening
docker run --rm -v "$PWD":/w/python:ro -w /w/python/tests/e2e python:3.11-slim sh run.sh
docker run --rm -v "$PWD":/w/python:ro -w /w/python/tests/e2e python:3.13-slim sh run.sh
docker run --rm -v "$PWD":/w/python:ro -w /w/python python:3.13-slim python tests/bench.py

The end-to-end run starts an unmodified application with python -m sentinel_probes.run against a fake backend. It checks the poll, statuses (including the line-probe refusal on 3.11), a redacted sample and the OTLP metrics.

Metadata

Release files for sentinel-probes 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sentinel-probes 0.1.0
File Size Uploaded Provenance Actions
sentinel_probes-0.1.0.tar.gz 46.6 kB Without provenance Details for sentinel_probes-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sentinel-probes 0.1.0
File Interpreter ABI Platform Actions
sentinel_probes-0.1.0-py3-none-any.whl Python 3 none any Details for sentinel_probes-0.1.0-py3-none-any.whl

Total release size: 83.1 kB

Release files / sentinel_probes-0.1.0.tar.gz

Download URL sentinel_probes-0.1.0.tar.gz
Size 46.6 kB
Tags Source
SHA-256 checksum
How to use checksums
7ef80522ab79272d73a34a7b713d4461a4752b7c8d34627492731e39503a0c14
BLAKE2b-256 checksum
How to use checksums
d9c9d66e2138b74cb97ae1a53c818530e0c363bd2dfc765940298c4c513a13e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Release files / sentinel_probes-0.1.0-py3-none-any.whl

Download URL sentinel_probes-0.1.0-py3-none-any.whl
Size 36.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a70a102382ed6a673ff57d84efa4272d1be3c08a2a05197601338d546addcab2
BLAKE2b-256 checksum
How to use checksums
bf493dd0b13ae01997450a10f3b90a4e8b82d4d01431469d1da4d4254360bd33
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.4

Security for sentinel-probes 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page