sentinel-probes: Dynamic Instrumentation for Python
Metric, span and span-tag probes on a running Python service, added, changed and removed from Sentinel without a deploy. Standard library only.
Start it
With no code change, the way ddtrace-run and opentelemetry-instrument work.
Every Python process the command starts runs the client, including
pre-forked workers:
pip install sentinel-probes
SENTINEL_PROBES_URL=https://ingest.example.com/otlp/v1/probes \
SENTINEL_API_KEY=<ingest key> \
OTEL_SERVICE_NAME=checkout \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,service.version=1.4.2 \
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 \
sentinel-probes-run gunicorn app:app
SENTINEL_SERVICE, SENTINEL_ENV, SENTINEL_VERSION and
SENTINEL_OTLP_ENDPOINT take precedence over the OpenTelemetry variables.
Without SENTINEL_PROBES_URL the client stays off. With the URL set but the
rest of the configuration incomplete, it logs one warning and stays off.
Or start it from code:
import sentinel_probes
sentinel_probes.start_probes(
service="checkout",
env="production",
api_key="<ingest key>",
version="1.4.2",
probes_url="https://ingest.example.com/otlp/v1/probes",
otlp_endpoint="http://otel-collector:4318",
)
How it stays cheap
- Python 3.12+: probes are placed with
sys.monitoringlocal events, only on the targeted code objects. Every other function runs untouched. - Rate limit: each probe runs at most 50 times a second. Once a probe has used up that allowance, its code object's events are switched off for the rest of the second, so a hot method runs at full speed between its sampled calls.
- Before 3.12: a method probe wraps the function. The signature is computed once, the rate limit is checked before arguments are bound, and once the allowance is used up the wrapper skips the lock entirely.
- CPU budget: probes may use at most 2% of a second's CPU time. Beyond that, they're refused until the next second.
- Off the application's threads: a daemon thread polls, reports and exports. It restarts in forked children with a new runtime id.
- Raw-socket HTTP: the client talks over a plain socket, not urllib, http.client or requests, so OpenTelemetry's and ddtrace's instrumentation never see its traffic. Each call is time-boxed and responses are capped at 1 MiB.
- Capture runs no application code:
- attributes are read through descriptors, never via properties,
__getattr__or__str__; - an exception's message is read only when its class keeps
BaseException.__str__; - only exact standard-library types (
datetime,UUID,Decimal, …) are turned into text.
- attributes are read through descriptors, never via properties,
- Bounded and redacted: capture is bounded (depth 2, 20 entries, 256
characters, 30 variables) and redacted by name (
password,token,session, …, plus the probe's keywords). - Never into the application: a failure becomes the probe's error.
Per call of a probed method (tests/bench.py). The probe is a gauge of the
return value; most calls are past the 50-a-second limit:
| Python | plain call | with the probe | added |
|---|---|---|---|
3.13 (sys.monitoring) |
0.10 µs | 0.10 µs | +0.01 µs |
| 3.11 (function wrapper) | 0.09 µs | 0.51 µs | +0.43 µs |
Continuous profiling
Set SENTINEL_PROFILING_ENABLED=true and the client also profiles CPU, with
or without probes. Profiles go to Sentinel once a minute and appear on the
service's Profiling tab.
| Setting | Default | Meaning |
|---|---|---|
SENTINEL_PROFILING_URL |
beside SENTINEL_PROBES_URL |
…/otlp/v1/profiles/pprof |
SENTINEL_PROFILING_INTERVAL_MS |
10 |
how often threads are sampled |
SENTINEL_PROFILING_MAX_OVERHEAD |
0.01 |
the most of one core the profiler may use |
How it stays cheap:
- CPU, not wall time. A daemon thread reads every thread's stack and its CPU clock. Only CPU used since the last sample is counted, so a thread waiting on I/O or a lock costs nothing and shows nothing.
- A budget it enforces on itself. The sampler measures its own CPU. Over 1% of one core, it samples less often, down to every 100 ms. If it is still over, it stops and logs why, once.
- Bounded memory. At most 10,000 distinct stacks a minute (the rest are counted as "other stacks"), 128 frames deep, and 5 minutes of profiles queued while Sentinel is unreachable.
- Off the application's threads. Encoding and upload run on their own threads. A 402 (not in the plan) pauses uploads for an hour; 429 and 5xx back off. It restarts in forked children.
Measured with tests/bench_profiling.py (a threaded HTTP server under load):
the sampler used 0.49% of the server's CPU at the default 10 ms interval.
What runs where
| Python | Method probes | Line probes |
|---|---|---|
| 3.12+ | yes | yes |
| 3.8–3.11 | yes | no: the only way is sys.settrace, which taxes every call in every thread. A line probe reports an error that says so. |
The client advertises what it can place (locations=line,method or method).
Before 3.12, the wrapper replaces the function where it is defined. Code that
imported it by name before the probe was placed (from orders import total)
keeps calling the original, and a nested function can't be reached at all; the
probe reports which.
Tests
python -m pytest -q # unit, conformance (79 cases, 3,030 patterns), hardening
docker run --rm -v "$PWD":/w/python:ro -w /w/python/tests/e2e python:3.11-slim sh run.sh
docker run --rm -v "$PWD":/w/python:ro -w /w/python/tests/e2e python:3.13-slim sh run.sh
docker run --rm -v "$PWD":/w/python:ro -w /w/python python:3.13-slim python tests/bench.py
The end-to-end run starts an unmodified application with
python -m sentinel_probes.run against a fake backend. It checks the poll,
statuses (including the line-probe refusal on 3.11), a redacted sample and the
OTLP metrics.
Metadata
Release files for sentinel-probes 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| sentinel_probes-0.2.0.tar.gz | 57.4 kB | Without provenance | Details for sentinel_probes-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sentinel_probes-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for sentinel_probes-0.2.0-py3-none-any.whl |
Total release size: 101.9 kB
Release files / sentinel_probes-0.2.0.tar.gz
| Download URL | sentinel_probes-0.2.0.tar.gz |
|---|---|
| Size | 57.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8751b7db7712a5a858fe681bf3b872e6e51482a65b4a5a8e30f93a4fa049c175
|
|
BLAKE2b-256 checksum How to use checksums |
a4db052fd1cc9a91c5bb439e9ce8919ae5e9f9c3a354a3dd6afff92005d0d6c6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
Release files / sentinel_probes-0.2.0-py3-none-any.whl
| Download URL | sentinel_probes-0.2.0-py3-none-any.whl |
|---|---|
| Size | 44.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
76ab99023c0ef3d21f3bdda1076fa16e567c5b17f530ec8d543e493dc8330ae5
|
|
BLAKE2b-256 checksum How to use checksums |
2b39d29075a2d4221aa02400508a07fbca2ad8b3baf0a4ad7bac43a4bcd6bb41
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.4
|
Security for sentinel-probes 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.