HyperProbe Python Agent

Production-grade, non-breaking live debugger and telemetry agent for Python.

HyperProbe captures stack frames, local variables, and logs at selected source lines without stopping the application in a debugger. At agent initialization, Python 3.12+ selects the sys.monitoring backend. CPython 3.7–3.11 selects the bytecode backend. Both support snapshot, log, counter, metric, and duration probes and use the same broker, evaluator, serializer, quotas, safety monitor, and output pipeline. Only the selected backend is imported; bytecode is not a dependency on 3.12+.

Features

• Non-Breaking Snapshot Probes: Capture local variables and stack traces dynamically at any line of code.
• Dynamic Log Probes: Inject live log templates to print formatted output without redeploying code.
• Counter Probes: Emit a value of 1 whenever a configured source line is reached.
• Metric Probes: Evaluate a numerical Python expression and emit its value.
• Duration Probes: Measure elapsed monotonic time in milliseconds between two source lines, with optional request correlation.
• Circular Reference & Deep Object Protection: Memory-safe serialization of deeply nested or cyclic variables.
• Safety Guard: Suspends instrumentation when scheduler lag or the capture pause budget crosses its configured threshold.

Bytecode backend (CPython 3.7–3.11)

The broker still supplies runtime_location and runtime_line. The backend uses the existing path resolver, locates functions through their loaded modules, and replaces only the selected functions' code objects. It does not use sys.settrace, threading.settrace, or a heap-wide gc.get_objects() scan. Pending probes can watch standard file imports; the normal loader executes the module after the SDK prepares its code. Loaded factories can register and instrument nested functions they create later. Conditions, watches, redaction, trace IDs, hit limits, and telemetry use the shared services. Duration probes index both source locations, including an end line in a later-imported module.

This backend also selects its own agent lifecycle controller at startup; the Python 3.12+ lifecycle and sys.monitoring implementation remain separate. Broker responses are compared against stable local definition records. An unchanged response (including reordered probes) does no module discovery, function planning, or bytecode rewriting. Condition, watch, and template changes replace evaluation state without rebuilding bytecode. Location changes discover the new target and replan affected functions; removal uses tracked functions. Callback-site tokens are held weakly, so short-lived nested functions do not accumulate permanent lookup entries. Broker-confirmed completed probes stay disabled until the broker removes them.

On RED, the backend closes its capture gate and synchronously restores the saved original code for functions it still owns, including factories. Pending import work cannot install new probes while suspended. Restoration runs before agent bookkeeping or logging, and recovery builds bytecode without holding the agent lock. Capture delivery checks the original definition identity and activation generation before queue admission, so an old capture cannot be accepted under an updated definition or after RED/recovery. A frame already running keeps its existing code, but its callbacks become no-ops. An expression already running cannot be forcibly interrupted safely. Instrumentation failures are reported through bounded engine diagnostics and best-effort error logging; normal exceptions from the application's own function or module still propagate.

Discovery supports module/class namespaces and bounded built-in containers. Arbitrary generated code, custom loaders, hidden pre-existing closures, and automatic reload coverage are not guaranteed. Function prologues are not probe locations. Performance, including a 2% overhead target, requires workload-based measurement; the shared evaluator is not a security sandbox.

Metric-family probes

Counter, metric, and duration probes are created through HyperProbe in the same way as snapshot and log probes. No application code changes or additional public SDK calls are required.

Conditions are evaluated at every configured location. Metric expressions use the same restricted Python evaluator as conditions and watches. Numerical strings follow Node SDK parseFloat prefix behavior for compatibility; invalid or non-finite values are exported with metric_value set to 0 and a populated capture_error. The bytecode backend examines at most the first 1,024 characters of a metric string.

Duration correlation expressions must evaluate to a string or number. When omitted or when they evaluate to None, a singleton correlation key is used. Unmatched duration ends are ignored, and pending starts expire after 60 seconds. On CPython 3.7–3.11, correlation keys are limited to 256 characters and at most 4,096 starts are kept per process; the oldest start is evicted if that limit is reached. Duration starts never consume a probe hit. A matching end consumes the hit when its event is admitted to the telemetry queue.

Installation

Install the package via pip (or testpypi for testing):

pip install hyperprobe-agent

Usage

Start the agent programmatically at your application's entrypoint:

import os
from hyperprobe import HyperProbe

agent = HyperProbe.start({
    "service_id": "<service-uuid-from-dashboard>",
    "environment": os.getenv("PYTHON_ENV"),
    "broker_url": "https://logger.app.hyperprobe.co",
    "commit_sha": os.getenv("GIT_COMMIT"),
})

Required identity values may also be supplied through the environment variables below. Explicit options take precedence. Initialization failures are reported and the application continues without instrumentation.

Configuration Environment Variables

Configure the agent using the following environment variables:

Programmatic integrations can set allow_non_uuid=True for the same legacy bypass. Without an explicit bypass, a non-UUIDv4 service ID prevents agent startup while the host application continues normally.

Safe Evaluation & Security Considerations

Captured strings are checked against configured value-redaction patterns before truncation. When value rules are configured, a string longer than 16,384 characters is masked rather than scanned partially. Log substitutions use the bounded serializer, so nested sensitive keys are masked before an object becomes log text. Telemetry errors carry categories rather than raw runtime values or exception messages. These rules do not infer the origin of an arbitrary watch result: a scalar watch expression such as password can still expose its value when no value pattern matches it.

By default, safe evaluation is enabled (HYPERPROBE_DISABLE_SAFE_EVALUATION=false). The existing evaluator checks function and method names against an allowlist. This reduces available operations but is not a sandbox: properties, overloaded operators, or allowed method names on custom objects can still execute application code. There is no hard timeout that can interrupt such code safely.

Disabling Safe Evaluation (HYPERPROBE_DISABLE_SAFE_EVALUATION=true):

⚠️ Security Warning: Setting HYPERPROBE_DISABLE_SAFE_EVALUATION=true allows conditions, watch expressions, and log templates to execute arbitrary functions and methods in the host application process. This may lead to unintended side effects, state mutations, performance degradation, or security risks if expressions invoke mutating or blocking operations. Only disable safe evaluation in trusted environments where dynamic method evaluation is strictly required.

Debug logging

Set DEBUG to a comma- or space-separated list of namespaces. Patterns accept *, and exclusions begin with -:

DEBUG=hyperprobe:*                               # all HyperProbe SDK logs
DEBUG=hyperprobe:broker,hyperprobe:monitor      # selected components
DEBUG=hyperprobe:*,-hyperprobe:evaluator         # exclude a component
DEBUG_COLORS=1                                  # force ANSI colors

Without *, a selector is exact: DEBUG=hyperprobe does not match hyperprobe:agent. Use DEBUG=hyperprobe* to enable every component. Set DEBUG_COLORS=0 to disable colors. Logging configuration is read during SDK initialization, so set these variables before starting the process.

Shutdown

Call HyperProbe.shutdown() when the application process stops to close monitoring, workers, and the broker connection cleanly.

For servers that import the application before forking workers, such as Gunicorn with --preload, enable worker fork mode:

agent = HyperProbe.start({
    "service_id": "<service-uuid-from-dashboard>",
    "environment": os.getenv("PYTHON_ENV"),
    "broker_url": "https://logger.app.hyperprobe.co",
    "commit_sha": os.getenv("GIT_COMMIT"),
    "fork_mode": "worker",
})

The prefork parent remains agent-free. Every worker receives a unique agent ID, telemetry queue, gRPC channel, monitoring engine, and set of background threads. Application code running inside those workers must use the spawn process start method rather than creating additional children with fork.