Runtime Labs local Python SDK

The SDK emits OpenTelemetry spans with full eval payloads for LangGraph execution and Google Gen AI non-streaming model calls. The scanner's strict DeploymentManifest adds runtime_labs.node_id when source proof succeeds. Tracing also works with missing or stale manifests: those operations remain unattributed and the handle reports fixed diagnostic codes.

Install and initialize

From this workspace:

uv sync
uv run python examples/sdk_offline/demo.py

For an application with its own environment, install both local distributions (substitute the checkout directory):

uv pip install /path/to/runtime-labs/packages/contracts \
  '/path/to/runtime-labs/packages/sdk[google,langgraph]'

runtime-labs-sdk is independently packaged and has no scanner dependency. The Google and LangGraph adapters are optional and loaded at initialization; importing runtime_labs does not import either integration. Unsupported dependency versions are diagnosed and their adapters are skipped.

Connect and scan the application checkout in the web UI first. Run the application inside its Git checkout with an origin remote. No manifest file is downloaded or configured by the customer. The SDK reads that remote and asks the backend for the successful scan's manifest at startup. SSH and HTTPS remotes normalize to the same host/owner/repository identity.

Optional application .env (the default is shown):

RUNTIME_LABS_API_URL=http://127.0.0.1:8000

At startup, before constructing provider clients or building/executing graphs:

import runtime_labs

sdk = runtime_labs.init(api_key="runtime-labs-local-dev-key")

try:
    # Normal application startup and execution.
    ...
finally:
    sdk.flush()
    sdk.shutdown()

The API key is sent to the backend as an Authorization bearer header for manifest lookup and automatic telemetry delivery. Authentication and workspace lookup are currently deferred: the backend uses the default tenant/workspace. The key is hidden from configuration representations and is never included in span payloads or provider credentials.

Configuration rules:

• api_url= overrides process RUNTIME_LABS_API_URL, then the nearest .env value, then the local default above. SDK settings are read without interpolation or mutating the application's environment.
• By default the SDK resolves the checkout's origin and HEAD, requests POST /api/sdk/manifest, validates the strict deployment manifest, and holds it in memory. No manifest cache file or customer download step is required.
• The backend resolves the complete canonical Git identity, reads the successful scan's saved artifact, and returns the manifest and repository ingestion route. Duplicate matching checkouts, unavailable artifacts, and revision mismatches are rejected rather than guessed.
• Existing source-path, source-digest, and loaded-code proofs still apply.
• For explicit offline/test use only, RUNTIME_LABS_MANIFEST_PATH remains a legacy override. Paths retain their existing ~ and nearest-.env resolution rules. An empty override enables normal runtime fetching instead of disabling attribution.
• A failed lookup never breaks application execution. Inspect sdk.diagnostics for repository_identity_unavailable, manifest_fetch_failed, or manifest_revision_mismatch. Unsupported or unverified spans stay unattributed.
• Deployed environments must retain Git metadata for automatic remote discovery. Initialization fetches once with a five-second HTTP timeout; retry after fixing backend/scan availability by shutting down and initializing again.

Public API and local output

init(*, api_key, exporter=None, tracer_provider=None, capture_payloads=True, api_url=None) returns an SDK handle with:

• diagnostics: a sorted tuple of sanitized, deduplicated diagnostic codes.
• sources.manifest: a detached strict-contract snapshot; editing it cannot change the immutable attribution indexes. Repository/revision metadata stay here.
• flush(timeout_millis=30000) -> bool: flush only this SDK's exporter.
• shutdown(): flush, disable instrumentation, restore owned hooks, and release SDK-owned resources. It is safe to call repeatedly.

Identical initialization returns the same handle. A different key, resolved path, exporter, explicit provider or payload setting returns that handle with initialization_conflict, without installing more hooks. Shut down and initialize again to change settings; construct new graphs after reinitialization. Shutdown should follow completion or cancellation of application work; it does not await running requests.

After successful runtime bootstrap, the SDK automatically exports OTLP/HTTP spans to the repository route supplied by the backend. The OTLP exporter is included in the base SDK dependency set. Explicit exporters and standard OTEL_* settings remain supported; OTEL_TRACES_EXPORTER=none disables delivery. Offline manifest mode retains the console default. Console records contain trace/span/parent IDs, start/end timestamps, duration, status, and allowed attributes including application payloads. HTTP authentication, environment dumps, and source bodies are not captured. Errors use only error.type values cancelled, timeout or _OTHER; automatic exception events and status descriptions are disabled. Application code and other instrumentation retain their own logging/export policies.

Payloads for evals

Full payload capture is enabled by default. The MVP performs no redaction or SDK size limiting. Each payload attribute is a JSON string, supported by standard OTel exporters. Decode it with json.loads(span.attributes[attribute_name]).

Span	Payload attributes
Workflow / agent	runtime_labs.input, runtime_labs.output: input state and returned value
Google model call	runtime_labs.request: original model, contents and supplied generation config; runtime_labs.response: returned response body, candidates, usage, parsed output and automatic tool-calling history
Google model call	gen_ai.input.messages, gen_ai.output.messages: standard message/part representation; gen_ai.system_instructions when supplied
Tool execution	gen_ai.tool.call.arguments, gen_ai.tool.call.result: bound arguments and returned value

The runtime_labs.* payload attributes carry application data, not additional source identity fields. runtime_labs.node_id remains the only custom source ID. Prompts, completions, all returned candidates and tool payloads are stored in full. Raw provider request/response attributes retain provider-specific body fields; HTTP options and sdk_http_response transport envelopes are not body payloads. The SDK does not inspect client credentials or automatically export its API key.

Request/input snapshots are taken before execution, so in-place application or Google automatic function-calling mutations cannot rewrite the original input. The returned automatic function-calling history retains its intermediate tool calls/results. Workflow outputs contain the final graph state, including the application's answer, enabling complete input/output evaluation examples. Runtime trace and span IDs join these payloads to the manifest's source node IDs.

JSON-compatible values, Pydantic models, dataclasses, enums, dates and bytes are supported; bytes are encoded as base64. Callable tool definitions store callable names and qualified names rather than executable code. Tool arguments are serialized independently: unsupported or cyclic argument values (such as database sessions and ORM models) are omitted while the other arguments, including defaults, remain captured. runtime_labs.tool.arguments.omitted lists the omitted parameter names, and the handle reports tool_arguments_omitted. No object internals or string representations are inspected as a fallback. Unsupported or cyclic values in other payloads produce payload_capture_failed and omit that payload attribute while preserving application results/exceptions and other span data. A failed or cancelled operation keeps its input; it does not invent a response.

To select metadata-only output explicitly:

sdk = runtime_labs.init(api_key="runtime-labs-local-dev-key", capture_payloads=False)

OTel/application processors and downstream collectors can have their own limits; the Runtime Labs SDK does not introduce payload truncation. Local JSON and OTLP export receive the same payload attributes. Evaluation execution/dataset storage are separate later components; this change supplies their runtime examples.

For assertions, supply the standard OTel in-memory exporter:

from runtime_labs.export import InMemorySpanExporter

exporter = InMemorySpanExporter()
sdk = runtime_labs.init(api_key="runtime-labs-local-dev-key", exporter=exporter)
# Execute work; then inspect exporter.get_finished_spans().

The explicitly selected in-memory exporter retains spans until clear() or shutdown(); applications should bound its use. Custom exporters use the standard OTel SpanExporter interface. Export/flush/shutdown exceptions become diagnostic codes and do not replace application results or exceptions. Local export is synchronous, so exporter work contributes latency.

An application-owned global TracerProvider is respected. An optional explicit tracer_provider also works. The SDK adds one reusable processor scoped to runtime_labs, never shuts down an application provider, and does not export its other instrumentation scopes through the local exporter. If no application provider exists, the SDK creates a private OTel provider without taking the global slot. Active OTel context still establishes parentage across both providers. Application processors on a shared provider can also observe SDK spans.

Remote OTLP/HTTP export requires both explicit standard settings:

uv pip install '/path/to/runtime-labs/packages/sdk[otlp]'
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf

OTEL_EXPORTER_OTLP_ENDPOINT, generic protocol/timeout/header settings and the trace-specific settings are handled by the standard exporter. HTTP/protobuf is the supported protocol. An endpoint alone never enables remote export; missing extras, missing endpoint or unsupported protocol produce diagnostics and local fallback. The pinned exporter's per-instance transport logger is silenced so server reasons and transport exceptions cannot echo private text; failures use sanitized SDK diagnostics. Its optional internal metrics use a no-op provider. OTEL_TRACES_EXPORTER=none discards SDK output. No collector/server is included.

Correlation identifiers and trace context

The SDK records the identifiers a framework uses to tie runs together, verbatim, and never decides what they mean. On every LangGraph invoke/ainvoke it copies allowlisted keys from the run config onto the workflow span as runtime_labs.ctx.<framework>.<key>, for example runtime_labs.ctx.langgraph.thread_id. The defaults are LangGraph's thread_id, checkpoint_id, checkpoint_ns and run_id. Only allowlisted keys leave the process, because applications often keep user IDs and secrets in the same config; only non-empty strings, integers and UUIDs are copied, truncated to 256 characters.

Extend the allowlist without code changes:

export RUNTIME_LABS_CORRELATION_KEYS=langgraph.tenant,langgraph.session

or with runtime_labs.init(api_key=..., correlation_keys={"langgraph": ["tenant"]}). Runtime Labs pipelines map these raw identifiers to concepts such as conversations with rules that are data, so new frameworks need a rule, not an SDK release.

Within one process, span parents follow OpenTelemetry context, which LangGraph and asyncio carry into their own threads and tasks. Work that crosses a process or service boundary (an HTTP call, a queue message, an outbox row) must carry the context itself, as a standard W3C traceparent:

headers = runtime_labs.inject_context()        # producer
with runtime_labs.continue_trace(headers):     # consumer
    handle(message)

Code that starts raw threads should run work with contextvars.copy_context().run(...) to keep the parent.

Supported interfaces and conventions

Component	Pinned/tested version	Coverage
CPython	3.11–3.13	Code identity and UTF-8 call positions
Google Gen AI	2.27.0	Models.generate_content, AsyncModels.generate_content
LangGraph	1.2.18	StateGraph construction, add_node, compile; compiled invoke/ainvoke
LangChain Core	1.7.0 (workspace lock)	LangGraph callback signature and context propagation
OTel API/SDK	1.45.1	Context, tracing, processors and standard exporters
OTel OTLP/HTTP exporter	1.45.1 (optional)	Explicit HTTP/protobuf export
OTel Python semantic-conventions package	0.66b1 (SDK dependency)	SDK resource conventions
GenAI conventions	51a0f580f8a6ad804ca4de5e033683c6fbbe5906	Development conventions, mapping in semconv.py

Verified official references: OTel Python instrumentation, GenAI inference conventions, agent/workflow/tool conventions, Google SDK, LangGraph, and streaming boundaries. GenAI conventions remain in development; their revision is explicit rather than silently adopting future changes.

Provider spans use generate_content, CLIENT, and gcp.vertex_ai, gcp.gemini or gcp.gen_ai according to runtime backend metadata. Request/response models come from actual request/response metadata. Tokens come only from provider usage: input = prompt count, output = candidate count plus reasoning count when reported; reasoning and cache-read counts are separately reported when available. Missing counts remain absent. For automatic function calling, the final SDK response's reported usage is recorded; counts from hidden HTTP rounds are not estimated or aggregated. One span covers the whole public logical call, including these rounds. Return identity, arguments, exceptions and async cancellation are preserved.

Workflow, agent and tool spans use INTERNAL and standard invoke_workflow, invoke_agent and execute_tool operations and names. The only custom source attribute is runtime_labs.node_id. No extra source IDs or static parent IDs are attached. The Google adapter honors and sets standard OTel instrumentation suppression, and deduplicates reentry. Uncooperative third-party instrumentors that ignore suppression cannot be deduplicated reliably.

Source-binding policy

The SDK consumes DeploymentManifest directly and validates all references using the shared strict contract. It neither scans nor executes repository source, calls an LLM, generates IDs, or imports modules selected by manifest data.

Attribution requires a unique kind and qualified symbol match, normalized for Python <locals> names, the complete canonical manifest-relative path suffix of the actual module/frame file, the verified full-file SHA-256, and agreement between the executing code and code compiled from those bytes. Compilation is never execution. Checkout filenames alone are normalized out of the code comparison; bytecode, constants, nested code, closure layout, flags and position tables remain. Different absolute checkout directories therefore work without scanner paths or basename-only guessing. Direct scripts use their module spec or a canonical module name derived from the manifest path; source proof is still required. Verification suppresses compiler warnings because those warnings can otherwise echo source lines.

For model calls, the executed CALL instruction must match the exact manifest range, including UTF-8 byte columns; same-line sync/async calls remain distinct. Workflow construction must fall within its unique verified constructor range. Registered callback objects identify nested agents. Known module-level Python tools and their already-loaded aliases receive wrappers; registration by itself never emits a tool span. Ordinary functions such as finalize_response receive no span. Stale, missing, ambiguous or unprovable code produces generic spans and diagnostics, with no guessed source identity.

Files are checked again at each attribution, including after construction or a previous successful match. Preloaded code is accepted only if its executable code and positions agree with the current verified file. This rejects a matching disk file paired with different loaded bytecode/constants. It does not reconstruct historical import-time bytes: comment-only changes with identical code/positions, changes outside the executing code unit, mutable globals/closures, or deliberate code-object forgery are not a deployment attestation. The current whole-file digest must still match the manifest. IDs alone do not validate a deployed revision or promise identity reconciliation across commits.

Runtime parents always come from OTel context. A direct shared-tool call is under the executing agent. A Google automatic tool call is under that logical provider span, itself under the executing agent. Static registration/containment edges do not override these parents. LangGraph's public runtime events primarily describe streaming and do not expose this example's exact Python callback identities, so this adapter uses narrowly tested construction/registration/execution wrappers without callback arguments or customer decorators.

Offline acceptance harness

examples/sdk_offline contains an unchanged source copy of the sibling simple example and its saved verified manifest. The sibling application's source and configuration are untouched. The demo denies socket connections and uses real pinned clients/graphs with HTTPX mock transports and fake Vertex credentials.

Each of billing, technical and general, through both sync and async graph entry points, asserts this six-span tree (the selected agent varies):

invoke_workflow build_workflow
  invoke_agent triage_agent
    generate_content offline-request-model
  invoke_agent billing_agent | technical_agent | general_agent
    execute_tool lookup_policy
    generate_content offline-request-model

All six spans in each run must have IDs in the loaded manifest. Across the three branches, the harness observes all ten distinct nodes, including all four model sites. It checks the actual tool parent on every run and prints node IDs with the tree. This is an observation of the offline runs, not a source coverage estimate. Tests additionally exercise true async callbacks/provider calls using a newly compiled scanner manifest, automatic tool execution, parallel requests, stale loaded code and failure/privacy/lifecycle cases. Tests deny network globally.

Limitations

Streaming graph execution has no workflow-lifetime adapter; registered callbacks may still emit leaf spans. Google streaming, chats and embeddings are not adapted. Provider-driven tools are covered when they are already-loaded, manifest-matched Python functions passed to Google; arbitrary callable objects, local tool closures, cached pre-init callable references, decorators with replaced code, packaged/zip or unavailable source, and other framework/provider versions are not guaranteed. Graphs must be constructed after initialization. No whole-program tracing or instrumentation of ordinary application functions is performed. Hosted lookup, authentication, collectors, other providers, evaluation execution and cross-commit reconciliation are outside this milestone.