Skip to main content

agentdevlabs

Python SDK for Agent Dev Labs Capture & Replay. adl(client) records the calls your app makes through an OpenAI or Anthropic client. Your provider key stays with your app.

pip install agentdevlabs
import openai
from agentdevlabs import adl

client = adl(openai.OpenAI())

Set up with a coding agent

  1. Check that AGENT_DEV_LABS_TOKEN is set in .env. If it isn't, stop and ask the user for it. Never create a token or write one into code, and never commit .env. The SDK reads the process environment, not .env itself, so make sure the app loads it. In Python call load_dotenv() from python-dotenv before wrapping the client, or export the variable.
  2. Install agentdevlabs and the provider SDK (openai or anthropic) with the project's own package manager.
  3. Wrap the provider client where it is created: adl(openai.OpenAI()) or adl(anthropic.Anthropic()). Leave AGENT_DEV_LABS_MODE unset unless the user asks, because it outranks the devkit toggle.
  4. Run the app twice with AGENT_DEV_LABS_VERBOSE=1, letting each run exit on its own (a killed run loses its upload). Every call prints a line starting [agentdevlabs]: run 1 should print captured and run 2 replayed. Unset the setting after.
    • passed-through or failed: the reason follows the word; read it.
    • captured on run 2: read the SDK's warnings, and look for a prompt that changes between runs, such as a date or an id.
    • No line: the call wasn't hooked; read the SDK's warning.
    • Ask the user to look in the console's Traces and Calls only if that doesn't settle it, since you can't sign in.

Settings are environment variables only:

Variable Meaning
AGENT_DEV_LABS_TOKEN Your workspace token. Without it the SDK warns once and records nothing: each call passes through, and its report says passed-through, off and disabled.
AGENT_DEV_LABS_MODE auto, capture, replay or off, in any case. Overrides the server's mode, except a server off for production or the plan limit, which always wins. Any other value runs as auto, with one warning naming it.
AGENT_DEV_LABS_DISABLED 1, true, yes or on turns the SDK off; 0, false, no, off or empty leaves it on. Any other value turns it off, with one warning naming it. Beats every other setting.
AGENT_DEV_LABS_VERBOSE 1, true, yes or on prints one line per call to standard error: [agentdevlabs] captured, replayed, passed-through or failed, the last two with their reason. 0, false, no, off or empty leaves it off; any other value leaves it off, with one warning naming it. Sends nothing.
AGENT_DEV_LABS_API_URL The API to talk to.
AGENT_DEV_LABS_REPLAY_SPEED How fast a replayed stream plays: realtime (the default), fast (10x), instant, or a number to divide the recorded pace by. Non-streamed replies are instant unless this is set.
AGENT_DEV_LABS_GROUP Names a trace group: calls are captured into it and replays use only its recordings. Letters, digits, ., _, / and -, up to 100.
AGENT_DEV_LABS_CACHE_DIR Where replayed captures are cached, encrypted. Defaults to your OS cache directory.

Make one trace from a block

Calls inside with trace(name): (or a function decorated with @trace(name)) make one trace. The trace ends when its last call finishes, which can be after the block returns.

from agentdevlabs import trace

with trace("checkout"):
    client.chat.completions.create(...)

@trace("summarise")
def summarise(text): ...

See what each call did

adl(client, on_call=...) calls your listener once per provider call with the mode in effect, where it came from, and what happened. A request that raises gets no report: your app gets the error. It sends nothing anywhere.

client = adl(
    openai.OpenAI(),
    on_call=lambda report: print(report.outcome, report.mode, report.mode_from),
)
Field Meaning
outcome replayed (a stored capture answered), captured (the provider answered and the call is recorded), passed-through (the provider answered and nothing was recorded) or failed (strict replay had no capture, so the provider was not called).
mode The mode this call ran in: auto, capture, replay or off.
mode_from env (AGENT_DEV_LABS_MODE), server (the console or devkit toggle, the environment's default, or the token), default (the server could not be reached or rejected the token) or disabled (AGENT_DEV_LABS_DISABLED or no token).
pin pin.trace_id and pin.set_at when the environment's pinned trace answered the call, else None.
trace_id, step, cache_key Which capture the call matched or made.
reason Why a call failed or went unrecorded.

The listener runs on the calling thread, or the calling event loop for an async client. A listener that raises is ignored: the call goes on and the SDK warns once. A report holds ids and reasons, never a prompt, a response or a key. A provider retry is its own call, and an attempt about to be retried gets no report.

Replay in CI

Record a group on your machine, then replay only that group in CI. A call with no recording in the group fails with an error naming the cache key and the group (or saying the group doesn't exist), so CI never reaches the provider.

The group name uses letters, digits, ., _, / and -, up to 100 characters; anything else fails at startup. AGENT_DEV_LABS_TOKEN in CI must be a replay-only service token, which an owner or admin creates in the console's Access tokens page.

AGENT_DEV_LABS_GROUP=checkout-flow AGENT_DEV_LABS_MODE=capture pytest   # once, locally
# .github/workflows/test.yml
name: test
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v5
      - run: uv sync
      - run: uv run pytest
        env:
          AGENT_DEV_LABS_TOKEN: ${{ secrets.AGENT_DEV_LABS_TOKEN }}
          AGENT_DEV_LABS_GROUP: checkout-flow
          AGENT_DEV_LABS_MODE: replay

Project rules: a worked example

A travel agent's system prompt carries today's date, a session UUID and a request id, so Monday's recording never matches on Tuesday. In the console's Project rules page, three masks make the calls match:

  • [0-9]{4}-[0-9]{2}-[0-9]{2} masks the date.
  • [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} masks the session UUID.
  • req_[A-Za-z0-9]+ masks the request id.

A mask changes only what matching reads, so the recording still holds the original text. To see which mask a pair of calls needs, open a trace, choose "Why didn't this replay?" on step 1 and "Try this rule" on a suggestion.

A card number the user typed should never be uploaded, so add a redact pattern such as [0-9]{4} [0-9]{4} [0-9]{4} [0-9]{4}. The SDK replaces it with [redacted] before upload; matching still uses the original, so redacting doesn't stop replays. The page's "Try it" box shows both results on a sample request that never leaves the browser.

Calls already recorded keep the rules they were captured with: after saving a rule, capture them again.

License and privacy

Apache-2.0. The SDK sends nothing beyond the capture and replay events described above: no usage analytics or telemetry.

Versions

Versions follow semantic versioning. Before 1.0, a minor release (0.1 to 0.2) may break your code; the changelog says how, and the API supports only the latest 0.x minor. From 1.0, the API supports the previous major version for 12 months after a new major ships. See the changelog.

Metadata

Release files for agentdevlabs 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentdevlabs 0.2.0
File Size Uploaded Provenance Actions
agentdevlabs-0.2.0.tar.gz 81.6 kB Without provenance Details for agentdevlabs-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentdevlabs 0.2.0
File Interpreter ABI Platform Actions
agentdevlabs-0.2.0-py3-none-any.whl Python 3 none any Details for agentdevlabs-0.2.0-py3-none-any.whl

Total release size: 130.7 kB

Release files / agentdevlabs-0.2.0.tar.gz

Download URL agentdevlabs-0.2.0.tar.gz
Size 81.6 kB
Tags Source
SHA-256 checksum
How to use checksums
51501fc059cd00aa337cd7d74a194ff107c75068824e9fa7f914509eff97c6c7
BLAKE2b-256 checksum
How to use checksums
3c944ea6200118b0d42a0cce9f8d9fba4244571c0bb5f2e2134f67d5d910464e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / agentdevlabs-0.2.0-py3-none-any.whl

Download URL agentdevlabs-0.2.0-py3-none-any.whl
Size 49.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a9ad87386600c38a15482f7dc5f76049ce7416bcd5917c2bc46aa204d44599ab
BLAKE2b-256 checksum
How to use checksums
c1d71a68908bde2b5521b1732c2c10c4c98e4d3b301d5c0cc5e37c6d2a69bdb5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Security for agentdevlabs 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page