Skip to main content

fakellm-recorder

Record real OpenAI/Anthropic API traffic and emit fakellm fakellm.yaml rules automatically.

VCR-style zero-effort capture, combined with fakellm's editable, error-injectable YAML rules. Run your real code against the real API once through the proxy, get a fakellm.yaml out, hand-edit it to add the error paths recordings can't capture, and commit it.

Install

pip install fakellm-recorder    # once published
# or, from source:
pip install -e .

The loop

1. Record. Start the proxy and point your SDK's base_url at it, then run your existing test or script once.

fakellm-recorder proxy --upstream auto --out sessions/run1.jsonl
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8888/v1", api_key="sk-...real-key...")
# run_my_agent(client, ...)  # real traffic flows through and is recorded

The proxy forwards byte-faithfully to the real upstream, so your code behaves exactly as in production. Credentials are stripped before anything is written to disk; prompts/completions are scrubbed for emails and obvious keys by default.

2. Emit. Turn the recorded session into rules.

fakellm-recorder emit sessions/run1.jsonl --out fakellm.yaml --match-strictness balanced

3. Edit + commit. Add the 429s/500s/malformed responses recordings can't capture, then commit fakellm.yaml alongside your tests. At replay time it's plain fakellm — no recorder needed.

fakellm serve   # from the fakellm package

Match strictness

The hard part is choosing a messages_contain substring specific enough to fire on the right turn but loose enough not to only match that exact transcript. The emitter ranks candidate n-grams by inverse frequency across the whole session, so shared boilerplate (e.g. "You are a helpful assistant") is never chosen.

Mode What it emits
loose mostly turn: + a model glob (gpt-4*); "any response will do" tests
balanced (default) turn: + one distinctive substring (or a tool_result_contains anchor on post-tool turns)
strict turn + substring + exact model_matches + tools_include; closest to a faithful replay

Lint

A standalone check for unreachable rules (shadowed by an earlier first-match), unknown condition/response keys, and responses that set neither content nor tool_calls unintentionally.

fakellm-recorder lint fakellm.yaml

Security

  • Credentials are never persisted. Header capture is allowlist-based, so Authorization / x-api-key / auth headers are dropped at capture time even if PII scrubbing is disabled.
  • PII scrubbing is on by default (emails, OpenAI/Anthropic keys, bearer tokens) because the generated YAML is a commitable artifact. Disable with --no-scrub (credentials are still stripped). Add your own regexes in code via Scrubber(custom_patterns=[...]).

Caveats

  • fakellm is a new, single-author beta (0.3.x). Emitted files are stamped with the targeted config version; pin a fakellm version and expect schema churn.
  • Streaming reconstruction handles both SSE dialects and keeps raw events for a future chunk-fidelity replay mode, but assembly of exotic event orderings may need tuning — the raw events are retained so you can fix assembly without re-recording.
  • Single worker. fakellm stores state in process memory; keep both it and this proxy single-worker.

License

MIT

Metadata

Release files for fakellm-recorder 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for fakellm-recorder 0.1.0
File Size Uploaded Provenance Actions
fakellm_recorder-0.1.0.tar.gz 15.9 kB Without provenance Details for fakellm_recorder-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for fakellm-recorder 0.1.0
File Interpreter ABI Platform Actions
fakellm_recorder-0.1.0-py3-none-any.whl Python 3 none any Details for fakellm_recorder-0.1.0-py3-none-any.whl

Total release size: 32.9 kB

Release files / fakellm_recorder-0.1.0.tar.gz

Download URL fakellm_recorder-0.1.0.tar.gz
Size 15.9 kB
Tags Source
SHA-256 checksum
How to use checksums
7f828573bd39a2e95050d8aedf2e6cfe82885d4508c18d36b162695b247dd76a
BLAKE2b-256 checksum
How to use checksums
372a3d828cb495502c9b9cb7d29a53a957ede9991651f6dcdd57ad227b97eea7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.2

Release files / fakellm_recorder-0.1.0-py3-none-any.whl

Download URL fakellm_recorder-0.1.0-py3-none-any.whl
Size 17.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9ef96e24e22231c3ef3330812cd7aba185b63548ceb772b3f64d55646fe34841
BLAKE2b-256 checksum
How to use checksums
e0a56cbffb44acd3d380a2f888d5dcd6153f63c3e3b2528ddcc17afde5c44f55
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.2

Security for fakellm-recorder 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page