fakellm-recorder
Record real OpenAI/Anthropic API traffic and emit fakellm fakellm.yaml rules automatically.
VCR-style zero-effort capture, combined with fakellm's editable, error-injectable
YAML rules. Run your real code against the real API once through the proxy, get a
fakellm.yaml out, hand-edit it to add the error paths recordings can't capture,
and commit it.
Install
pip install fakellm-recorder # once published
# or, from source:
pip install -e .
The loop
1. Record. Start the proxy and point your SDK's base_url at it, then run
your existing test or script once.
fakellm-recorder proxy --upstream auto --out sessions/run1.jsonl
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8888/v1", api_key="sk-...real-key...")
# run_my_agent(client, ...) # real traffic flows through and is recorded
The proxy forwards byte-faithfully to the real upstream, so your code behaves exactly as in production. Credentials are stripped before anything is written to disk; prompts/completions are scrubbed for emails and obvious keys by default.
2. Emit. Turn the recorded session into rules.
fakellm-recorder emit sessions/run1.jsonl --out fakellm.yaml --match-strictness balanced
3. Edit + commit. Add the 429s/500s/malformed responses recordings can't
capture, then commit fakellm.yaml alongside your tests. At replay time it's
plain fakellm — no recorder needed.
fakellm serve # from the fakellm package
Match strictness
The hard part is choosing a messages_contain substring specific enough to fire
on the right turn but loose enough not to only match that exact transcript. The
emitter ranks candidate n-grams by inverse frequency across the whole session, so
shared boilerplate (e.g. "You are a helpful assistant") is never chosen.
| Mode | What it emits |
|---|---|
loose |
mostly turn: + a model glob (gpt-4*); "any response will do" tests |
balanced (default) |
turn: + one distinctive substring (or a tool_result_contains anchor on post-tool turns) |
strict |
turn + substring + exact model_matches + tools_include; closest to a faithful replay |
Lint
A standalone check for unreachable rules (shadowed by an earlier first-match),
unknown condition/response keys, and responses that set neither content nor
tool_calls unintentionally.
fakellm-recorder lint fakellm.yaml
Security
- Credentials are never persisted. Header capture is allowlist-based, so
Authorization/x-api-key/ auth headers are dropped at capture time even if PII scrubbing is disabled. - PII scrubbing is on by default (emails, OpenAI/Anthropic keys, bearer
tokens) because the generated YAML is a commitable artifact. Disable with
--no-scrub(credentials are still stripped). Add your own regexes in code viaScrubber(custom_patterns=[...]).
Caveats
- fakellm is a new, single-author beta (0.3.x). Emitted files are stamped with the targeted config version; pin a fakellm version and expect schema churn.
- Streaming reconstruction handles both SSE dialects and keeps raw events for a future chunk-fidelity replay mode, but assembly of exotic event orderings may need tuning — the raw events are retained so you can fix assembly without re-recording.
- Single worker. fakellm stores state in process memory; keep both it and this proxy single-worker.
License
MIT
Metadata
Release files for fakellm-recorder 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| fakellm_recorder-0.1.0.tar.gz | 15.9 kB | Without provenance | Details for fakellm_recorder-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| fakellm_recorder-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for fakellm_recorder-0.1.0-py3-none-any.whl |
Total release size: 32.9 kB
Release files / fakellm_recorder-0.1.0.tar.gz
| Download URL | fakellm_recorder-0.1.0.tar.gz |
|---|---|
| Size | 15.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7f828573bd39a2e95050d8aedf2e6cfe82885d4508c18d36b162695b247dd76a
|
|
BLAKE2b-256 checksum How to use checksums |
372a3d828cb495502c9b9cb7d29a53a957ede9991651f6dcdd57ad227b97eea7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Release files / fakellm_recorder-0.1.0-py3-none-any.whl
| Download URL | fakellm_recorder-0.1.0-py3-none-any.whl |
|---|---|
| Size | 17.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9ef96e24e22231c3ef3330812cd7aba185b63548ceb772b3f64d55646fe34841
|
|
BLAKE2b-256 checksum How to use checksums |
e0a56cbffb44acd3d380a2f888d5dcd6153f63c3e3b2528ddcc17afde5c44f55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Security for fakellm-recorder 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.