llm_recorder

llm_recorder is a basic observability and debugging tool for language model apps. It stores interactions with LLMs in a local directory and provides a way to replay them. It is particularly useful for debugging chained LLM calls.

With llm_recorder, you can:

• Save every request/response pair to a directory.
• Replay part of previously recorded interactions instead of making live LLM calls.

You can use it to:

• Inspect LLM responses and requests that lead to them
• Record an execution path that leads to a specific response and then replay it and see how the application behaves downstream
• Modify the recorded responses before replaying them

Installation

pip install llm_recorder

Getting Started

For a quick start use enable_replay_mode; it monkey patches litellm to record and replay responses.

llm_recorder records any calls to litellm.completion and their responses and stores them in the directory specified by store_path.

Specify replay_count to replay previously recorded interactions (by default no interactions are replayed). Once the replayed interactions are exhausted, llm_recorder falls back to live LLM calls.

The store_path is cleaned up at the start of each run, but interactions from the previous run are read before that cleanup.

Basic Example

from llm_recorder import enable_replay_mode

import litellm


enable_replay_mode(store_path="saves")

print("After:", litellm.completion.__name__)

# Now just use litellm as usual:

response = litellm.completion(
    model="openai/gpt-4o",
    messages=[{ "content": "Hello, how are you?","role": "user"}]
)

Then have a look at the 'saves' directory to see the recorded interactions.

Chained Calls Example

See examples/chained_calls.py for an example of how to use llm_recorder to record and replay chained LLM calls. I saved the first interaction in examples/saves/chained_calls/ and the example replays it before making live calls.

Advanced Usage

For more fine-grained control instantiate and use the LiteLLMRecorder class directly. See examples/direct_recorder.py.

Limitations

• Direct support for SDKs (not through litellm) and the recording HTTP client is experimental.
• Only supports synchronous calls.

Contributing

Contributions are welcome! Please open issues or pull requests on GitHub.

License

This project is licensed under the MIT License.