flow-benchmark-tools 1.0.0

A library containing LLM benchmarking tools

Flow Benchmark Tools

Create and run LLM benchmarks.

Installation

Just the library:

pip install flow-benchmark-tools

Library + Example benchmarks (see below):

pip install flow-benchmark-tools[examples]

Usage

• Create an agent by inheriting BenchmarkAgent and implementing the run_benchmark_case method.

• Create a Benchmark by compiling a list of BenchmarkCases. These can be read from a JSONL file.

• Associate agent and benchmark in a BenchmarkRun.

• Use a BenchmarkRunner to run your BenchmarkRun.

Running example benchmarks

Two end-to-end benchmark examples are provided in the examples folder: a LangChain RAG application and an OpenAI Assistant agent.

To run the LangChain RAG benchmark:

python src/examples/langchain_rag_agent.py

To run the OpenAI Assistant benchmark:

python src/examples/openai_assistant_agent.py

The benchmark cases are defined in data/rag_benchmark.jsonl.

The two examples follow the typical usage pattern of the library:

• define an agent by implementing the BenchmarkAgent interface and overriding the run_benchmark_case method (you can also override the before and after methods, if needed),
• create a set of benchmark cases, typically as a JSONL file such as data/rag_benchmark.jsonl,
• use a BenchmarkRunner to run the benchmark.