graphrag-eval 6.4.0

For assessing question answering systems' final answers and intermediate steps, against a given set of questions, reference answers and steps.

QA Evaluation

This is a Python library for assessing the quality of question-answering systems, such as systems built with LLM-based agents. It is agnostic to the agent implementation and the LLM it uses.

The evaluation is based on a user-provided reference dataset containing queries, reference responses, and optional reference steps, such as expected tool uses. The evaluator compares these references with the agent's actual responses and executed steps. Reference steps can be grouped to allow some expected steps to occur in any order.

The library provides built-in evaluation metrics and supports user-defined custom metrics (§ Metrics).

Documentation

• Quickstart
• Metrics
• Configuration
• Input
• Output

Maintainers

Developed and maintained by Graphwise. For issues and feature requests, please open a GitHub issue.

License

Apache-2.0 License. See the LICENSE file for details.