ape-framework 2.1.0

Package for evaluating algebra problems using AI systems.

Algebra Problems Evaluator (APE)

What is APE?

APE is a framework that simplifies building and running mathematical benchmarks for AI systems.

It was initially meant as a way to evaluate mathematical problems from the field of algebra on whether they are suited to be a target goal for LLM reasoning research (hence the name). However it might be more useful to think about it the other way around - the subjects of the evaluation are LLMs or more broadly - AI systems and they are being evaluated on their ability to solve specific algebra problems. Problems, which solutions are hard to generate, but relatively easy to automatically check for correctness.

APE was created as a part of a Bachelor's project at the University of Warsaw.

Documentation

Full documentation is published at matikasp.github.io/Algebra-Problems-Evaluator. Good starting points (source under docs/):

• Installation and Quickstart — install APE and run a tiny benchmark end to end.
• Core Concepts — the pipeline and the pieces you implement.
• Building a Custom Benchmark — what you need to write for a complete benchmark (a DataModel, a SolutionAdapter, and a Checker) and how to run it.
• Developer Guide and API Reference — to understand or extend the framework itself.

Development setup

Install development dependencies:

pip install -e .[dev]

Install git hooks:

pre-commit install

Run all hooks manually:

pre-commit run --all-files