SmallBench
Small, simple agent task environments for training and evaluation.
Designed to challenge a broad spectrum of lm-agent abilities.
Spinning Up - Use
uv add smallbench
or
pip install smallbench
Spinning Up - Dev
uv venv smallbench-dev
source smallbench-dev/bin/activate
uv sync
uv run ruff format .
Easy Benchmarks
BigCodeBench - Agent Harness
This benchmark provides a stateful environment for lm-based agents to solve coding problems from the BigCodeBench dataset. Agents are given a scratchpad (soon), a way to prepare and use unit tests, editing handlers, and a way to submit their solution.
Please see the BigCodeBench page for more information about the underlying dataset.
Get Started
Local
add GROQ_API_KEY and any other API keys supported by the apropos-ai library to the .env file.
- Note: Groq, Google, and possibly other providers offer free tiers.
If you use a Docker backend, ensure you have the Docker app running. If you use Modal, please add all necessary credentials.
Then, run the test script:
uv run python -m src.smallbench.benchmarks.bcb_a.test
Colab
Check out the Colab if you prefer to run the benchmark in the cloud.
Medium Benchmarks
TBD
Hard Benchmarks
TBD
Difficult Benchmarks
TBD
Scores
BigCodeBench - Agent Harness (ReAct)
| LM | Number Correct - Train | Number Correct - Test | Sample Size | Avg. Cost Per Run |
|---|---|---|---|---|
| gpt-4o-mini-2024-07-18-ft* | ??? | 21 | 100 | $0.006 |
| gpt-4o-2024-08-06 | 17 | 18 | 100 | $0.057 |
| gpt-4o-mini-2024-07-18-ft** | ??? | 13 | 100 | $0.006 |
| deepseek-v2.5 | 12 | ??? | 100 | $0.0029 |
| gpt-4o-mini-2024-07-18 | 12 | 8 | 100 | $0.003 |
| gemini-1.5-flash-latest | 6 | 7 | 100 | $0.0018 |
- "*" fine-tuned on a minimal subset (<500k tokens) of trajectories using a variation of the Filtered Behavioral Cloning approach (2024-09-17).
- "**" fine-tuned on a minimal subset (<500k tokens) of trajectories using a variation of the Filtered Behavioral Cloning approach (2024-09-08).
Animation credits: ZZ
1 maintainer
Metadata
1 maintainer
Release files for smallbench 0.2.28
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| smallbench-0.2.28.tar.gz | 3.5 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| smallbench-0.2.28-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.6 MB
Release files / smallbench-0.2.28.tar.gz
| Download URL | smallbench-0.2.28.tar.gz |
|---|---|
| Size | 3.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
37d62491a2aded238b802a8fee5eba5ee9d372b6cf6c9114174bcaec461c6f57
|
|
BLAKE2b-256 checksum How to use checksums |
4297354bc5debae4e2e712448984ed0ac582d280539b9c8278e37d24d800ab93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.11.8
|
Release files / smallbench-0.2.28-py3-none-any.whl
| Download URL | smallbench-0.2.28-py3-none-any.whl |
|---|---|
| Size | 25.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ba24d92c2b9da60c55ebf8d7bd46abbbc8a8d664f433bd2e59bd461efc482c2a
|
|
BLAKE2b-256 checksum How to use checksums |
51ddc140f67beac26908e6a160433ccfbda93d2ed2e5da181c3074715da7bbcf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.11.8
|
Security for smallbench 0.2.28
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
