slm_eval

An evaluation suite for small language models (SLMs). It ships a small, hand-picked set of benchmarks, each chosen because it can be scored objectively, reproducibly, and on a single CPU machine.

Maintained by TheLimeDev (Hugging Face: @Monster-Code).

The evaluation engine, model backends, filters and metrics come from the MIT-licensed LM Evaluation Harness by EleutherAI. All ~14,000 of its task definitions were removed. Only the benchmarks below remain, so anything you already know about running the engine still applies — the task registry is different.

The supported benchmarks

Task name	Benchmark	What it measures	Data source	Licence
hellaswag	HellaSwag	Commonsense continuation of a described activity	Rowan/hellaswag	MIT
arc_easy	ARC Easy	Grade-school science questions	allenai/ai2_arc (ARC-Easy)	CC-BY-SA-4.0
arc_challenge	ARC Challenge	Harder science questions that retrieval alone does not solve	allenai/ai2_arc (ARC-Challenge)	CC-BY-SA-4.0
arithmark_3_0	ArithMark 3.0	Arithmetic continuation, resistant to memorisation	AxiomicLabs/Arithmark-3.0	Apache-2.0
arithmark_2	ArithMark 2.0	The predecessor of ArithMark 3.0	AxiomicLabs/ArithMark-2.0	Apache-2.0
math_500	MATH-500	Competition mathematics needing multi-step reasoning	HuggingFaceH4/MATH-500	MIT
gpqa_diamond	GPQA Diamond	Graduate-level "Google-proof" science questions	Idavidrein/gpqa, mirror fingertap/GPQA-Diamond	CC-BY-4.0
bfcl_v4	BFCL v4	Function calling and JSON-schema adherence	bfcl-eval package data	Apache-2.0
int_index	Int Index	A composite score over HellaSwag, ARC and ArithMark-3	derived, not a dataset	n/a

int_index is a group, not a dataset: it evaluates its component benchmarks and then combines them with the published formula. See slm_eval/tasks/slm/int_index/README.md.

Every licence here is usable commercially

No benchmark in this suite is NonCommercial (NC) or NoDerivatives (ND), and every benchmark rests on a licence value that names a specific commercial-use-safe licence. Two benchmarks carry obligations, and both are met by this repository:

• ARC is CC-BY-SA-4.0. Attribution is required, and redistributing a modified copy of the ARC data must stay under CC-BY-SA-4.0. This repository never redistributes the data: the task downloads it from Hugging Face at run time. The attribution is in NOTICE and in the task README.
• GPQA is CC-BY-4.0. Attribution is required. The public GPQA repository is gated, so a Hugging Face token helps; without one the task falls back to an open mirror and says so in the log. Attribution is in NOTICE.

Full research, with sources checked against the live dataset metadata, is in LICENSE_NOTES.md.

Benchmarks that were removed, and why

Benchmark	Reason
GSM-Symbolic	CC-BY-NC-ND-4.0: NonCommercial and NoDerivatives
AgentIF	CC-BY-NC-4.0: NonCommercial
WildBench	Subjective: needs an LLM judge, so results are biased and costly
BrowseComp	Too complex: needs a live browsing agent
SWE-bench Lite	Too heavy: needs a full Docker sandbox per sample
PIQA	The dataset card declares no licence, so commercial use cannot be verified
LiveCodeBench	The card declares only the unversioned cc tag, and the problems come from third-party contest sites
BananaMind Base Bench 1.1	The dataset card declares no licence, and the repository is gated

PIQA is a published component of the Int Index. Because it is not shipped here, the Int Index is computed from HellaSwag, ARC and ArithMark-3 with a denominator of 2.65, which is what the published method prescribes for a missing component.

LiveCodeBench is the only removal that costs a whole capability: the suite no longer ships a coding benchmark. LICENSE_NOTES.md records the full reasoning.

Install

pip install -e ".[hf,api,benchmarks]"

hf provides the HuggingFace transformers backend. benchmarks adds the scoring dependencies the tasks need (math_verify, langdetect, and friends).

Run

# Everything at once. `slm` is a tag group: every task tagged `slm` runs.
python3 -m slm_eval \
  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks slm --device cpu --batch_size 8

# One benchmark.
python3 -m slm_eval \
  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks arithmark_3_0 --device cpu

# A quick wiring check on a few examples.
python3 -m slm_eval \
  --model hf --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks hellaswag --limit 16 --device cpu

List the available tasks:

python3 -m slm_eval ls tasks

The Int Index

The Int Index is the composite score published on the Open SLM Leaderboard:

N(score, chance) = 100 x (score - chance) / (100 - chance)

Int Index = (HellaSwag + ARC + PIQA + 0.65 x ArithMark-3) / 3.65

Run the components in one pass, then compute the index from the saved results:

python3 -m slm_eval --model hf \
  --model_args pretrained=HuggingFaceTB/SmolLM2-135M \
  --tasks int_index --device cpu --output_path logs/tiny-run

# The tracker writes logs/tiny-run/<model name>/results_<timestamp>.json,
# so let the shell expand the path.
python3 -m slm_eval.int_index --results logs/tiny-run/*/results_*.json

The scale is small: chance level scores about 0, and the strongest reported sub-10B models reach roughly 5. It is not a percentage. Each component is rescaled so that guessing maps to 0 and a perfect score maps to 100.

Average percentage (eval_avg)

eval_avg is the plain number: the average of the per-benchmark percentages in a result file, with no weighting and no chance correction.

python3 -m slm_eval.eval_avg --results logs/tiny-run/*/results_*.json

It picks exactly one metric per benchmark, in this order of preference: acc_norm, acc, exact_match, function_name_match, ast_match, valid_json. If a benchmark reports none of those, it uses its alphabetically first scored metric. acc_norm is preferred because raw acc carries a token-length bias.

Diagnostics such as unparseable and output_emitted measure the harness, not the model, so they are never chosen. A benchmark with no usable metric is reported as skipped and leaves the average, instead of counting as zero. The printed table names the metric used for every benchmark, so the number stays auditable.

eval_avg is a plain mean, so it is comparable only between runs over the same benchmark set. It is not the Int Index, which is chance-corrected, weighted and published.

Scoring tiers

bfcl_v4 is built in two tiers:

• Tier 1 always works. It builds the real prompt, generates a real response, and computes a deterministic format or rule-based metric.
• Tier 2 is the official metric. It is implemented, but gated behind an explicit prerequisite. If you request it without the prerequisite, the task raises a RuntimeError that names exactly what is missing. It never silently returns zero.

Task	Tier 1	Tier 2
bfcl_v4	valid_json, function_name_match	ast_match, needs --metadata enable_tier2_metric=True

No shipped benchmark executes generated code: BFCL Tier 2 runs the official AST checker over the predicted call. The engine still enforces --confirm_run_unsafe_code for any task that marks itself unsafe, so a task added later cannot execute code without that confirmation. Per-task limits and prerequisites are documented in each task's README.md.

A note on the inherited documentation

docs/, scripts/ and examples/ document the engine, the CLI and the model backends, which this suite does not change. Some examples inside them name benchmarks that were removed here. For the tasks that do exist, slm_eval/tasks/slm/CONVENTIONS.md and the per-task README.md files are authoritative.

Layout

slm_eval/                     evaluation engine
  int_index.py                the Int Index composite calculator
  eval_avg.py                 the plain average-percentage calculator
  tasks/
    CONVENTIONS.md            the task authoring contract
    slm/                      the supported benchmarks live here
      hellaswag/ arc_easy/ arc_challenge/
      arithmark_2/ arithmark_3_0/
      math_500/ gpqa_diamond/ bfcl_v4/
      int_index/              composite group (not a dataset)
tests/                        tests for this suite
LICENSE_NOTES.md              licence research for every benchmark
NOTICE                        attribution for the CCPL benchmarks and the engine

Add or change a task

Read slm_eval/tasks/slm/CONVENTIONS.md. Task discovery globs slm_eval/tasks/**/*.yaml, so a new directory with a YAML file is enough.

Licence

The engine is MIT, from EleutherAI. LICENSE.md keeps that notice. See NOTICE for the attribution required by the CC-BY-4.0 and CC-BY-SA-4.0 datasets, and LICENSE_NOTES.md for the per-benchmark research.