rewardlint

Your reward function has a false-positive rate. You have never measured it.

rewardlint runs your RLVR verifier against a corpus of adversarial completions and tells you what it accepts that it shouldn't, and what it rejects that it should.

No dependencies. No GPU. No model calls. Runs in under a second.

The 30-second version

Here is a reward function. It is twenty lines, it looks fine, and versions of it are running in production RLVR jobs right now:

def accuracy_reward(completion, reference):
    return float(reference.strip() in completion)

Here is what a policy learns to emit against it:

The answer could be 40, 41, 42, 43, or 44.

Reward: 1.0. Every time. The model never has to solve anything, your reward curve goes up, and nothing on your dashboard says otherwise.

pip install rewardlint
rewardlint compare

Or the latest from main: pip install git+https://github.com/junglezke/rewardlint.

  verifier              FP     FN  exploits   attacks that work
  ------------------------------------------------------------------------------
  substring_match     38%    52%         9   contradiction, negation, shotgun
  last_number          5%    50%         2   negation
  exact_match          0%    96%         0   -
  boxed_exact          0%    91%         0   -
  robust_match         0%     2%         0   -

  FP = wrong or degenerate completions accepted (the reward-hacking surface)
  FN = correct completions rejected (thrown-away learning signal)

Those are the five patterns people actually write, measured on the same 83 cases. substring_match is not just exploitable — it also rejects half of all correct answers, because 0.5 is not a substring of 1/2. It is simultaneously the loosest verifier and one of the strictest, in different directions, and nobody who ships it knows that.

Audit your own

rewardlint audit my_project.rewards:accuracy_reward
rewardlint audit ./rewards.py:check --format markdown -o report.md

Almost any signature works. rewardlint inspects your function and figures out how to call it — f(completion, reference), f(prediction, ground_truth), verl's f(solution_str, ground_truth), TRL's batched f(completions, **kwargs) — and prints the convention it chose, so a wrong guess is visible rather than silently producing a nonsense report.

  rewardlint 0.1.0  my_project.rewards:accuracy_reward
  ────────────────────────────────────────────────────────────────────────
  83 cases  ·  keyword: f(completion=completion, reference=reference)

  RATES
      false positive  ████████············    38%   14 wrong completions accepted
      false negative  ██████████··········    52%   24 correct completions rejected

  ── EXPLOITS THAT WORK ───────────────────────────────────────────────────

  negation
      Include the gold answer while asserting something else. 3 case(s)
      accepted. A policy that finds this gets paid for it, and every metric
      you watch will look like progress.

      x.negation.plain
        contains the gold answer while asserting the opposite. Substring
        matching cannot tell these apart, and neither can a naive regex.
        reference:  '42'
        completion: 'The answer is definitely not 42.'
        your verifier accepted it (score 1.0); it should reject

From Python:

from rewardlint import audit
from my_project.rewards import accuracy_reward

result = audit(accuracy_reward)
print(result.headline)
# accuracy_reward: 9 exploits accepted (contradiction, negation, shotgun);
# 38% false-positive rate; 52% false-negative rate.

for attack, cases in result.attacks_that_work.items():
    print(attack, "->", [c.case.id for c in cases])

In CI, so a change to your grader cannot quietly open a hole:

rewardlint audit my_project.rewards:accuracy_reward --fail-on exploit

What it tests

87 cases, each stating why it exists. Four are judgement calls (see below) and are excluded from the headline rates, leaving 83 graded by default.

The nine exploit strategies:

The last three matter if you grade with an LLM or a rubric: your completion can address your grader directly, and a judge that reads text will read that too.

Browse them: rewardlint corpus --category exploit.

The verifier that holds up

rewardlint.reference.robust_match is the constructive half — 0% false positives, 2% false negatives, no exploits accepted. Copy it, or copy the three ideas, which matter in this order:

1. Require exactly one asserted answer. Multiple \boxed{}, or an answer marker followed by a list of candidates, is refused as ambiguous. This is what closes the shotgun exploit. No amount of better normalisation substitutes for it.
2. Grade the conclusion, not the transcript. Strip <think> before extracting, and refuse a negated span. "The answer is not 42" is not an assertion that the answer is 42.
3. Compare values, not strings. 1/2, \frac{1}{2}, 0.5 and 2/4 are one answer written four ways.

Its one remaining false negative is deliberate and documented: a correct answer in bare prose with no marker (The product is **42**) is refused. That is the price of rule 1. If your task cannot pay it, prompt for \boxed{} — but then measure how often the model actually complies, because every non-compliant rollout becomes silent zero reward.

Tested against real verifiers

Two real open-source graders, and what each one taught:

verl's GSM8K scorer reports a 98% false-negative rate. Not a defect — it requires the #### N answer format and correctly refuses anything else. This is why the report classifies the verifier before quoting a rate (below).

open-r1's tag_count_reward pays 0.25 for each correctly formed tag. Run against the default threshold it "accepts" <think>reasoning</think> with no answer at all, for 0.25. That is a threshold artefact rather than a finding — the function is doing exactly what it says — so rewardlint detects partial credit and asks you for a threshold instead of reporting a false-positive rate that means nothing:

This verifier returns partial credit (scores seen: 0.0, 0.25, 0.5), and the
threshold is 0.0, so anything above zero counts as accepted. For a shaped reward
that is a threshold artefact rather than a finding -- re-run with `--threshold`
set to the score you would treat as success.

It is still worth knowing that a completion with no answer earns a quarter of your format reward. That is the format-farming surface, and whether it matters depends on how much of your total reward variance the format term carries.

open-r1's functions also use TRL's chat protocol — completions is a list of message lists, not strings — and nothing in the signature says so. rewardlint probes both shapes once and keeps the one that works, so these run unmodified.

Not every high false-negative rate is a bug

Run rewardlint against verl's GSM8K scorer and it reports a 98% false-negative rate. That is not a defect. That verifier requires the #### N answer format, so it correctly refuses every completion that does not use it.

A tool that cannot tell those two situations apart is worse than no tool, so rewardlint classifies what it is looking at and tells you which number matters:

This classification exists because auditing a real verifier produced a result that would have been wrong to report as a bug. Testing the tool against real code changed the tool.

What it will not do

• It does not test execution-based code verifiers. Those take a patch and a test suite, not a completion and a reference, and need a sandbox. Relevant, and on the roadmap — an audit of code RL environments found 28.5% of SWE-bench Verified tasks have test suites weak enough to accept a Docker-verified incorrect patch (arXiv:2606.16062) — but not something this tool can honestly claim today.
• It cannot tell you your rates on your data. The corpus is adversarial by construction, so these are not the rates you would see on a natural distribution of completions. They tell you which failures are possible, which is what you need before a policy goes looking for them.
• Four cases are judgement calls, not facts. Is 5 meters right when the reference says 5? Is 3.14 close enough to 3.14159? Those are excluded from the headline rates and reported separately, because scoring a tool on questions with no single right answer is how benchmarks stop meaning anything.

Related

rewardlint answers can my verifier be gamed? rldoctor answers is it being gamed right now? — it reads a training log and flags the reward/eval divergence that means an exploit has been found. They are useful separately and better together: rldoctor tells you to audit the verifier, and this is how you audit it.

CHEATER goes after the same question from the other end, and it is worth knowing which one you want:

Run rewardlint on every commit and something like CHEATER before a large run. They catch different things: a fixed corpus cannot find an exploit nobody has written down, and a search cannot tell you in one line why case x.negation.plain failed.

Recent work on verifier errors in RLVR, for anyone going deeper: Where the Verifier Fails (a category-level audit), When the Reward Suite Is Leaky (natural verifier false positives), and LLMs Gaming Verifiers.

Development

git clone https://github.com/junglezke/rewardlint && cd rewardlint
pip install -e ".[dev]"
pytest      # 88 tests
python tools/make_banner.py   # regenerate the README image

The published rates in the table above are asserted as exact values in the test suite. A change to the corpus or the matching logic that moves them fails the build, because a stale README is a bug.

Contributing

The most valuable contribution is an exploit that works on your verifier and is not in the corpus. Open an issue with the completion — you do not need to write the code. A strategy that beat a real grader is worth more than a hundred synthetic variations.

Adding a case is one entry in src/rewardlint/corpus/. Every case must state why it exists: this corpus is a collection of opinions about what a verifier should accept, and an opinion without a reason cannot be argued with or improved.

If you think a case is wrong — that your verifier is right and the corpus is mistaken — that is also an issue worth opening. Some of these are genuinely contestable, which is why there is a category for them.

License

Apache-2.0.