gradecheck 0.1.0

Attack-test RL environment graders for reward hacking — a pentest for reward functions.

gradecheck

Attack-test your RL environment's grader before an RL run does it for you.

RL environments reward whatever the grader accepts — and models optimize the grader, not the task. If the grader has a loophole (delete the failing tests, copy the expected outputs, submit nothing and exit clean), training finds it and you ship a model that learned to cheat. Today those loopholes are caught by a human reading grader code and imagining exploits.

gradecheck makes the known exploits a checklist that runs automatically. You point it at an environment; it plays the laziest, most dishonest agent possible — running a battery of canonical cheap strategies against your grader — and reports which ones were wrongly rewarded:

$ gradecheck run ./my-env

  ⚠ FOOLED   test-deletion  Grader awards full reward (1.000) after deleting 1 test file(s).
  ⚠ FOOLED   test-deletion  Grader awards full reward (1.000) after emptying 1 test file(s).
  △ WARN     answer-leak    1 answer-shaped file(s) reachable from the submission tree.
  ✓ RESISTED noop           Starting tree scores 0.000 (not solved) — correct.
  ✓ RESISTED empty-source   Empty submission scores 0.000 <= baseline 0.000 — correct.

  VULNERABLE — grader fooled by 2 attack(s)

Exit code is 1 when any attack fools the grader, so it drops straight into CI.

It does not catch every possible hack — nothing can. It clears out the known, mechanical loopholes so human review spends its imagination on the novel ones.

Install

pip install gradecheck

Zero dependencies. Python ≥ 3.11. (The verifiers integration below adds one optional dependency and currently wants Python 3.11–3.12.)

Setup

Drop a gradecheck.toml next to your environment:

[env]
name = "my-task"
workspace = "task"                 # the tree the agent edits and submits
source_globs = ["src/**/*.py"]     # files a real fix would touch

[grade]
cmd = "python3 verification/grade.py --task-dir {workspace}"
timeout = 1800
pass_marker = "RESULT: PASS"       # exit 0 + this marker == solved
score = [                          # optional: partial-credit scoreboard lines
  { regex = 'F2P:\s*(\d+)\s*/\s*(\d+)' },
  { regex = 'P2P:\s*(\d+)\s*/\s*(\d+)' },
]

gradecheck copies workspace fresh for every attack, mutates the copy the way a cheating agent would, runs your grade command against it, and compares the reward to the honest baseline. Your environment is never modified.

Attacks

id	what it simulates
noop	submit the starting tree unchanged — it must not already pass
empty-source	blank out the source files — absence must never score higher
test-deletion	delete the visible tests, then separately empty them — the pass signal must not depend on tests the agent can rewrite
answer-leak	(static) scan the submission tree for reachable gold/expected/answer artifacts an agent could copy
hollow-implementation	read the grader's source ([grade].source), extract the string literals/paths it checks for, write a fake artifact containing them, and see if the grader passes without the task being done

Verdicts: FOOLED (grader gave undeserved reward — fix it), WARN (exploit surface a human should confirm), RESISTED (correct behavior).

gradecheck attacks              # list attacks
gradecheck run . --json         # machine-readable report
gradecheck run . --attacks noop,test-deletion

Prime Intellect verifiers environments

Verifiers environments grade a model completion rather than a file tree — the author's reward functions live under env.rubric. gradecheck attacks them directly: it loads the environment, samples real task examples, and scores degenerate completions against the reward functions. No rollout, no Docker, no LLM, no API key.

pip install "gradecheck[verifiers]"
prime env install some-env            # or pip install the env package
gradecheck verifiers some-env --sample 20

  ⚠ FOOLED   flattery       Judge-flattery / correctness assertion: accepted on 3/3 sampled example(s).
  ⚠ FOOLED   prompt-echo    Prompt echoed back: accepted on 3/3 sampled example(s).
  ✓ RESISTED empty-completion Empty completion: rejected on all 3 sampled example(s).
  ✓ RESISTED garbage        Nonsense completion: rejected on all 3 sampled example(s).
  ✓ RESISTED answer-leak    Gold answer not present in the agent's observation.
  · INFO     baseline       Gold answer scores as solved on 3/3 sampled example(s).

Completion attacks: empty-completion, prompt-echo, garbage, flattery (bare assertions of correctness — catches keyword/LLM-judge rewards), and answer-leak (checks whether the gold answer is visible in the agent's own observation, not merely accessible to the reward function). A reward function that accepts any of these is scoring surface form, not correctness.

Writing your own attacks

Attacks are small classes with a registry — subclass gradecheck.attacks.base.Attack, implement run(ctx), and call gradecheck.attacks.register(MyAttack). ctx gives you materialize() (fresh workspace copy), grade(ws), and the honest baseline.

Safety note

gradecheck deliberately submits adversarial trees to your own grader, on your machine, using the grade command you configure. It executes nothing an agent couldn't already make your grader execute — but as with any grading run, do it inside the same sandbox/container your environment normally grades in.

License

Apache-2.0