bohrin

Check the grader of an RL coding environment before you train on it.

In a coding environment, a grader decides whether each submission solved the task, and it is the only thing the model learns from. If it pays for code that does not do the work, the model learns to not do the work.

Bohrin gives you the submissions a correct grader must reject — each one wrong by construction — and scores how many your grader caught.

Install

$ pip install bohrin

Use

from bohrin.ir.task import Task
from bohrin.mutate.battery import battery
from bohrin.scoring.coverage import Attempt, coverage_score

reference = "def add(a, b):\n    return a + b\n"
task = Task(id="add", prompt="Write add(a, b) that returns the sum.", reference=reference)


def my_grader(task, submission):
    # A common weak shape: checks that the function exists and runs, not what it returns.
    namespace = {}
    try:
        exec(submission, namespace)
        namespace["add"](2, 3)
        return True
    except Exception:
        return False


attempts = [
    Attempt(task.id, candidate, accepted=my_grader(task, candidate.payload)) for candidate in battery(task).candidates
]
print(coverage_score(attempts))

COVERAGE SCORE: 75 / 100   95% CI 30–95   3 of 4 caught   categories: 4 of 5 (battery 1)

That grader rejects an empty submission, a truncated reply and a constant, and pays for the reference with every function body replaced by pass — it runs, so the grader never notices it returns nothing.

What is tried

A submission is called wrong only when that is established without asking your grader. One that cannot be shown to be wrong — a refusal on a task with no declared answer, say — is a lead: tried and shown to you, never counted. See docs/SPEC.md for the exact rules.

License

Apache-2.0.