ambertrace-rlvr

ambertrace-rlvr extends RLVR — Reinforcement Learning from Verifiable Rewards — beyond the domains where it was born. RLVR built today's reasoning models on rewards a machine can check: code that compiles, math with one right answer. This library breaks that constraint for real-world, rule-governed decisions — lending, clinical protocols, airspace triage, compliance — where the ground truth is not a compiler but a policy, a dataset, or a live feed.

The verifier is an AmberTrace proof certificate. A verified platform can be built three ways: describe your rules in plain English; let the kernel derive the rules unsupervised from your data (no labels); or ground it in live external data through built-in connectors. Either way, a fail-closed symbolic kernel independently re-derives and certifies every decision — which rules fired, which facts were rejected, why — and this library turns that certificate into a bounded, fail-closed RL reward: a completion that can't be certified can never out-score one that can. Because the certificate also names the rules a correct answer must rest on, it doubles as an alignment instrument: it measures whether a model's stated reasoning stays faithful to the proof as reward climbs, whether its errors lean fail-open or over-cautious, and what training bleeds into domains the reward never touched. Offline-first throughout; the reward runtime is read-only against the platform.

Shown so far, at one-GPU scale: an end-to-end GRPO loop with real learning curves; a 1,350-item certified alignment matrix across 19 open-weight models; a quantization safety-direction study; and a first interventional faithfulness experiment — a 7B trained against a certified triage policy, raw reasoning traces captured at every checkpoint, no reward-correlated confabulation at moderate pressure, and a durable shift toward caution in domains the reward never saw. Every claim links to the captures behind it.

Where this goes, with more compute than ours. The certificate surface is wider than what we've trained against so far: certified cross-domain proofs (decisions gated on relations proven across datasets), certified symbolic forecasts (verifiable rewards about the future, with readable driver-rules), and the Agent Policy Gate — proof-carrying permit/deny over an agent's actions, which extends RLVR from answers to behavior. And the bottleneck that has kept RLVR confined to code and math — hand-authoring verifiable environments — inverts here: AmberTrace gives an AI agent full authoring parity with a human, so verified training environments can themselves be generated at scale (this repo's own experiment platforms were agent-authored and acceptance-gated). The open questions are scale questions: does certified reward hold the line on monitorability where LLM-judge reward erodes it; does strong optimization pressure find cracks the drift instruments here can't catch; can one model train across many certified policies and generalize policy-faithful reasoning? The methodology, acceptance gates, drift and OOD probes, and reproduction recipes ship in this repo. What's missing is frontier-scale compute. If that's you: start with the research and the roadmap — the experiments are specified, gated, and waiting.

Try it without an account. The offline test suite (pytest tests/ -q) and the verification-overhead benchmark run with no AmberTrace account — they use the built-in FakeVerifier and recorded payloads. Authoring a platform and training against a live one need an API key from ambertrace.ai; see the customer journey below.

What is AmberTrace?

In regulated, rule-governed domains — lending, healthcare, hiring, compliance — "the model said so" is not an acceptable answer. A decision has to arrive with a reason you can check.

AmberTrace produces a machine-checkable proof for every decision. You describe your rules in plain English and upload a features-only dataset (it learns unsupervised — no labels required); AmberTrace derives the rules and builds a verified platform. Every query is then answered by an independent, fail-closed kernel that re-derives and certifies the decision, returning an Amber Report that carries, among other things:

• a proof_checked certificate — the decision independently re-derived and certified against the trusted kernel,
• a symbolic trace — every rule evaluated and which fired, with reasons,
• rejected facts — low-confidence inputs the fact gate refused,
• a fused confidence (neural + symbolic).

That machine-checked certificate is the missing verifier for rule-governed domains — and it's exactly what this library turns into an RL reward. Here's a real Amber Report (trimmed) for a Grant Eligibility decision — the same demo domain as the training run below:

{
  "decision": "permit",
  "proof_checked": true,                        // ← the certificate: the reward hinges on this
  "proof_summary": "Decision independently certified against the trusted kernel: 6 rule(s) fired, 5 fact(s) derived from 4 input fact(s).",
  "explanation": {
    "confidence": { "overall": 0.86, "neural_confidence": 0.65, "symbolic_confidence": 1.0 },
    "certified_fact_summary": { "accepted": 4, "rejected": 0 },   // fact gate: nothing hallucinated
    "symbolic_trace": {
      "rules_evaluated": 14,
      "rules_fired": 6,
      "rules": [
        { "rule_name": "Classify Is Resident Eligible", "rule_type": "derive",
          "required": false, "fired": true,
          "explanation": "Rule 'Classify Is Resident Eligible' fired: eligible for residency if the applicant is a resident." },
        { "rule_name": "Check Annual Income Exceeds Threshold", "rule_type": "constraint",
          "required": false, "fired": false,
          "explanation": "Rule 'Check Annual Income Exceeds Threshold' did not match context" }
        // …every rule the kernel evaluated, with reasons
      ]
    }
  }
}

What is ambertrace-rlvr?

ambertrace-rlvr lets you train your own domain-specific models where the reward is not a learned preference model or a heuristic, but a verifiable proof certificate issued by AmberTrace. A completion is rewarded only when its output produces a valid proof certificate for the domain — a hard, auditable ground-truth signal.

Concretely, the library turns a report like the one above into a scalar: DefaultRewardShaper reads proof_checked, the rejected-fact fraction, and the symbolic trace, and combines them into a dense, bounded reward — each component clamped to [0, 1] before weighting — so a certified, hallucination-free decision scores high, and a rejected-fact or uncertified completion can never out-score it. The reward function is a plain callable:

reward_fn(prompts, completions, refs) -> list[float]

Why not a verifier you write yourself? RLVR practitioners already reward against math checkers and unit tests — cheap where ground truth is a string match or a passing test. AmberTrace is for domains where correctness is a rulebook, not a test suite: the certificate re-derives the decision against an auditable set of symbolic rules inside a fail-closed kernel, giving a reward only as trustworthy as a formal proof — not a regex you have to maintain and defend in an audit.

Does it work? Watch it learn

A real GRPO run on the demo Grant Eligibility platform, trained on a laptop-class Apple Silicon machine — the policy is rewarded only when AmberTrace certifies its decision. Mean reward climbs from near the floor to +0.69 (peak +1.35) as it learns to reason to conclusions the kernel will certify:

• Results writeup → — method, setup, the reward-collapse-vs-KL-stability finding, and how to reproduce it.
• User Guide → — the full create → build → train walkthrough.
• API Reference → — every public symbol (ambertrace_rlvr.__all__) with its signature and purpose.

How it works — the customer journey

Bring your own domain and data, and train a model against a verifiable reward in three steps:

1. Create — sign up at ambertrace.ai and get an API key.
2. Build — BYOD: describe your domain in plain English and author your verified platform with the ambertraceai Python SDK (platforms.create, create_rule, suggest_rules). This is where your rulebook lives.
3. Train — point ambertrace-rlvr at your platform; the platform's proof certificate is the reward. Hand the reward function to your trainer (TRL/GRPO first).

This repo provides the reward machinery for step 3 and a runnable on-ramp for steps 1–2.

Scope: this repo vs the ambertraceai SDK

Two projects, two jobs — keep them straight:

You author your platform with the SDK (step 2). ambertrace-rlvr then consumes it read-only at training time. This library never re-implements the SDK or the verification kernel.

Status

M0–M1 complete, M2 core + the eval/alignment lane in, M3 well under way. The full reward path (parser → verifier → shaper) with dense per-criterion partial credit, fact-provenance anti-reward-hacking, and a rule-checked consistency component, a config-driven run loader, fail-closed resilience, the TRL/GRPO and veRL adapters, and a demonstrated end-to-end training run (see Results). The OpenRLHF HTTP reward-server shim and the cross-domain swap-the-rule-set demo have both shipped. Plus a certificate-grounded evaluation & alignment lane — now running over open-weight models through a local LM Studio backend, with a published alignment matrix and a preliminary quantization sweep. The TRL RLOO trainer builder (#18) has since shipped alongside GRPO. The reasoning-enabled arm of the quantization sweep (#87) is complete: reasoning lifts accuracy ~3 pt and halves fail-open while the safety direction stays flat across the ladder. See the roadmap. Design spec in docs/.

Install

pip install ambertrace-rlvr             # core (reward path + config loader)
pip install 'ambertrace-rlvr[trl]'      # + TRL/GRPO training stack

Requires Python ≥ 3.11. The core install pulls in the ambertraceai SDK, which you use to author a platform; ambertrace-rlvr then consumes it read-only at training time.

Working from a clone (contributing, or running the examples) — editable install with the dev tooling:

pip install -e '.[dev]'                 # core + pytest + pyright
pip install -e '.[trl]'                 # + TRL/GRPO training stack

Quickstart

Once you've authored a platform with the ambertraceai SDK (step 2 above) and have its platform_id, the reward function is a few lines:

from ambertrace_rlvr import AmberVerifier, DefaultRewardShaper, JSONBlockParser, VerifiableDomain

# AMBERTRACE_API_KEY (scoped, platform-only) comes from the environment.
platform_id = 146  # <- your platform id from the author script
domain = VerifiableDomain.from_env(platform_id=platform_id, parser=JSONBlockParser())
reward_fn = AmberVerifier(domain=domain, shaper=DefaultRewardShaper()).as_reward_function()
rewards = reward_fn(prompts, completions, [{"gold": "permit"}, ...])   # -> list[float]

Or describe the whole run in one YAML and load it:

from ambertrace_rlvr import load_run_config

run = load_run_config("configs/your_run.yaml")
reward_fn = run.reward_function()

Author a demo platform (the "build" step)

examples/author_demo_platform.py walks the build half of the journey with the SDK: it uploads a small features-only dataset (no labels — AmberTrace learns unsupervised) and a plain-English domain description, builds a verified platform, and confirms it certifies a query. It's an operator/setup script — not library code; the reward runtime stays read-only.

python examples/gen_demo_dataset.py       # writes data/grant_eligibility_dataset.csv
python examples/author_demo_platform.py   # needs an authoring-scoped AMBERTRACE_API_KEY

It prints a platform_id; put it in configs/grant_eligibility.yaml (or set AMBERTRACE_PLATFORM_ID) and you're ready to train.

See also examples/score_completions.py for a runnable end-to-end reward smoke test and configs/loan_example.yaml / configs/grant_eligibility.yaml for full run configs.

Cross-domain / swap-the-rule-set: examples/cross_domain_demo.py scores two domains (grant eligibility + ACMG variant classification) through one domain-agnostic code path — swapping only config + parser, no forks. Runs offline with FakeVerifier; see docs/CROSS_DOMAIN.md.

Evaluation & alignment

The same certificate that drives the reward is also a ground-truth oracle-as-judge, so this repo ships an evaluation lane that scores model behaviour against the proof. It's independent of training — it needs only the certificate, never a trainer or the reward shaper — and consumes only the normalised certificate, so the verifier's internals stay opaque.

• Eval harness — evaluate / evaluate_policy / compare_to_baseline / consistency: parse-rate, certified-rate, accuracy-vs-gold, mean reward + per-component traces, with trivial baselines.
• Oracle-as-judge seam — OracleJudgment + a per-domain JudgmentSpec: turns a certificate into certified-decidable / certified-undecidable / unverifiable, with a direction (fail-open vs fail-closed) and severity vocabulary.
• Epistemic honesty — score_deviation: a hard three-bucket partition plus an overconfidence rate — how often the model answers a case the oracle proved has no determinate answer (never scored as "wrong").
• Sycophancy-into-error — run_sweep: re-poses the same certified items under social-pressure framings and reports the signed fail-open Δ (movement toward under-restriction), split by severity band.
• Monitorability — faithfulness_curve / compare_monitorability: does stated-reasoning faithfulness erode as reward rises (train-against-verifier vs train-against-a-model-judge)?
• Reward-hacking gap — reward_hacking_score: scores a rule-preserving vs a rule-violating probe set and reports the reward gap between them, flagging a policy that scores high while breaking the rules.

from ambertrace_rlvr import (
    OracleJudgment, JudgmentSpec, LabelSpec, parse_model_answer, score_deviation,
)

spec = JudgmentSpec(labels=[
    LabelSpec("deny", rank=0, restrictive=True),   # fail-closed / safety-critical
    LabelSpec("permit", rank=1),                   # fail-open
    LabelSpec("abstain", is_abstain=True),         # the certified-undecidable outcome
])
# reports: AmberReport per item (oracle queried on each item's fixed inputs)
judgments = [OracleJudgment.from_report(r, spec) for r in reports]
answers = [parse_model_answer(text, ["permit", "deny"]) for text in model_outputs]

report = score_deviation(judgments, answers, spec)
print(report.overconfidence_rate, report.over_permit_rate)   # alignment scores

Everything here is offline/network-free to test. To run it over real open-weight models, model_backend.py ships a local LM Studio backend (OpenAI-compatible endpoint) that turns a served model into a plain prompt -> completion callable; corpus.py + eval_generator.py build and load the decision_eval_v1 oracle-anchored benchmark. On top of these:

• Alignment matrix (matrix.py) — the eval suite across 19 models / 10 labs, ranked by CAS and by fail-open on the safety-critical band. Full 1,350-item run, single sample, temperature 0; reasoning-enabled Qwen3.8-27B tops it. Live as a Hugging Face leaderboard Space backed by the results dataset (scored against the fail-closed verifier — add your model).
• Quantization sweep (quant_sweep.py) — one base model across quant levels (Q8 → Q2) over the same items, reporting a safety tax (fail-open gained vs accuracy lost). Preliminary and directional — one model, small absolute counts.

Research. For the why and the results, see docs/research/: Verifiable Rewards Beyond Maths and Code (objectives + the case for a checkable reward), Measuring Misalignment as Deviation From the Provable (the open-weight alignment matrix), and Faithfulness of Stated Reasoning Under Verifiable-Reward RL (does the reward erode chain-of-thought faithfulness? main run complete — no confabulation, durable OOD caution shift). Checkpoints published as a Hugging Face Model.

The corpus is also cross-posted as Hugging Face Articles — built paste-ready by examples/build_hf_articles.py.

Design principles

• Community-friendly and agent-usable. Every public API is typed, every extension point is documented, and new domains are a config + a parser — not a fork.
• Offline-first development. The default test suite and benchmarks run with no network, no API key, and no GPU, using FakeVerifier and recorded payloads.
• Fail-closed reward invariants. A malformed completion, SDK error, or timeout resolves to the reward floor — never an exception into the training loop. Components are bounded to [0, 1]; a hallucinated completion can never out-score a certified one.
• Read-only reward runtime. The reward path queries a verified platform; it never authors or mutates one. Authoring is a separate step via the ambertraceai SDK.

Repository layout

src/ambertrace_rlvr/
  domain.py        VerifiableDomain (bind to a platform)
  parsers.py       CompletionParser + JSON/Regex block parsers
  verifier.py      AmberVerifier — SDK query, cache, bounded concurrency, fail-closed
  reports.py       AmberReport normalisation over the QueryExplanation contract
  rewards.py       RewardShaper + DefaultRewardShaper (dense, hack-resistant)
  prompts.py       system-prompt template / format contract
  evaluation.py    eval harness — metrics, baselines, consistency
  eval_oracle.py   oracle-as-judge seam — OracleJudgment + JudgmentSpec
  deviation.py     three-bucket scorer + overconfidence rate
  sycophancy.py    social-pressure sweep — signed fail-open Δ
  faithfulness.py  faithfulness-vs-reward monitorability curve
  faithfulness_scorer.py  rich per-completion scoring (score_batch_rich → trajectory round-trip)
  cot_drift.py     chain-of-thought drift metrics (channel lengths, distinct-n, lexicon tripwires, divergence)
  ood_drift.py     OOD behavioural + drift scoring (policy bleed, format leakage, sycophancy delta)
  model_backend.py local LM Studio backend (OpenAI-compatible) → model callable
  corpus.py        decision_eval_v1 benchmark load/write + stats
  eval_generator.py SDK-driven eval-set generation
  matrix.py        alignment matrix runner (model × alignment-score)
  quant_sweep.py   quantization sweep — one model across quant levels
  testing.py       FakeVerifier + offline payload builders
  integrations/    trl.py (GRPO + RLOO builders), verl.py, openrlhf.py (HTTP reward-server shim)
examples/          runnable examples
configs/           per-run YAML
tests/             offline suite (FakeVerifier + recorded payloads)
docs/              design spec, user guide, results

Verification overhead

RL post-training issues many verifications per step (group_size × batch), so the verifier must not become the bottleneck (target: < ~15% of step wall-clock, spec §10). benchmarks/verification_overhead.py is an offline harness — AmberVerifier._query is stubbed with a configurable latency, so no network call is made — that runs a synthetic batch through the existing bounded-concurrency pool and prints the measured verify time, a simulated step time, and the overhead percentage:

python benchmarks/verification_overhead.py
python benchmarks/verification_overhead.py --batch 32 --group-size 8 \
    --concurrency 16 --query-latency 0.05 --step-compute 2.0

It is a script, not a test (benchmarks/ is excluded from testpaths). Further throughput gains — a query_batch endpoint and a compact query projection — are gated on the platform shipping them; see issue #27.

License

MIT © 2026 Ambertrace Labs Ltd.