JevRAG

A common decision substrate for RAG (retrieval-augmented generation) pipelines — a shared abstraction, a swappable decision backend, and a calibration-first evaluation harness.

state → Decision → confidence → action

pip install jevrag
# or with uv:
uv add jevrag              # in a uv project
uvx jevrag --help          # run the CLI once, no install

See Quickstart for keys and a first run.

Per-primitive results and caveats live in docs/primitives.md; the summary table is Results at a glance.

Jump to: The idea · Quickstart · What's built · Full CLI · Results at a glance · The real pipeline · Reproducing this · Project layout · Per-primitive detail

The idea

Every RAG pipeline has moments where it has to decide something mid-flight: is this chunk of text a coherent unit or should it be split, is the evidence gathered so far enough to answer the question, which retrieved passages actually earn a slot in the context window, is a cached answer safe to reuse, does the generated answer actually stay grounded in the evidence. Most systems make these calls with a hardcoded threshold or a heuristic buried somewhere in the code, untested and unmeasured.

JevRAG's idea is to make each of these a first-class, explicit decision: hand a decision backend the relevant state, get back a typed answer with a calibrated confidence, and let the calling code decide what to do with that confidence rather than letting the model decide silently. The decision backend itself is swappable — a different model, a trained classifier, a simple rule — behind one interface. Jev, a decision-focused model from TypeSafe AI, is the first backend actually wired up; it is not the point of the project. The point is the substrate and, just as importantly, an evaluation harness that reports honestly on whether a decision's confidence means anything, rather than trusting a vendor's claim about it.

Where the five decisions sit in a RAG pipeline — this is the real, wired architecture (each box below is one Decision call). It's live: jevrag/pipeline.py chains all five end to end, verified on real documents (see The real pipeline in Results):

flowchart TD
    DOC[Document ingestion] --> CB{Chunk-Boundary}
    CB -->|split/merge decision| IDX[(Index)]

    Q[Query] --> RET[Retrieval]
    IDX --> RET
    RET --> CS{Context-Selection}
    CS -->|passage earns a slot?| SUFF{Sufficiency}
    SUFF -->|NO: retrieve another round| RET
    SUFF -->|YES| GEN[Generate answer]
    GEN --> AA{Answer-Abstain}
    AA -->|grounded| ANS[Answer]
    AA -->|not grounded| NOANS[No answer]

    Q --> CACHE[Cache lookup]
    CACHE --> CT{Cache-Trust}
    CT -->|safe| CACHED[Cached answer]
    CT -->|unsafe| RET

Diamonds are the five Decision calls this repo evaluates; the cache path is a separate, parallel short-circuit, not a step inside the main flow.

Quickstart

What you need:

• Python 3.11+
• HotpotQA data, only if you want to work with that dataset specifically — the frozen 1000-question val/test split and prebuilt BM25 index this project's sufficiency/chunk-boundary/context-selection results were measured on currently live in a private, unpublished research checkout, not (yet) independently downloadable. The risk-coverage/AURC math and EM/F1 scoring themselves are vendored directly into this package (jevrag/_vendor/, with attribution) — installing and running jevrag, including the calibration harness itself, needs no external checkout at all. Only code that loads the actual HotpotQA question set (jevrag/benchmarks/hotpotqa.py, and by extension scripts/produce_records.py) looks for that data, at D:\Research\RAG-Gate by default, overridable via RAG_GATE_PATH. Every other primitive (chunk-boundary, context-selection, answer-abstain, cache-trust) and the abstraction itself run and test fine without it.
• A Jev API key (TypeSafe AI) — needed to generate fresh records. Not needed if you're only evaluating records someone already gave you.
• An OpenRouter key, only if you want to generate your own fresh records rather than evaluate existing ones — it's the answer generator.

git clone https://github.com/ajanm007/jevrag.git
cd jevrag
pip install -e .

Create a .env file at the repo root with your keys:

JEV_API_KEY=sk-...
OPENROUTER_API_KEY=sk-or-...

Fastest path to real numbers — generate a small batch, then evaluate it:

python scripts/produce_records.py --split val --limit 20
jevrag eval sufficiency --dataset hotpotqa \
    --records records/sufficiency_hotpotqa_val.jsonl \
    --baseline-records records/fixed_iter3_hotpotqa_val.jsonl \
    --split val

The first command makes real Jev + generator calls for 20 HotpotQA questions (cheap — a few cents) and writes two JSONL files under records/. The second reads them back and prints the full report: accuracy, calibration (ECE/Brier), a risk-coverage table, and a cost breakdown — the same report shape the full n=700 results below came from, just on a smaller, faster sample. Drop --limit 20 (and switch --split val to --split test) to reproduce the real headline numbers, though that means the full 700-question run.

No Jev key yet? The jevrag eval commands never call a model themselves — they only read a records file and report on it — so if you already have a records file from someone else's run (records/ and outputs/ aren't committed to this repo, since they're regenerated output, but they're easy to share directly), you can run the eval step immediately without any key at all.

What's actually built and proven so far

All five originally-scoped decisions are built end-to-end and evaluated for real, on the identical Decision abstraction and the identical calibration harness, with zero modifications to either required by any of them — verified by cryptographic hash comparison (sha256 of every protected module, before and after each new primitive's first live run), not just by claim. All five now run through one unified CLI.

• evidence-sufficiency — given a question and the evidence retrieved so far, is that enough to answer, or should retrieval run another round? An iterative, multi-round stopping decision, asked after every retrieval round.
• chunk-boundary — given a candidate split point in a document, is this a coherent boundary or should the text stay merged? A one-shot, pre-retrieval decision, the structural opposite of sufficiency's loop.
• context-selection — given a retrieved candidate passage, does it earn a slot in the context sent to the generator? A one-shot, per-item filter/select decision — evaluated two ways (see below).
• answer-abstain — given the question, the retrieved evidence, and the answer the generator actually produced, does that answer stay grounded in the evidence, or should it be suppressed? A one-shot, post-generation grounding check — the only one that looks at generated output rather than input.
• cache-trust — given a semantic-cache hit, is the cached answer safe to serve, or should the system regenerate? A one-shot decision over purely structural state (similarity score, entry age, scoping — never the cached content itself, by design). Extracted from real, shipped prior art and independently corroborated by a second, blind design attempt that converged on the same architecture without ever seeing the first.

Concretely, what exists:

• A Decision protocol (jevrag/decision.py) — ask(state, questions) -> DecisionResult, with Jev as the live backend and a swappable interface verified by plugging in a second, trivial backend without touching anything else. Also verified against a non-text, purely structural state and a multi-question single call — neither assumption held only for text-shaped, single-question decisions.
• Five primitives (jevrag/primitives/) — sufficiency.py, chunk_boundary.py, context_selection.py, answer_abstain.py, cache_trust.py (plus cache_safety_check.py, an independently-designed second implementation of the cache-trust decision, built blind without reading the first — see Results below).
• A calibration-first evaluation harness (jevrag/eval/) — computes AURC and risk-coverage curves (a standard method from the selective-prediction literature for scoring how well a confidence signal separates likely-correct from likely-wrong answers), plus ECE, MCE, Brier score and Brier skill score (standard calibration metrics: is a stated confidence of 0.9 actually right about 90% of the time?). This harness is what turns "the model said it was confident" into a number you can actually trust or distrust — and it is the same code, unmodified, behind every one of the five primitives above.
• A fixed-iteration baseline (jevrag/baselines/fixed_iteration.py) — the same retrieval and generation stack with the gate disabled, always running a fixed number of rounds. This is what the gated sufficiency version has to beat, or at least match more cheaply.
• Two wrapped/reproduced external comparisons — a real, third-party package (rag-jev, PyPI) wrapped via adapter for context-selection, and a real, independently-run reproduction of its own published benchmark result (see Results below); and a free, logprob-based confidence signal (rag-gate) benchmarked directly against answer-abstain on identical ground truth.
• HotpotQA benchmark wiring — a public multi-hop question-answering dataset, with a frozen 300/700 validation/test split, scored with standard exact-match and F1 metrics.
• BEIR/SciFact benchmark wiring — a scientific claim-verification retrieval benchmark, used specifically to reproduce and compare against a real external package's own published result on its own home benchmark, not just on a dataset of convenience.
• DocBench wiring (jevrag/benchmarks/docbench.py) — a second, structurally different document benchmark (real PDFs, not short-span QA). Used to independently test all five primitives on two separate real documents, with two different generators — see Results below.
• A CLI — jevrag eval sufficiency|chunk-boundary|context-selection| answer-abstain|cache-trust, one command per primitive that reads pre-generated records and prints the full report: accuracy, calibration, risk-coverage, cost, and (where applicable) a baseline comparison. Every primitive is reachable this way, not just the first one built.
• A real, live, chained pipeline (jevrag/pipeline.py) — all five decisions actually wired end to end per the locked architecture (chunk-boundary → index → context-selection every round → sufficiency → answer-abstain, cache-trust as a parallel bypass), not just evaluated side by side. See The real pipeline.
• A per-decision CRC calibration layer (jevrag/eval/crc.py) — pick a threshold from a stated error budget instead of a guess, with a real, disclosed distinction between the risk quantity CRC actually bounds and the one people usually mean by "risk." See The CRC calibration layer.
• A second real Decision backend (jevrag/backends/ logprob_decision.py) — proves the abstraction's backend-swappability claim against something other than a trivial test double, and gives every primitive a zero-cost baseline. See A second Decision backend.
• A shadow/observe wrapper (jevrag/backends/shadow.py) — run a challenger policy alongside a real one without ever affecting the real result, logged to a JSONL ledger. See Shadow / observe mode.

The full CLI, all five primitives

The Quickstart above covers sufficiency end to end. Every other primitive follows the identical records-in/report-out shape — generate with that primitive's own scripts/eval_*.py script, then evaluate:

jevrag eval chunk-boundary --records outputs/chunk_boundary_wikipedia.jsonl
jevrag eval context-selection --records outputs/context_selection_hotpotqa_val.jsonl
jevrag eval answer-abstain --records outputs/answer_abstain_val.jsonl
jevrag eval cache-trust --records outputs/cache_trust_val.jsonl

None of the jevrag eval commands call a model or retriever themselves — generating records and evaluating them are kept deliberately separate, so the same eval command works identically on a run from five minutes ago or five days ago. Full flag reference for any primitive: jevrag eval <primitive> --help. See INTERFACE.md for the exact records shape sufficiency expects; each other primitive documents its own shape in its module docstring.

Results

Results at a glance

Every number below is traceable to a raw run output or a report in tempnotes/. Full per-primitive write-ups — caveats, failure modes, and what each result does not claim — are in docs/primitives.md. AURC is lower-is-better throughout.

What the v0.3 suite is, and is not. It measures how the evidence-sufficiency gate and the CRC calibration layer behave across three multi-hop datasets. It is deliberately narrow, and its limits are part of the result:

• Closed pools. Each dataset's retrieval pool is built from the paragraphs of its own sampled questions (a shared pool for HotpotQA, per-question pools for MuSiQue and 2Wiki). This is not open-domain retrieval; absolute accuracy is not comparable to open-domain systems or across datasets.
• No chunk-boundary ingestion. The suite does not exercise the chunk-boundary decision. DocBench accuracy elsewhere in this README is retrieval-limited.
• Sample. Each dataset uses the first 150 calibrate and the first 150 report questions of a frozen, seeded 500-question draw (down from the planned 500 to fit a $5 monthly credit). All three calibrate halves are complete at n=150. The HotpotQA and MuSiQue runs were halted by the generator account running out of credits (the runner fails closed, so every kept row is clean), so their report halves are n=98 and n=111; 2Wiki is complete at n=150. Report-half questions are the first N in file order, which is not a random subset. Intervals are wide, and the same 50 questions re-run gave results a question or two apart; differences of a few points are not findings. The first 50 questions of a split are unrepresentative (the 2Wiki gate answered 12 of the first 50 but 80 of the next 100), which is why early 50-question checkpoints are not reported.
• Shift test. A single-source shift (calibrate on HotpotQA) across closed-pool multi-hop datasets: a shift in difficulty more than in task type.
• Known limitations.
  • The CRC tie-break is keyed by question text at deploy time and by question id at calibration time, and recorded CRC-arm confidences already carry the deploy-time shift. The CRC rows above are therefore read as the arm's live decision (answered iff the recorded confidence is at or above the recorded threshold), not through the id-keyed readout used for the gated arm. At tied confidences (they are quantized to two decimals) the shift-test answered counts depend on the tie-break convention; no answered question was wrong either way. Unifying the keys is a later release.
  • The shared HotpotQA pool file carries a header row that the suite runner's loader rejects, and a relative pool_file in the records header resolves against the working directory; the Kaggle notebook strips the header and passes an explicit path. Fixed in the next release, not this one, so the code in this tree is exactly the code that produced the reported numbers.
  • Resuming a suite run treats a row that recorded a transient error as finished, so it is not retried. None of the three result files contains an error row, so no reported number is affected.
  • Closed pools and the truncated report halves above limit how far these numbers generalize; they are not open-domain results.

Not in v0.3, planned for v0.4: NQ and TriviaQA with a real open-domain index; the full 250 / 250 split on every dataset (the runner resumes, so only the missing rows need re-running); crc_conditional; LangChain and LlamaIndex adapters; RAGTruth and CRAG; chunk-boundary in the suite; DocBench past five documents; HotpotQA validation regeneration; selection-cache persistence; and a logprob-based answer-abstain backend from another model provider as a further comparison.

The answer-abstain row is the one that moved under more data, and it did not move in Jev's favour. At n=30 Jev led on AURC by 0.0080 (p=0.879). At n=210 the sign flipped — Jev 0.0199 behind (p=0.567) — and at n=300 it is 0.0189 behind, p=0.493, not significant either way. The honest reading is a tie on ranking quality between two signals, one of which costs nothing to produce. Jev's real advantages on this decision are calibration (ECE 0.258 vs 0.488) and usefulness at a fixed threshold (passed-subset EM 0.583 vs 0.456). A second label proxy — wrong and ungrounded — was tracked separately; on it both backends score worse than an uninformative signal (AURC 0.9053 and 0.8625 against a 0.8167 random line).

Security disclosure (2026-09-24): the pinned ragas==0.4.3 — every version in the ragas extra's supported range — has an unpatched SSRF in its multi-modal faithfulness collections module (attacker-controlled retrieved_contexts can force the library to fetch arbitrary URLs or local paths). No patched release exists; the vendor did not respond to disclosure. The ragas extra is never installed by default or in CI, and this project's own use of it only scores a fixed local benchmark dataset — but if you install jevrag[ragas] yourself, do not pass untrusted or externally-sourced text as retrieved_contexts until a fix ships. See the [ragas] extra's comment in pyproject.toml for the exact vulnerable functions.

The real pipeline

All five decisions above were evaluated independently at first. They're now actually chained: jevrag/pipeline.py implements the locked architecture exactly — chunk-boundary runs once at ingestion to build the index, context-selection filters candidates on every sufficiency round (not just once after retrieval), the sufficiency loop runs as before, and its output now really does flow into answer-abstain as the final gate (pass → answer, veto → no answer). Cache-trust stays a separate, parallel short-circuit, exactly as designed.

Two real, live runs on an actual document (7 questions, P19-1598.pdf, a genuinely hard academic-prose document — base rates here have always been low, this is a wiring proof, not a fresh benchmark claim):

The CRC run is the actual proof the opt-in threshold does something real inside a live chain, not just in isolation: a question that passed under the default threshold at 0.92 confidence — and was actually wrong — was correctly withheld once a real error budget was applied.

The flagged cost tradeoff, measured rather than estimated: filtering on every round instead of once is real — a large majority of the default run's Jev calls were context-selection calls, since the same candidate pool gets re-judged each retrieval round with no memoization. That's a known, named optimization opportunity for a future pass, not a hidden cost.

A real, unflattering finding from the very first live run — chased down rather than smoothed over. The first live run found chunk-boundary's chunking nondeterministic on the identical document (3 splits in one run, 4 in another). Diagnosed with real evidence before fixing anything: candidate construction is byte-deterministic (identical candidate hashes across 7 fresh runs); the actual cause is Jev's own confidence varying near the 0.5 split/merge threshold on 2 of 9 boundary candidates. Fixed with mean-of-3 voting at ingestion (chunk_repeats=3 in the pipeline, repeats=1 legacy default elsewhere) — chosen over majority voting specifically because a 3-vote average stays a continuous probability, where a vote fraction would quantize and corrupt the calibration signal downstream. Verified 5/5 identical boundary sets across repeated post-fix runs, at roughly 3x the ingestion call cost (9 → 27 calls on this document) — and disclosed honestly, not oversold: one post-fix run's mean landed just 0.007 over the threshold on the same straddling candidate, so the fix reduces the flip probability, it doesn't guarantee it away.

The re-run's 2/7 → 3/7 answered delta above is not claimed as an accuracy improvement — judge-correct stayed flat at 1/7 in both runs. The extra answered question's abstain confidence swung from 0.09 to 0.85 across the pre/post-fix runs, coinciding with both the changed chunking and this primitive's already-known abstain-side variation — inseparable at n=7, stated as such rather than credited to the fix.

Run it yourself: python scripts/run_pipeline_doc.py (see the script's own --help for CRC-alpha flags, --repeats, and document overrides).

The CRC calibration layer

Every result above uses a single threshold per decision, picked by coverage target or by hand. jevrag/eval/crc.py adds a real alternative: conformal risk control (Angelopoulos et al. 2022) — state an acceptable error rate (alpha), get back the threshold whose expected risk is bounded by it, instead of guessing a cutoff and hoping.

Vendored, then a real bug found on real data, then fixed — and fixing it surfaced something more important than the bug itself. The pure CRC math ported cleanly from prior art (same pattern as selective.py). Run on real sufficiency records, the guarantee held at alpha=0.30 but failed at alpha=0.10/0.20 — root-caused to Jev's confidence being 2-decimal quantized, which breaks the naive threshold-comparison read-out under ties (the rank-selected cutoff was right; the score >= threshold rule over-answered whenever many rows tied at that value). Fixed with a deterministic per-row tie-break.

Re-verifying the fix surfaced a real methodological correction, not just a passing test: CRC's actual guarantee bounds marginal risk (P(answered AND wrong)), not the conditional risk (P(wrong | answered)) every report — including the original prior art's own — had been comparing against alpha. They're different numbers. On real held-out data at alpha=0.10: conditional risk read 0.14 (over budget), marginal risk read 0.017 (comfortably under). Both numbers are correct; conflating them makes a working guarantee look broken. Across 191 independent tie-break realizations, the marginal guarantee held 100% of the time at every alpha tested; the conditional number (the one people actually want — "if I answer, how often am I wrong") did not, and getting that bounded reliably needs a different method. crc_calibrate/crc_readout now return both numbers explicitly, never blended into one "risk."

A note on this was also left in the upstream research repo this method was vendored from, since the same conflation applies to its own published results, not just JevRAG's reuse of it.

A second Decision backend

The abstraction's whole claim is that decision backends are swappable without touching the harness, the controller, or any primitive. Until now that had only been tested against StubDecision, a trivial, zero-cost test double built for unit tests. jevrag/backends/ logprob_decision.py is the first real second implementation: confidence is the geometric-mean per-token probability of a generation (exp(mean_logprob)) — free, no network call, no API key, and a genuine probability with zero tunable parameters (rejected a sigmoid squash specifically because its parameters would be arbitrary, or worse, fitted to whatever it's being evaluated against).

It plugged in at the strongest acceptance-test level yet: not one pre-existing file changed, not even decision.py. Run on answer-abstain's real 30-question 4o-mini records: AURC 0.1624, reproducing the earlier throwaway-script comparison to four decimals (expected — the mapping is strictly monotonic). The honest finding: ECE 0.31, Brier skill −0.20 — the same "ranks well, calibrates poorly" shape sufficiency showed, but from a backend with zero vendor model in it, which is real evidence the pattern isn't a Jev quirk. The mapping was frozen before running the eval and was not retuned to make that number look better. (Those figures are the original n=30 run; the n=300 re-run in Results at a glance puts this backend at AURC 0.3310, ECE 0.4877, Brier skill −0.8764 — worse calibration, but nominally ahead of Jev on ranking (AURC 0.3310 vs 0.3498; not significant, p=0.493).)

Every primitive now has a real, zero-cost baseline available to compare Jev against.

Shadow / observe mode

jevrag/backends/shadow.py wraps any Decision backend to log what it would decide alongside a real decision, without ever changing what gets returned. Useful two ways: wrap the real governing backend for a production audit trail, or wrap a challenger backend for a live what-if comparison — the latter is what's actually demonstrated, since every JevRAG decision already evaluates for real (unlike some prior-art observe modes built for systems that otherwise skip evaluation entirely).

Real demo: the free logprob backend run as a challenger against recorded Jev production decisions (same 30-question set). They agreed on only 20/30 decisions at a normal 0.5 threshold; where they disagreed, the challenger passed 10 answers Jev's real policy didn't. Production's passed-subset accuracy was 85%; the challenger's own passed subset was only 63% — no better than the overall base rate. Consistent with the overconfidence already found above: at an ordinary threshold, the free signal would have let more wrong answers through.

Real documents, two of them, all five primitives, two different generators

Beyond the paired-benchmark comparisons above, all five primitives were run independently (not chained) against two real, complete documents — an academic paper and a RAG research paper — using two different generators (a low-volume OpenAI-compatible model, and Gemini, integrated for the first time). Each run ended in an explicit, pre-agreed pass/fail verdict per primitive against a stated numeric bar, not just raw numbers left for interpretation. Context-selection passed cleanly both times with the largest margins measured anywhere in the project; sufficiency and answer-abstain passed but remain data-thin at this scale; chunk-boundary passed on one document and failed the strict recall bar on the second, for the reason described above. Cache-trust was correctly skipped both times rather than forced through a scenario that would have added nothing new.

The standing calibration finding, across every real measurement

Across every decision measured — different domains (multi-hop QA, document structure, scientific claim verification, post-generation grounding, cache safety), different implementations, the same Jev backend throughout — calibration quality varies by more than a full Brier-skill point, from −0.45 (worst: sufficiency, an iterative stopping decision with a skewed outcome distribution) to +0.96 (best: cache-trust, on an admittedly easy first scenario):

(Higher is better; 0 = no better than the base rate, negative = worse than guessing. answer-abstain's row is the n=300 measurement from outputs/v03_b1_summary_n300.json; at n=30 it read +0.07 and the sign has since flipped, which is why the label carries both. Its Brier score is 0.2612 — Jev is still far better calibrated than the logprob backend it is compared against, whose skill is −0.88.)

The pattern that holds across every measurement: Jev's confidence reliably ranks correct above incorrect — every single AURC measured has sat meaningfully between the oracle and random baselines — but its absolute calibration is a property of the specific decision and its base-rate structure, not a fixed trait of the backend. A sparse or skewed positive rate (sufficiency's stopping decision, SciFact's ~5% relevance rate) consistently produces worse absolute calibration than a more balanced one, regardless of which decision or implementation is being scored.

Reproducing this

The retrieval logic, the prompt shaping, and the sufficiency/baseline loops all live in this repository and are portable — scripts/produce_records.py runs against any OpenAI-compatible endpoint. The actual records behind the sufficiency results above were generated using a locally-run open model on a cloud GPU notebook; that notebook isn't included in this repository since it's throwaway infrastructure specific to one hosting setup, not part of the reusable library. The other primitives' results were generated with a mix of hosted API calls (OpenRouter, Gemini) — small, cheap runs, reproducible against any equivalent account.

Project layout

jevrag/
  decision.py              # the Decision interface + the Jev backend
  pipeline.py               # the real end-to-end controller, all five decisions chained
  _rag_gate.py              # resolves the (only remaining) HotpotQA-data checkout path
  _vendor/                  # vendored risk-coverage/AURC + EM/F1 + CRC math (no external checkout needed)
  primitives/
    sufficiency.py          # iterative, multi-round stopping decision
    chunk_boundary.py       # one-shot, pre-retrieval split/merge decision
    context_selection.py    # one-shot, per-passage filter/select decision
    answer_abstain.py       # one-shot, post-generation grounding decision
    cache_trust.py          # one-shot, structural-state cache-serve decision (extraction)
    cache_safety_check.py   # independent second design of the cache-trust decision
  backends/
    logprob_decision.py     # a second real Decision backend, zero-cost, no API calls
    shadow.py                # observe-only wrapper: log a challenger decision, never change the real one
  adapters/
    rag_jev_selector.py     # wraps the real rag-jev PyPI package for comparison
  eval/
    calibration.py          # AURC/risk-coverage + ECE/MCE/Brier — shared, unmodified across all five primitives
    crc.py                   # per-decision CRC threshold selection (conformal risk control)
    cost.py                  # token/latency accounting
  benchmarks/
    hotpotqa.py              # dataset loading + scoring
    scifact.py                # BEIR/SciFact loading + NDCG@10 scoring
    docbench.py                # real-document benchmark (two documents tested)
    llm_judge.py                # LLM-judge scoring for benchmarks without exact-match ground truth
  baselines/
    fixed_iteration.py      # the baseline evidence-sufficiency has to match or beat
  __main__.py                # the `jevrag eval <primitive>` command, all five wired in, CRC-alpha flag
scripts/
  produce_records.py                 # generates sufficiency's records
  eval_chunk_boundary.py             # chunk-boundary's Wikipedia eval script
  eval_chunk_boundary_doc.py         # chunk-boundary's real-document eval script
  eval_context_selection.py          # context-selection's own eval script (from-scratch primitive)
  eval_rag_jev.py                    # context-selection's wrapped-adapter eval script
  eval_scifact_ragjev.py             # rag-jev's SciFact reproduction
  eval_scifact_context_selection.py  # SciFact head-to-head, from-scratch primitive
  eval_answer_abstain.py             # answer-abstain's own eval script
  eval_ragas_faithfulness.py         # RAGAS faithfulness as a third abstention-gate signal
  eval_cache_trust.py                # cache-trust's own eval script
  eval_cache_trust_hard.py           # cache-trust's near-threshold hardening eval
  eval_cache_safety_check.py         # the independent second cache-decision design's eval
  eval_docbench_primitives.py        # all five primitives, one real document, independent
  demo_docbench.py                   # DocBench exploratory demo (unscored)
  demo_shadow_mode.py                # shadow-mode challenger demo against real records
  report_shadow_ledger.py            # reads a shadow-mode ledger back into a report
  run_pipeline_doc.py                # runs the real end-to-end pipeline on a document
  snapshot_protected_hashes.py       # sha256 manifest proving a new primitive touched nothing protected
tests/                               # 282 tests across all five primitives, the pipeline, and both new backends

records/, reports/, and outputs/ are regenerated by running the pipeline, not committed to the repository.

Status

V1 (evidence-sufficiency) and V1.1 (chunk-boundary) were the original frozen-scope deliverables and are complete. Everything past that — context-selection, answer-abstain, cache-trust, and now the whole v0.2.0 pass (the real pipeline, the CRC calibration layer, a second Decision backend, and shadow/observe mode) — is explicit, approved scope expansion, done because it held up under the same rigor as V1, not because scope crept. See What's actually built for the full picture and the Results section — with per-primitive detail in docs/primitives.md — for what's still open (e.g. calibration is real but uneven, and CRC's own guarantee needed a real correction mid-build — both are findings, not defects).

Still genuinely open: a memoization pass on context-selection's per-round re-filtering (a measured cost, not yet optimized); HotpotQA val regeneration to remove a two-provenance disclosure; a LangChain/LlamaIndex adapter. (Chunk-boundary's ingestion nondeterminism, found by the real pipeline's first run, is now diagnosed and mitigated — see The real pipeline — with a residual, disclosed flip probability rather than a guarantee.)

License: MIT.

_{Built as a research/eval project; Jev is TypeSafe AI's decision model and is not affiliated with this repository.}