larpie 0.2.0

Attribution tooling for LLM responses.

larpie

The problem

LLM traces show what happened: the prompt that went in, the response that came out. They don't show which specific context segments, retrieved chunks, tool outputs, memory, earlier turns in the conversation, actually caused that response. When a RAG or agent pipeline gives a wrong or hallucinated answer, debugging it today usually means reading the whole trace by hand and guessing.

What larpie is

larpie is a developer tooling layer built around existing context-attribution research. It implements two attribution algorithms, at different points on that spectrum:

• SimilarityAlgorithm is a cheap proxy: it regenerates a response for each ablated context and compares embeddings, so it works against any hosted chat API (OpenAI, Anthropic, Ollama, via litellm) with no logprob access needed.
• LikelihoodAlgorithm implements the actual mechanism ContextCite: Attributing Model Generation to Context (Cohen-Wang, Shah, Georgiev, Madry; NeurIPS 2024) describes: teacher-forced scoring of the original response's log-likelihood under each ablated context, fit with the same LASSO-plus-bootstrap approach. This needs an engine with direct model access - currently a local Hugging Face model via HFEngine - since hosted chat APIs don't expose that.

Both use the same ablate-and-regress idea as AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution (Liu, Kandpal, Raffel; ICLR 2025) too. larpie's own, simpler take on AttriBoT's hierarchical idea is available as an opt-in mode (see Hierarchical attribution, below): group segments explicitly, ablate whole groups first, drill into only the ones that turn out to matter. AttriBoT's other efficiency tricks (proxy models, cached-activation reuse) aren't implemented - ablation here is otherwise the straightforward exhaustive-or-sampled approach, not the faster approximation.

What larpie adds beyond either paper is the developer-tool layer around them: real integration with LLM providers (via litellm, across OpenAI, Anthropic, and Ollama, plus a local Hugging Face model for likelihood attribution), segmentation that understands RAG metadata (which parts of a conversation are retrieved chunks, tool outputs, or plain turns), a capability-based engine architecture that fails loudly instead of silently producing a wrong result when an engine can't support an algorithm, a CLI workflow for inspecting logged calls, and SQLite-backed caching so re-inspecting the same call is free.

Quickstart

pip install larpie

This requires a local Ollama server with a model pulled (ollama pull llama3), or any OpenAI/Anthropic model string if you'd rather pay for API calls.

Wrap a real litellm.completion call so it gets logged, then inspect it:

from litellm import completion
from larpie.middleware import with_logging

logged_completion = with_logging(completion)

response = logged_completion(
    model="ollama/llama3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "The Eiffel Tower was completed in 1889. When was it finished?"},
    ],
)

# with_logging logs the call and stamps the id of the row it just wrote
# onto the response:
call_id = response.larpie_call_id
print(call_id)

larpie inspect <call_id>

inspect looks up the logged call, segments its messages, re-runs the response with pieces of the context ablated, and prints a ranked table of which part of the context actually drove the answer. --model defaults to the model the call was logged with, so it isn't needed here; pass --yes/-y to skip the paid-call confirmation prompt for non-Ollama models (Ollama runs locally and never asks).

See it in action

examples/rag_attribution_demo.py runs a small RAG conversation (one supporting chunk, two distractors, and a fabricated fact the model can't already know from pretraining) against a real local Ollama model and prints the ranked attribution table. Run it yourself:

python examples/rag_attribution_demo.py

This is real output from an actual run (ollama/qwen2.5:3b-instruct-q4_K_M, temperature 0), not illustrative numbers:

Closed-book response (no context, sanity check):
  'I need more specific context to answer this question accurately. The provided context does not contain information about the Aldemoor Lantern Festival or its founding year.'
  Good: no '1997' here - the model can't answer this without the supporting chunk.

Full-context response:
  'The Aldemoor Lantern Festival began in 1997.'

Ranked attribution results (query segment excluded):
  1. [seg_1] (rag_chunk) score=0.1789 ci=(0.1277834502965072, 0.21895693424139875) text="The Aldemoor Lantern Festival began in 1997, founded by local artist Mira Thorne to celebrate the town's bicentennial."
  2. [seg_2] (rag_chunk) score=0.0014 ci=(0.0, 0.025790114892243907) text='Mira Thorne is a sculptor known for her large-scale kinetic light installations across Aldemoor.'
  3. [seg_0] (system) score=-0.0025 ci=(-0.027141633458489772, 0.0) text="You are a helpful assistant. Answer the user's question using only the provided context snippets, as briefly as possible."
  4. [seg_3] (rag_chunk) score=-0.0059 ci=(-0.042601720261782446, -0.0) text='The neighboring Bellhaven Torch Parade started in 1985 and features similar lantern displays.'

The closed-book check confirms the model can't answer from pretraining alone. Once the supporting chunk is added back, the model answers correctly, and attribution correctly ranks that exact chunk first, with a confidence interval sitting entirely above zero. Every other segment, including a distractor that names a different year for a different event, scores near zero with a confidence interval that includes zero.

Architecture

larpie sits between your LLM application and the model provider as a thin logging and analysis layer:

User's LLM application
  -> larpie middleware (logs the call to SQLite)
  -> Segmenter (splits context into turns, RAG chunks, tool outputs; dedups near-identical chunks)
  -> Attribution algorithm (declares which engine capability it needs)
  -> Engine capability layer (checks what the model/runtime can actually do)
  -> LLM provider (via litellm: OpenAI, Anthropic, Ollama)

Engine is how larpie talks to a specific model or runtime. An engine can implement generate (produce text), embed (produce a vector), and score_continuation (return the log-likelihood of a fixed continuation given a context), but not every engine implements all three, and larpie checks for the ones it actually needs at runtime instead of assuming.

Algorithm is how attribution gets computed from whatever capabilities an engine offers. SimilarityAlgorithm needs generate and embed. LikelihoodAlgorithm, implementing the ContextCite mechanism, needs score_continuation instead: it compares the model's actual log-likelihood of the original response with and without each piece of context, rather than regenerating a response and comparing embeddings.

Capability matrix

Not every engine can run every attribution algorithm. algorithms.base.check_capability enforces this at runtime: a missing capability raises a clear error instead of silently falling back to something else.

Engine	generate	score_continuation	embed	Compatible algorithm(s)
LiteLLMEngine (OpenAI / Anthropic / Ollama, via litellm)	yes	no	yes (local sentence-transformers)	SimilarityAlgorithm
HFEngine (local Hugging Face transformers model)	yes	yes	no	LikelihoodAlgorithm
vLLM / SGLang (planned)	yes	yes	n/a	LikelihoodAlgorithm (planned support for larger models than a local transformers instance can comfortably run)

Likelihood attribution

LikelihoodAlgorithm measures the actual log-likelihood of the original response under each ablated context via teacher-forced scoring, not a regenerate-and-compare-embeddings proxy like SimilarityAlgorithm. Run it with:

larpie inspect <call_id> --algorithm likelihood --hf-model Qwen/Qwen2.5-0.5B-Instruct

--hf-model defaults to Qwen/Qwen2.5-0.5B-Instruct if omitted. It runs fully local: no Ollama server, no API key, no network access beyond the one-time model download from Hugging Face. A small model like that default runs fine on CPU; anything meaningfully larger needs real GPU memory to be usable in practice.

Real output from an actual run, on a small RAG conversation (a system prompt, one supporting chunk, one distractor) using the same fabricated "Aldemoor Lantern Festival" fact as the similarity example above:

This will make 8 engine.score_continuation() call(s): 7 ablation(s) + 1 baseline call.
[QUERY - not ranked] [seg_3] (user_turn) 'In what year did the Aldemoor Lantern Festival begin?'
segment    type               score  ci                         text
seg_1      rag_chunk         5.6799  (3.155, 7.061)             'The Aldemoor Lantern Festival began in 1997, founded by Mira Thorne.'
seg_2      rag_chunk         1.8188  (-0.016, 3.557)            'The neighboring Bellhaven Torch Parade started in 1985.'
seg_0      system           -2.2877  (-3.777, -0.134)           'Answer using only the provided context.'

Note: no noise-floor check for likelihood attribution; see README.

The supporting chunk (seg_1) scores clearly highest, with a confidence interval sitting entirely above zero - the same "CI excludes zero means a reliably detected effect" reading used everywhere else in larpie. Unlike the similarity table, there's no NOISE column here; see Limitations for why.

Hierarchical attribution

Both algorithms accept hierarchical=True (default off, fully opt-in) for contexts where several segments are known to come from the same underlying source - multiple chunks retrieved from one document, for example. Enable it from the CLI with:

larpie inspect <call_id> --hierarchical

or directly: SimilarityAlgorithm(hierarchical=True) / LikelihoodAlgorithm(hierarchical=True).

Instead of one ablation column per segment, hierarchical mode ablates whole groups together in a first pass. Only a group whose group-level confidence interval excludes zero - a reliably detected effect - gets a second, smaller pass drilling into which of its own members actually drives that. A group whose CI includes zero is left alone: every member of that group inherits the group's own score and CI instead of getting one computed individually, marked with is_group_estimate=True on the result ([group est.] in the CLI table).

The tradeoff is explicit: fewer engine calls overall - a group that doesn't matter costs one ablation instead of one per member - at the cost of coarser resolution for exactly those groups. You get "this whole group doesn't matter" rather than "which member of it doesn't matter, specifically". A group that does matter isn't shortchanged: it still gets drilled into individually, same resolution as the flat path.

This needs a real grouping signal to do anything: set a shared metadata["group_id"] on messages that come from the same source (see larpie.segmenter.Segment.group_id). Segments without a group_id are each their own singleton group, so with none set anywhere, hierarchical=True is a no-op - it falls back to exactly the flat, per-segment behavior described above.

Limitations

• Similarity attribution is a proxy. It compares the embedding of a regenerated response to the embedding of the original response; it does not measure the model's actual probability of the response (true likelihood attribution). Two different pieces of text can embed similarly even when the model's confidence in them differs a lot - a negated claim or a changed number often reads as barely different to an embedding model, even though it completely changes the answer.

  SimilarityAlgorithm accepts an optional metric callable (Callable[[str, str], float], taking (original_response, regenerated_response) and returning a dissimilarity score in [0, 1]) so you can swap in a sharper comparison for your case instead of being stuck with embedding similarity's specific blind spots. larpie ships one concrete alternative, numeric_aware_dissimilarity, which catches changed numbers that embedding similarity misses:

  from larpie.algorithms.similarity import SimilarityAlgorithm, numeric_aware_dissimilarity
  
  algo = SimilarityAlgorithm(metric=numeric_aware_dissimilarity)
  

  This doesn't solve the general problem - no single metric catches every kind of meaning change, and a swapped-in metric has its own blind spots in exchange for closing this one. It just makes the comparison step swappable, rather than fixed to the default's specific weaknesses.

• Likelihood attribution needs an engine that can compute score_continuation directly. That exists today for a local Hugging Face transformers model (HFEngine + LikelihoodAlgorithm, see above); vLLM and SGLang support doesn't exist yet and is still planned, mainly for models too large to run comfortably as a local transformers instance.

• There is no noise-floor check for likelihood attribution, unlike similarity attribution. That check works by calling generate() against the same unablated context several times and measuring how much the sampled output varies; score_continuation is one deterministic forward pass over fixed input with no sampling step, so repeating it would just return the same number every time - there is nothing to measure. The residual risk: on a backend whose forward pass isn't actually bit-reproducible (see the point below about local quantized models), that non-determinism could still produce spurious variation in scores that nothing here catches.

• If the model already knows a fact from its own pretraining, a retrieved chunk supporting that fact will correctly score near zero: removing it doesn't change the answer, because the model didn't need it to answer correctly. This is expected behavior, not a bug, but it means a low or zero score for a chunk doesn't mean the chunk was irrelevant to the topic, only that it wasn't necessary for this particular answer.

• larpie is built for complex contexts: 10+ segments, RAG chunks mixed with agent tool call histories and long-running memory, where reading the raw trace by hand doesn't scale. It is not meant as a general debugger for small or simple prompts; if your context is two or three chunks, just read them.

• Local quantized models are not guaranteed to be bit-reproducible even at temperature 0, since floating-point reduction order in batched or threaded inference isn't always deterministic on those backends. Caching gives you exact reproducibility for repeated inspection of the same logged call, since cached ablation results get reused instead of regenerated, but it does not make a fresh run against a live model deterministic on its own.

• The dedup threshold (0.9 cosine similarity) can merge near-duplicate templated text, e.g. numbered filler turns differing only by a label ("follow-up question number 1" vs. "number 2"). This isn't just theoretical: it happened during scale testing and had to be fixed by making the filler text genuinely distinct, not just relabeled.

• Likelihood attribution's real forward passes (HFEngine.score_continuation) must fit the full ablated context plus the original response within the local model's context window. This is handled per ablation: a vector that would exceed the limit raises a clear ContextTooLongError (checked against the model's actual config.max_position_embeddings - confirmed more reliable than tokenizer.model_max_length, which reports an unrelated, much larger number for the default Qwen model) instead of truncating silently or crashing inside the forward pass, and LikelihoodAlgorithm skips just that one ablation vector, printing how many were skipped and why. One case is still unhandled: since ablation only ever removes segments, the full, unablated baseline call is always the largest context in a run - if that alone already exceeds the limit, the whole run still aborts, since there's no smaller context left to fall back to.

License

MIT. See LICENSE.

Development

pip install -e ".[dev]"
pytest