Drex for LlamaIndex

llama-index-evaluation-nace

Replace your LLM judge. LlamaIndex evaluators backed by Drex decision models from Nace: Drex returns probabilities, not text that has to be parsed.

Install

pip install llama-index-evaluation-nace

Set NACE_API_KEY. Optional: NACE_BASE_URL, NACE_DEFAULT_DECISION_MODEL. For OpenRouter, pass base_url="https://openrouter.ai/api/v1" and set OPENROUTER_API_KEY. See Backends and credentials in the repository README.

The default model is drex-max-v1.1. An explicit model= is sent exactly as given; without one, NACE_DEFAULT_DECISION_MODEL is used when set, else drex-max-v1.1. If that default is a text-only model (drex-v1.5, drex-v2.0) and the request has images, drex-max-v1.1 is used for that request. OpenRouter doesn't list v1.1 yet, so there the default is the original Drex Max, nace-ai/drex-max.

Usage

from llama_index.evaluation.nace import NaceFaithfulnessEvaluator

result = NaceFaithfulnessEvaluator().evaluate(
    query="When was the Eiffel Tower completed?",
    response="It was completed in 1889.",
    contexts=["The Eiffel Tower was completed in March 1889."],
)
print(result.passing, result.score, result.feedback)  # True 0.95 P(faithful)=0.95

evaluate() is sync and works in Jupyter; aevaluate() is async. evaluate_response() reads source nodes, so the evaluators work with BatchEvalRunner.

raise_error=True (faithfulness, relevancy) raises ValueError on a failing result, as LlamaIndex's evaluators do. The question text is editable through update_prompts({"instructions": ...}).

Results are NaceEvaluationResult: probability, probabilities (score levels), model, request_id, usage.

Closing

Call close() (or await aclose()) to release the HTTP connections an evaluator opened; a client= you passed in stays open.

Images

Evaluators read ImageNode sources (image_path, base64 image, or a data: URL) and send them to Drex. The default model, drex-max-v1.1, reads images; when NACE_DEFAULT_DECISION_MODEL names a text-only model, requests with images still go to drex-max-v1.1, and an explicit text-only model= with images raises NaceImageModelError. image_urls= and ImageNode.image_url take data: URLs only; pass files as image_paths (a file path in a URL field is refused, never read). http(s) image URLs are refused, because Drex never fetches URLs. max_images=N keeps the N highest-scored image nodes, whatever their count; without it, more than 8 images (Drex's per-request limit) raise NaceImageLimitError rather than being trimmed silently. Strict faithfulness on image-only sources tends to score lower than on text; compare probabilities rather than trusting the default cut-off.

See examples/ for runnable scripts and notebooks.

Security

Versions 0.1.0 and 0.1.1 could send NACE_API_KEY (or DREX_API_KEY) to a custom base_url or to OpenRouter, and could forward internal deployment headers to other hosts, including through an HTTP client that follows redirects. Upgrade to 0.1.2. See the security note and changelog for details.