Skip to main content

agentforge-eval-geval

LLM-judge evaluators for AgentForge (feat-006).

This package ships the G-Eval engine — an LLM-as-judge grader that scores agent outputs against a rubric YAML — plus six named graders that wrap G-Eval with reference rubrics:

Grader Scores
Correctness Output matches the ground-truth answer (rubric-tunable for binary / Likert / ordinal)
Faithfulness Output is supported by the retrieved evidence (no claims beyond what was retrieved)
Groundedness Output stays inside the provided sources (no off-source content)
Hallucination Output contains content not derivable from inputs (faithfulness + groundedness combined into a single risk score)
Relevance Output addresses the user's question vs going off-topic
Helpfulness Output is useful — actionable, complete, well-structured

Each grader takes a judge LLMClient at construction. The judge is typically a cheaper model than the agent's primary model (e.g. Haiku judging Sonnet output) — cuts cost and reduces "judge agrees with itself" bias.

Quick start

from agentforge import Agent
from agentforge_bedrock import BedrockClient
from agentforge_eval_geval import Correctness, Faithfulness

judge = BedrockClient(model_id="us.anthropic.claude-haiku-4-5")

agent = Agent(
    model="bedrock:us.anthropic.claude-sonnet-4-5-20250929",
    evaluators=[
        Correctness(judge=judge, ground_truth_field="expected"),
        Faithfulness(judge=judge, sources_field="retrieved_docs"),
    ],
)
result = await agent.run("Summarise PR #42")
print(result.eval_scores)

Cost-bounded: each judge call bills against the run's BudgetPolicy (feat-007). When the remaining budget falls below the grader's cost_estimate_usd, the agent skips the grader and logs a WARN.

Custom rubrics with G-Eval

from agentforge_eval_geval import GEval

grader = GEval(
    judge=judge,
    name="code-review-quality",
    rubric={
        "criteria": "Score the PR description's accuracy and completeness.",
        "scoring": "0.0 = incorrect/incomplete; 1.0 = accurate and complete",
        "examples": [
            {"output": "...", "score": 0.9, "reasoning": "..."},
        ],
    },
)

Or load from a YAML file:

grader = GEval.from_rubric_file("./rubrics/code-review.yaml", judge=judge)

Metadata

Release files for agentforge-eval-geval 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentforge-eval-geval 0.4.0
File Size Uploaded Provenance Actions
agentforge_eval_geval-0.4.0.tar.gz 14.8 kB With provenance Details for agentforge_eval_geval-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentforge-eval-geval 0.4.0
File Interpreter ABI Platform Actions
agentforge_eval_geval-0.4.0-py3-none-any.whl Python 3 none any Details for agentforge_eval_geval-0.4.0-py3-none-any.whl

Total release size: 30.6 kB

Release files / agentforge_eval_geval-0.4.0.tar.gz

Download URL agentforge_eval_geval-0.4.0.tar.gz
Size 14.8 kB
Tags Source
SHA-256 checksum
How to use checksums
633a89f732c226504f0563088ba24b50d72d8a94a8dc8c35378dac17477c1477
BLAKE2b-256 checksum
How to use checksums
16309111e728d73f4b129295bc0e919c40333e01168df19af05ca0658c8b88ab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 24, 2026.

Transparency log

Release files / agentforge_eval_geval-0.4.0-py3-none-any.whl

Download URL agentforge_eval_geval-0.4.0-py3-none-any.whl
Size 15.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8595cda41998b6184808f46361077a3ce7b29e8d17458358594061d419c74c92
BLAKE2b-256 checksum
How to use checksums
918f9b5ed7b84fcaaf1c9e0a7aa4a8a6edbbe523cf04ee1ca25eaa25c0ac01f7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 24, 2026.

Transparency log

Security for agentforge-eval-geval 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for agentforge_eval_geval-0.4.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jun 24, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page