gooddata-eval

CLI to evaluate the GoodData AI agent against a dataset of natural-language questions on a chosen workspace and LLM model — including multi-model comparison.

Install

uv add gooddata-eval

Or install gd-eval as a standalone tool:

uv tool install gooddata-eval

Commands

gd-eval run

Quick start — single model

export GOODDATA_TOKEN='your-api-token'

gd-eval run \
  --host  https://your.gooddata.cloud \
  --workspace  ecommerce_demo \
  --dataset  ./my-dataset \
  --model  gpt-5.2 \
  --runs  1 \
  --json  results.json

Multi-model comparison

Pass --model multiple times to evaluate the same dataset against several models and get a side-by-side comparison:

gd-eval run \
  --host  https://your.gooddata.cloud \
  --workspace  ecommerce_demo \
  --dataset  ./my-dataset \
  --model  gpt-5.2 \
  --model  claude-opus-4-7 \
  --runs  1 \
  --json  comparison.json

When the same model id is offered by multiple providers, use the provider/model syntax to disambiguate:

  --model  "Foundry4o_4.1_5.2/gpt-5.2" \
  --model  "HN_Anthropic/claude-opus-4-7"

Both provider name and provider id are accepted as the prefix.

Targeting a specific AI Hub agent

GoodData has no admin-settable "default agent": when a conversation doesn't name one, the platform picks whichever agent was last used or last edited in that workspace. If your org has several AI Hub agents configured (e.g. one scoped to visualization only, another with every skill enabled), evaluating without --agent-id can silently exercise the wrong one — a metric_skill/alert_skill item run against a visualization-only agent will never pass, no matter how well-formed the question is.

export GD_EVAL_AGENT_ID='eval-all-skills'

gd-eval run \
  --host  https://your.gooddata.cloud \
  --workspace  ecommerce_demo \
  --dataset  ./my-dataset \
  --model  gpt-5.2 \
  --runs  1 \
  --json  results.json

Or pass it explicitly instead of via the env var:

gd-eval run \
  --host  https://your.gooddata.cloud \
  --workspace  ecommerce_demo \
  --dataset  ./my-dataset \
  --agent-id  eval-all-skills \
  --model  gpt-5.2 \
  --runs  1

All flags

Connection

Dataset source (pick one)

Model selection

Evaluation

Concurrency and workspace safety. Agentic kinds that create workspace objects (agentic_metric_skill, agentic_alert_skill, agentic_conversation, agentic_kda_skill) always run one at a time whatever --concurrency says — a metric or alert created and dropped mid-run would otherwise be visible to another item reading the same catalog. That protection is for the agentic kinds only: the single-turn metric_skill and alert_skill kinds are still fanned out and the agent performs the same server-side writes on that path, so avoid raising --concurrency on a dataset of those against a shared workspace. Progress output interleaves when K > 1, and per-item latencies rise, so they stop being clean single-request measurements.

Output

Langfuse sink

Set TAVERN_E2E_SKIP_TRACE_LINK=1 to skip trace lookup entirely (scores are then orphaned; the run says so).

A local --dataset cannot be attached to a Langfuse run. --langfuse is refused alongside --dataset because a local folder's item ids are not Langfuse dataset item ids. But trace linking does not depend on that flag — each evaluate_agentic_* builds its own client whenever LANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY are exported — so a local run still finds its traces and writes its scores onto them, and only the per-run grouping fails, with one 404 from dataset-run-items reported per run. The run warns about this before it starts. Use --langfuse-dataset when you want runs that are comparable across models, or TAVERN_E2E_SKIP_TRACE_LINK=1 to skip linking altogether.

When trace linking happens. Finding a gen-ai trace means polling until Langfuse has ingested it, which is lag measured in seconds to minutes. That work produces no verdict — the pass/fail is already decided — so it does not run inline per item. Every item's Langfuse block is queued and the whole batch runs after the agent phase, draining before any report is written. Two consequences worth knowing:

• No item's latency_s includes trace linking. Its cost is reported separately as latency_breakdown_s.langfuse_s, and the run prints [langfuse] trace linking finished in Xs for N item(s); slowest Ys. If slowest approaches the 120s batched retry budget, links are timing out and scores are being orphaned — look for [langfuse] WARNING: no trace found for conversation ....
• The budget depends on who is waiting. 120s is affordable only because the batch blocks nobody. A direct library caller (evaluate_agentic_* without a submit_trace_link) polls inline, on its own critical path, and gets 35s instead — the same cost as before batching existed, so no inline caller pays for a budget raised on the CLI's behalf. Either way a trace that is already ingested costs nothing: the loop looks before it sleeps. Scores are always final before the command exits — the run blocks on the batch. Interrupting with Ctrl-C drops whatever is still queued rather than making you wait it out: both the queued trace links and, under --concurrency, the items that have not started. The handful of items already in flight still have to finish — worker threads are joined at exit and an in-progress agent call cannot be cancelled — so expect to wait up to one --concurrency-wide wave, not the rest of the dataset.

JSON report shape

The JSON report always uses the nested multi-model shape:

{
  "models": ["gpt-5.2", "claude-opus-4-7"],
  "runs": {
    "gpt-5.2":        { "summary": { "passed": 22, ... }, "items": { ... } },
    "claude-opus-4-7": { "summary": { "passed": 18, ... }, "items": { ... } }
  },
  "comparison": {
    "gpt-5.2":        { "passed": 22, "total": 31, "pass_rate": 0.71, "avg_quality_score": 0.81, ... },
    "claude-opus-4-7": { "passed": 18, "total": 31, "pass_rate": 0.58, "avg_quality_score": 0.72, ... }
  }
}

Winner is selected by pass rate → quality score → latency (lower latency wins all-equal ties).

Each item reports how many of its runs passed, not only whether one did:

"runs": 5, "runs_passed": 4, "pass_at_k": true, "pass_power_k": false

pass_at_k is "did any run pass" and is what passed counts. runs_passed is the fact that separates a reliable item from a coin-flip — without it a 5/5 item and a 1/5 item are identical in every field, because quality_score is derived from the best run alone. pass_power_k is true only when every run passed, and the run summary carries passed_all_runs beside passed; a large gap between the two means the model is inconsistent rather than wrong. The console shows 4/5 runs passed in Notes for a non-unanimous pass and stays quiet for a unanimous one, and its summary line reads 3/4 passed, 1 on every run.

runs is what the item actually ran, which is not always the requested --runs: agentic_conversation takes no K and drives its fixture exactly once.

Each item additionally carries a per-phase breakdown:

"latency_breakdown_s": {
  "agent_s": 4.02,          // GoodData's own response time — the system under test
  "judge_s": 1.31,          // LLM-as-judge scoring, post-hoc
  "simulated_user_s": 0.0,  // our simulated user composing the next turn (multi-turn kinds)
  "langfuse_s": 5.70        // trace lookup + score writing, off the critical path
}

Every item reports runs_ungraded beside runs_passed: runs the agent answered but the LLM judge returned nothing readable for. Agentic items also list them as unscored_runs / judge_errors in their detail, and a dashboard_summary item carries ungraded_criteria. Such a run — or, for dashboard_summary, such a criterion — is excluded from pass@K and from the quality score rather than counted as a failure: scoring it 0 would be indistinguishable from the judge genuinely failing the answer, which is the confusion JudgeResponseError exists to end. pass@K still holds on the runs that were graded, so an item can pass with runs_ungraded set; pass^K cannot, because a run nobody graded leaves "all K passed" unverified. For dashboard_summary an ungraded must_include / must_not_include criterion likewise cannot carry a pass — "the judge could not tell" is not evidence the fact is present — while an ungraded rubric line only narrows the quality score. When no run could be graded (for dashboard_summary: no gating criterion on any run) the item errors instead of reporting failures. A non-zero count means pass@K was computed over fewer runs than --runs asked for, so treat the result as weaker evidence and check the judge (GD_EVAL_JUDGE_DIAGNOSTICS=1, or raise JUDGE_MAX_COMPLETION_TOKENS if the cause is finish_reason=length). The console says so in Notes (1 run(s) ungraded, 2 criterion(s) ungraded).

agent_s + judge_s + simulated_user_s are the instrumented parts of the item's latency_s; they do not add up to it exactly, because latency_s is wall-clock around the whole item and also covers the conversation create/delete round trips, SDK construction and any cleanup. langfuse_s sits beside latency_s, never inside it, because trace linking runs outside every item's critical path (see above) — summing all four would re-inflate exactly what that design removes.

A phase that a kind does not have reports 0.0 rather than an invented number, so read the zeroes as "not applicable here", not "instant". Today:

The other six agentic kinds report 0.0 for the first three. Trace linking itself happens whenever LANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY are exported, with or without --langfuse, because each evaluate_agentic_* falls back to try_make_langfuse_client(). But its duration is measured by the CLI runner rather than by evaluate_agentic_*, so a direct library caller sees langfuse_s: 0.0 even though its linking ran. Pass TAVERN_E2E_SKIP_TRACE_LINK=1 to opt out of linking altogether.

gd-eval models

List all LLM providers and their models in the org. Marks the active model for a workspace when --workspace is given:

gd-eval models \
  --host  https://your.gooddata.cloud \
  --workspace  ecommerce_demo

┃ Provider       ┃ Provider ID ┃ Model ID          ┃ Family    ┃ Active   ┃
│ Foundry4o      │ foundry_…   │ gpt-5.2           │ OPENAI    │ ◀ active │
│                │             │ gpt-4o            │ OPENAI    │          │
│ HN_Anthropic   │ hn_anthr_…  │ claude-opus-4-7   │ ANTHROPIC │          │

Dataset format

A dataset is a folder of .json files, one per question:

{
  "id":           "stable-unique-id",
  "dataset_name": "my_dataset",
  "test_kind":    "visualization",
  "question":     "Show revenue by quarter",
  "expected_output": { }
}

Supported test_kind values: visualization, metric_skill, alert_skill, search_tool, general_question, guardrail, dashboard_summary.

dashboard_summary items

Summary items call the dedicated summary endpoint (POST /api/v1/ai/workspaces/{ws}/summary) instead of the chat endpoint, so they carry an extra summary_input block, and the expected_output is a rubric rather than an exact answer (summaries are free text):

{
  "id": "summary-001",
  "dataset_name": "summary_pilot",
  "test_kind": "dashboard_summary",
  "question": "Summarize the Sales Overview dashboard.",
  "summary_input": {
    "dashboard_id": "sales_overview"
  },
  "expected_output": {
    "must_include":     ["States the overall revenue trend", "Identifies the top segment"],
    "must_not_include": ["Numbers or segments not present in the visualizations"],
    "rubric":           ["Reads as a coherent business summary"]
  }
}

summary_input requires only dashboard_id (the endpoint summarizes the whole dashboard). Optional fields narrow the scope: visualizations (list of ids), filter_context (AFM filters), tab_id, and format_hint.

The expected_output rubric:

• must_include — facts a good summary must contain; all must pass for the item to pass.
• must_not_include — hallucination/accuracy guards; any violation fails the item.
• rubric — soft quality dimensions; they affect quality_score but do not gate pass/fail.

Each criterion is scored independently by the LLM judge, so quality_score is the fraction of satisfied criteria.

Supported test kinds

Optional extras

[llm-judge] — LLM-as-judge evaluators

general_question and guardrail items are scored by a GPT-4o judge. Requires the OpenAI package and OPENAI_API_KEY:

uv add 'gooddata-eval[llm-judge]'
# or for the standalone tool:
uv tool install 'gooddata-eval[llm-judge]'

Without [llm-judge], those items are skipped.

Exit codes

Scores (in JSON report and Langfuse)