path-index 0.0.1

PATH-Index: Pathwise Assessment of Trajectory Health - a diagnostic framework for the execution integrity of AI agent trajectories.

PATH-Index

Pathwise Assessment of Trajectory Health — a diagnostic framework for measuring the execution integrity of AI agent trajectories.

📄 Companion paper: PATH-Index: Decomposing Where and Why AI Agents Fail (in preparation; preprint link coming soon)

PATH-Index is the trajectory-level sibling of VerifyIndex. The two frameworks form one measurement program:

• VerifyIndex evaluates the evidential integrity of AI outputs (claim-level).
• PATH-Index evaluates the execution integrity of AI agent trajectories (step-level, pathwise).

"Pathwise" is meant in the stochastic-process sense: quality is assessed along each individual trajectory, not merely in expectation over tasks.

Why another agent evaluation framework?

Most agent benchmarks report aggregate task success. That tells you whether an agent failed, but not where in the trajectory the failure originated, how far it propagated, or whether the agent recovered. PATH-Index is designed for failure localization: separating the causal origin of a failure from its downstream manifestation, and reporting a diagnostic profile rather than a single leaderboard number.

The Trajectory Funnel

Each step of a trajectory passes through five nested gates, evaluated in order:

Gate	Question it answers
G — Goal Alignment	Is the step's sub-goal well-formed and consistent with the user's task, as evidenced by observable actions?
S — State Grounding	Does the agent have (and use) the environment state and context the step requires?
T — Tool Selection	Given goal and grounding, is the chosen tool (or "respond directly") appropriate?
A — Argument Fidelity	Are the tool arguments correct, well-typed, and consistent with the grounded state?
O — Outcome	Did the call produce its specified effect (execution validity), as opposed to erroring, no-op-ing, or silently corrupting state?

Because the gates are nested, step yield factorizes as a product of sequential conditionals:

H_T = P(G) · P(S | G) · P(T | S) · P(A | T) · P(O | A)

Trajectory Health is reported as a profile across these stages plus cross-step diagnostics — not as a single scalar. A composite score is optional and derived, never the headline.

Beyond the funnel: cross-step diagnostics

The reported profile also includes quantities that live across steps rather than inside the yield identity:

• Failure persistence (how long a failure run lasts) — reported separately from
• Causal propagation (counterfactually defined downstream damage, requiring paired trajectories)
• Recovery — binary recovery events and graded recovery quality, reported as distinct objects
• Oversight — recall, precision, and false-defer rate of escalation/deferral behavior
• Cost profile — a vector over steps, tool calls, tokens, latency, and monetary cost
• Reliability — PassAll@k and PassAny@k across repeated runs

Diagnostic evaluation is scored against six metrics, including origin identification (Top-1 and Top-2 recall), manifestation F1, affected-stage recall, propagation-depth error, and recovery-time error.

Status

🚧 Pre-release. The founding paper is in preparation. This repository reserves the name and will host:

• Paper preprint and citation info
• Reference implementation (annotation rubrics, scoring pipeline)
• Calibration / pilot-threshold simulation code
• Example annotated trajectories

Watch or star the repo to be notified when the preprint and reference implementation land.

Citation

Citation information will be added when the preprint is available. In the meantime, please reference this repository:

@misc{pathindex2026,
  title  = {PATH-Index: Decomposing Where and Why AI Agents Fail},
  author = {Srivastava, V.},
  year   = {2026},
  note   = {Framework repository: https://github.com/vsrivas7/path-index}
}

Related

• VerifyIndex — sequential measurement of claim-level Verification Yield for AI outputs.

License

TBD (code will be released under a permissive open-source license; annotation rubrics and paper text under CC BY 4.0).