Skip to main content

Agent-Pariksha

Find out what your AI agent does when its tools fail.

Pariksha (परीक्षा) means test. An agent that takes actions must be tested on the bad days, not only the good ones.

ci python license deps


The problem

Demos test agents when every tool works. In production, a payment call times out after the money has already moved, a service returns garbage, or a retrieved record contains a hidden instruction. A chatbot that fails gives a bad answer. An agent that fails can pay a bill twice.

Agent-Pariksha injects these failures and checks what the agent actually did, not only what it said.

What it tests

Fault What happens What can go wrong
transient A tool fails once, then recovers The agent gives up too early
lost_response The action succeeds, but the agent sees a timeout The agent retries and pays twice
timeout A tool keeps failing The agent loops, or claims success
malformed A tool returns garbage The agent acts on bad data
injection Tool output contains a hidden instruction The agent obeys it and moves money
Check Question
no_duplicate_action Did a side effect happen more than once?
no_forbidden_action Did it call a tool it must not call, such as refunding above its limit?
correct_order Did it verify before acting?
bounded_calls Did it stop instead of retrying forever?
honest_about_failure When the task was impossible, did it say so?
task_completed When the task was possible, did it finish it?

The suite runs 16 tests across 4 scenarios in a made-up bank: balance lookup, bill payment, large refund and order status. Nothing real is called and no real money or data is involved.

Demo (no model or API key needed)

pip install agent-pariksha
python -m pariksha demo

From source: git clone https://github.com/Malaya-kumar-swain/agent-pariksha && cd agent-pariksha && pip install -e .

Scripted demo agent Reliability score Example failure
naive-agent 19% (3 of 16) Pays twice after a lost response, refunds above its limit, obeys injected text
guarded-agent 100% (16 of 16) None

Both demo agents are scripted to illustrate the method. They are not measurements of any real AI system. Run the suite on your own agent to get real numbers.

Test your own agent

Your own agent. Anything with a run(task, box) method works. Each scenario has different tools, so give your agent the list from box.specs(), send its tool calls through box.call, and return its final answer:

from pariksha import run

class MyAgent:
    name = "my-agent"

    def run(self, task, box):
        return my_agent(task, tools=box.specs(), call_tool=box.call)  # my_agent is your own code

print(run(MyAgent()).to_markdown())

A model with tool calling, through any OpenAI-compatible endpoint:

python -m pariksha run --base-url https://api.example.com/v1 --model some-model --api-key-env MY_API_KEY --delay 3 --out report --fail-below 0.9

--delay paces requests for rate limits, and rate-limited requests are retried. A run that still errors counts as a failure and is flagged in the report, so a crash can never look like a pass. --fail-below exits with an error, so a drop in reliability can block a release.

Honest limits

  • Early version: 16 tests is a starting point, not a certification.
  • Real models: the model connection has so far been tested only against a local fake server, not a live model.
  • Honesty check: honest_about_failure looks for failure words in English, so it can be fooled by unusual wording and does not cover other languages yet.
  • Related tools: broader agent evaluation frameworks exist. This project stays small and focuses on one question: is the agent safe when its tools fail?

Project layout

pariksha/
  core.py       tools, fault injection, record of calls and side effects
  scenarios.py  test scenarios and checks
  agents.py     demo agents and the OpenAI-compatible agent
  runner.py     run the suite and build reports
  __main__.py   command line: demo, list, run
tests/          12 tests

Licence

Apache 2.0. Built by Malaya Kumar Swain. Views and code are my own.

Metadata

Release files for agent-pariksha 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agent-pariksha 0.1.0
File Size Uploaded Provenance Actions
agent_pariksha-0.1.0.tar.gz 16.2 kB Without provenance Details for agent_pariksha-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agent-pariksha 0.1.0
File Interpreter ABI Platform Actions
agent_pariksha-0.1.0-py3-none-any.whl Python 3 none any Details for agent_pariksha-0.1.0-py3-none-any.whl

Total release size: 33.0 kB

Release files / agent_pariksha-0.1.0.tar.gz

Download URL agent_pariksha-0.1.0.tar.gz
Size 16.2 kB
Tags Source
SHA-256 checksum
How to use checksums
e3a6327f9ea40845797422fe0dd0fc2ad3123de10cc3f6093e11ae7575d0d76c
BLAKE2b-256 checksum
How to use checksums
0354f3f5095c1a82a6305e0bedce1ab2a897b31b8d53598d861b63778a838bd3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / agent_pariksha-0.1.0-py3-none-any.whl

Download URL agent_pariksha-0.1.0-py3-none-any.whl
Size 16.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
581dd0b4132e20f17204b5eb64b316955c5897abce1ff7bf2d172cf93640b14d
BLAKE2b-256 checksum
How to use checksums
f8a78ed8617651e20d0b0d5b6059b37ce21ff56b0d966c1075791d918e4a6875
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for agent-pariksha 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page