Contents

What is this? · Who it's for · Install and run · 1. Overview · 2. Experiment · 3. Evaluate · Bring your own domain · Configuration · Architecture · Development · Privacy & security · Limitations · License

What is this?

Before an AI approves, denies or escalates something, it usually needs to check records. Every check costs money and time, and every wrong decision costs more. Checking everything is safe but wasteful. Checking nothing is cheap but reckless. Yantraa Evidence gives you a controlled way to find the middle ground and to prove it with numbers.

You get three building blocks. Use them from Python, a notebook or the CLI, and build any app you like on top:

flowchart LR
    A[Request] --> B{Strategy decides<br/>next step}
    B -- check a source --> C[(Evidence sources<br/>defined by the domain)]
    C -- evidence --> B
    B -- stop --> D[Shared decision-maker<br/>domain labels]
    D --> E[Grade vs answer key]
    E --> F[Total cost =<br/>checks + AI + mistake penalty]
    F --> G[Report · paired 95% CIs ·<br/>vs baselines and past runs]

Built-in guarantees:

• 🔒 Strict information boundary. The decision-maker sees only what has been fetched, never the answer key or unfetched records. Tests enforce this.
• ⚖️ Fair comparison. Every strategy shares the same decision-maker, instructions and limits.
• 🧯 Fails safe and loudly. Limits are checked before each call. Any failure returns the domain's fallback label (for example escalate) with the reason recorded. There are no silent retries.
• 📏 Honest statistics. Paired, seeded bootstrap intervals and penalty-sensitivity analysis. Unknown prices are reported as unknown, never as zero.
• 🧾 Reproducible. Every run stores its config hash, seed, data and prompt versions, model version and full trace.

Who it's for

More detail, with evidence sources and mistake costs for each industry: docs/use-cases.md.

Install and run

Prerequisites: Python 3.11 or newer, plus uv (recommended) or pip. No API key is needed: the built-in mock decision-maker is free and offline.

Install from PyPI (run the commands from the folder where you want data and results kept):

# With pip (inside a virtual environment)
pip install yantraa-evidence
yantraa overview

# With uv, as a dependency of your own project
uv add yantraa-evidence
uv run yantraa overview

# With uv, as a standalone tool (no project needed)
uv tool install yantraa-evidence     # or try it once: uvx --from yantraa-evidence yantraa overview
yantraa overview

Then continue with yantraa experiment and yantraa evaluate (see step 2 below), or import yantraa_evidence in Python.

From a source checkout (to change the code or run the tests):

1. Get the code and install it:

   git clone https://github.com/Shalihotri/Yantraa.git
   cd Yantraa
   uv sync
   

2. Run the three steps:

   uv run yantraa overview                  # what's available
   uv run yantraa experiment --limit 40     # run all strategies on 40 synthetic cases (data is generated on first run)
   uv run yantraa evaluate                  # report the latest experiment against the fetch_all baseline
   

3. Or run the same flow from Python: uv run python examples/quickstart.py

4. (Optional, for a real AI) copy .env.example to .env and set OPENAI_API_KEY=.... Then add --provider openai to an experiment. Always check the spend first with --estimate-only.

Source checkout without uv: python -m venv .venv, activate it, then run pip install -e .. After that, use yantraa ....

Real AI with a pip install: put a .env file containing OPENAI_API_KEY=... in the folder you run yantraa from, or set the variable in your shell.

1. Overview

from yantraa_evidence import overview
from yantraa_evidence.overview import to_text

info = overview()            # plain dict: domain, sources & costs, strategies, providers, data, saved results
print(to_text(info))

yantraa overview prints the same information. Add --json for machine-readable output, or --problems 20 for the latest logged failures.

2. Experiment

from yantraa_evidence import Experiment

exp = Experiment(
    domain="returns",                    # or "my_pkg.domains:CLAIMS" or "path/file.py:DOMAIN"
    split="dev",                         # tune on dev; keep test locked until the end
    strategies=["selective", "fixed_order", "fetch_all"],
    provider="mock",                     # "openai" (Decisions API) or "openai-responses"
    limit=40,
    overrides={"costs": {"get_inspection": 0.25}, "loss": {"approve": 50}},  # your economics
)
print(exp.estimate())                    # worst-case calls and spend, before anything runs
result = exp.run()                       # scored, saved to SQLite, exported to CSV / JSONL / JSON
trace = exp.run_case(0, "selective")     # one decision, fully traced

Built-in strategies: selective (the AI picks the next check: the idea being tested), fetch_all (reference baseline), fixed_order (checklist, stop early), confidence_threshold (stop when the AI is sure) and initial_only (no checks: the lower bound).

Add your own strategy in a few lines. It is then available to Experiment and the CLI:

from yantraa_evidence import register_strategy

@register_strategy
def cheapest_first(s):
    """Check sources from cheapest to most expensive; stop once the decision is no longer the fallback."""
    d = s.classify()
    for src in sorted(s.sources, key=lambda x: s.cfg.sources[x].cost):
        if d.action != s.domain.fallback:
            break
        s.fetch(src)
        d = s.classify()
    return d

CLI:

yantraa experiment --split dev --strategies selective,fetch_all --limit 40
yantraa experiment --provider openai --limit 10 --estimate-only
yantraa experiment --case 0 --strategies selective        # one traced decision
yantraa experiment --generate --seed 7                    # regenerate synthetic cases

3. Evaluate

from yantraa_evidence import evaluate, compare_to_baseline

report = evaluate(result)                 # or evaluate("b_2026...") for a saved experiment
print(report.to_text())                   # headline, scoreboard, paired verdicts vs fetch_all, caveats
report.metrics                            # accuracy, wrong_<label>, fallback_rate, checks, costs, latency
report.paired_comparisons                 # each strategy minus the baseline, with 95% intervals
report.penalty_sensitivity                # does the winner change if mistakes cost 0.25x to 4x?
report.accuracy_by_scenario, report.mistakes

compare_to_baseline(new_result, "b_2026...")   # new prompt / model / costs vs a saved benchmark, paired by case

CLI:

yantraa evaluate                          # latest experiment
yantraa evaluate --list                   # saved experiments
yantraa evaluate b_NEW --against b_OLD    # regression check against an earlier benchmark
yantraa evaluate --reference fixed_order --json

Every report repeats its caveats: synthetic data, assumed penalties, mock ≠ model quality, confidence ≠ calibration.

Bring your own domain

A domain is one class. Declare the sources, labels, fallback and penalties, then implement assess. That one function produces the answer key and drives the offline mock, so the two can never disagree.

from yantraa_evidence import Assessment, Domain, Source

class Chargebacks(Domain):
    name = "chargebacks"
    task = "Decide whether to accept a card chargeback."
    sources = (Source("get_transaction", "Transaction log", cost=0.01),
               Source("get_merchant_evidence", "Merchant's delivery proof", cost=0.20),
               Source("get_customer_history", "Prior disputes", cost=0.05))
    actions = {"accept": "...", "reject": "...", "review": "..."}
    fallback = "review"
    loss = {"accept": 80, "reject": 30, "review": 3}

    def assess(self, evidence):           # evidence = {source_id: [payload, ...]} for fetched sources
        ...
        return Assessment(action, conditions)

To use it, pass Experiment(domain="my_module:Chargebacks") or --domain path/to/file.py:DOMAIN. Alternatively, call register_domain(Chargebacks()).

• Data: implement generate(seed) for synthetic cases, or write your own Case records (with the correct answer in oracle_action) to data/<domain>/<split>.jsonl. You can also pass Experiment(cases=[...]).
• Full worked example: examples/insurance_claims.py covers claims triage with a generator, and runs in CI.
• Another model: implement select(obs) and classify(obs), then pass Experiment(adapter_factory=...).

Configuration

Unknown source or label names in config fail loudly instead of being ignored.

Architecture

src/yantraa_evidence/
  overview.py      1. what is available (domains, strategies, providers, data, results)
  experiment.py    2. Experiment / ExperimentResult: design, estimate, run, save
  evaluate.py      3. scoring, metrics, paired bootstrap, sensitivity, Report, compare_to_baseline
  domain.py        Domain / Source / Assessment base classes and the domain registry
  domains/         built-in domains (returns: retail product returns, with a seeded generator)
  strategies.py    built-in strategies + @register_strategy
  adapters.py      decision-makers: mock (offline) and OpenAI (Decisions, Responses)
  runner.py        guarded execution engine: Session, limits, validation, tracing
  tools.py         case-data-backed evidence tools (same interface a live connector would use)
  datasets.py      case files (JSONL) and synthetic data generation
  storage.py       SQLite results + evidence graph, CSV/JSONL/JSON export
  schemas.py       typed contracts: Case, Observation, decisions, RunResult
  config.py        YAML config, domain defaults, pricing, .env
  logtracker.py    structured "what went wrong" log (keys redacted)
  cli.py           `yantraa overview | experiment | evaluate`
examples/          quickstart.py · insurance_claims.py (custom domain)
docs/              design.md (method, guarantees, findings log) · use-cases.md (industries and personas)
tests/             isolation, limits, failures, graph round-trip, known answers, custom domains, CLI

Read docs/design.md before citing a result.

Development

uv sync                                    # Python env with dev tools
uv run pytest                              # all tests, offline, no API key
uv run ruff check src tests examples       # lint
uv build                                   # wheel + sdist in dist/

CI (.github/workflows/ci.yml) runs lint and tests on Linux and Windows with Python 3.11 and 3.12. It also runs the Overview → Experiment → Evaluate flow offline for both domains, a wheel-install smoke test and a secret scan. See CONTRIBUTING.md.

Privacy & security

• Everything runs locally. There is no server, no account and no telemetry.
• Your API key stays on your machine. It is read from the environment or .env, and it is never logged or stored. Provider error messages are redacted before logging. .env, databases, logs and results are git-ignored.
• Only live decision-makers send data out. With --provider openai, the fetched evidence for each case goes to OpenAI (store=False where the API allows). The mock sends nothing.
• Bundled data is synthetic. It contains no real people, orders or personal information. If you use real records, anonymise them first and keep them in a git-ignored folder (for example private/).

Limitations

• Synthetic data is balanced for debugging. It is not a production distribution.
• Default mistake penalties are illustrative. Set your own, and read the sensitivity table.
• The mock uses the same rule as the answer key: it validates mechanics, not model quality.
• An AI's stated confidence is not calibrated correctness. The tool supports comparisons, not causal claims.
• Live results so far are small-sample. See the findings log.

Roadmap

• More built-in domains (chargebacks, access requests) and adapters for other model providers
• Live evidence connectors behind the same tool interface
• Larger live runs on the practice set, then a single locked run on the held-out set
• External validation on real, anonymised decision logs

Contributing

Issues and pull requests are welcome. Please read CONTRIBUTING.md. In short: keep the information boundary intact, add tests, and keep negative findings visible.

License

Apache-2.0.