WorthIR

English | 简体中文

WorthIR evaluates query-level retrieval routing under a declared effectiveness measure, cumulative route costs, and cost preference. It compares a frozen routing policy with fixed routes and reports effectiveness, cost, utility, regret, and the fixed-route Pareto curve.

If you are an AI tool, read README_FOR_AI.md before searching the repository.

Choose the reproduction scope

Start from zero

Python 3.10 or newer is required. The recommended path is PyPI:

python -m pip install worthir-eval==1.3.1
worthir demo-custom

If worthir is not on PATH, run python -m worthir demo-custom.

To use the pinned source tree instead:

git clone --branch v1.3.1 --depth 1 https://github.com/HongTianShi/worthir-artifact.git
cd worthir-artifact
./worthir demo-custom

On Windows, replace the last line with .\worthir.cmd demo-custom. The source launcher creates .venv on first use; do not install the PyPI package as an additional setup step.

A successful run ends with lines like these:

BUILT: .../reproduced/custom_task
WROTE: .../reproduced/custom_router_choices.csv
OPEN: .../reproduced/custom_task/comparison.md

It creates:

reproduced/
├── custom_router_choices.csv
└── custom_task/
    ├── contracts/              route definitions and task contract
    ├── participant/            legal router inputs and frozen actions
    ├── evaluator/ledger.csv    organizer-only query--route outcomes
    ├── comparison.csv
    ├── comparison.md
    └── fixed_routes.csv

The report includes the router and every fixed route, for example:

| Policy              | Effectiveness | Cost   | Utility | Delta U vs. fixed |
| example-rule-router | 0.9000        | 0.1507 | 0.8774  | +0.0625           |

Evaluate your own task

Copy the four files in examples/custom_task/source/:

• task.json: metric, lambda, declared sensitivity grids, and fixed reference;
• queries.csv: one row per query with only information legal at route-selection time;
• routes.csv: route labels, prerequisites, costs, and the development-selected route;
• outcomes.csv: organizer-only effectiveness and cost for every query--route pair.

Build and check the task:

worthir build-custom my_source my_task
worthir validate-task my_task

Your router—not WorthIR—reads my_task/participant/ and writes choices.csv. The minimum format is:

query_uid,selected_route_id
q001,base
q002,rerank

Bind those frozen choices to the task contract and compare them with all fixed routes:

worthir evaluate my_task choices.csv --policy-id my-router

The complete router example is under examples/custom_router/. For qrels and six-column TREC runs, use build-trec. All generic input formats are documented in docs/ADAPT_TO_NEW_TASK.md.

Organizer analyses

These commands join frozen actions with the evaluator ledger. They default to my_task/organizer_private/ and refuse to write under participant/:

worthir analyze my_task --organizer-output my_task/organizer_private/per_query_scores.csv
worthir sensitivity my_task
worthir budget my_task
worthir plot my_task

analyze reports selected outcomes, the development-fixed reference, oracle route, regret, and opportunity stratum for each query. sensitivity and budget use the grids declared in task.json; a command-line grid is labeled non-prespecified. plot writes a dependency-free SVG Pareto chart. Every output is marked descriptive and evaluator_only. Parquet output is available when pyarrow is installed; CSV requires no extra package.

See docs/OUTPUTS.md for field definitions and docs/TROUBLESHOOTING.md for installation and download failures.

Recompute the paper tables and figures

This step requires a source checkout because the released query--route ledgers are not part of the core PyPI wheel:

python paper_results/run.py

Open paper_results/reproduced/INDEX.md. It links every paper figure and table to its input, command, output, and reproduction level.

Rebuild retrieval routes

Raw corpora, indexes, and model weights are not distributed here. The staged rebuild interface and resource estimates are in paper_results/full_replay/README.md. FiQA-Compression260 has a runnable public-corpus adapter and a CPU installation path in FIQA260.md.

WorthIR code uses the MIT License. Third-party data and model terms are listed in NOTICE.