WorthIR

English | 简体中文

WorthIR compares query-level routing policies with fixed retrieval strategies under a declared effectiveness measure and cost profile. It reports effectiveness, cost, utility, regret, and the fixed-route Pareto curve.

If you are an AI tool, read README_FOR_AI.md before searching the repository.

60-second demo

Python 3.10 or newer is required. Choose one installation path.

Source archive or Git clone: run the local launcher. It creates the local environment when first needed.

.\worthir.cmd demo-custom

./worthir demo-custom

PyPI: install the released package, then use the global command.

python -m pip install worthir-eval==1.2.1
worthir demo-custom

If the worthir command is not on PATH, use the equivalent module entry:

python -m worthir demo-custom

Do not run both setup paths. Open reproduced/custom_task/comparison.md after the command finishes. The published wheel uses English terminal messages; the Chinese source branch provides Chinese launchers and documentation.

Use your own task

Prepare task.json, queries.csv, routes.csv, and outcomes.csv as shown in examples/custom_task/source/, then run:

.\worthir.cmd build-custom my_source my_task
.\worthir.cmd validate-task my_task
.\worthir.cmd evaluate my_task choices.csv --policy-id my-router

This path accepts any named higher-is-better effectiveness measure, arbitrary route prerequisites, fixed or query-dependent costs, and either cumulative or incremental cost input. The router receives queries.csv, the public route registry, lambda, and any costs declared as known at commitment time. Evaluator outcomes and costs measured only after execution remain separate.

For qrels and six-column TREC runs, use the shorter build-trec walkthrough. All input formats are described in docs/ADAPT_TO_NEW_TASK.md. For direct library use, see the worthir_eval Python example.

Recompute the paper results

This uses released query--route ledgers and frozen route selections. It does not redownload corpora or rerun retrieval models.

python paper_results/run.py

Open paper_results/reproduced/INDEX.md. The index names the exact paper version, caption, output, and reproduction level for every main-paper and appendix figure or table.

Rebuild the original retrieval routes

This is a separate, resource-intensive workflow. It checks licensed corpora and checkpoints, invokes a configured task adapter, and constructs new query--route ledgers through five explicit stages. Start with paper_results/full_replay/README.md and its task-specific resource estimates. Raw corpora, indexes, and model weights are not included in this repository.

FiQA-Compression260 is directly runnable from the official public corpus and models; see the FiQA260 route-rebuild guide.

The v1.2.1 release is the published artifact currently bound to the 2026-08-16 paper mapping. The earlier v1.0.0-ipmc2026 release remains the artifact submitted with the accepted IP&MC 2026 paper.

WorthIR-authored code is released under the MIT License. Third-party data and model terms are listed in NOTICE.