WorthIR

Python 3.10+

WorthIR compares query-level routing policies with fixed retrieval strategies under a declared effectiveness measure and cost profile. It reports effectiveness, cost, utility, regret, and the fixed-route Pareto curve.

If you are an AI tool, read README_FOR_AI.md before searching the repository.

60-second demo

Python 3.10 or newer is required. Choose one installation path.

Source archive or Git clone: run the local launcher. It creates the local environment when first needed.

.\worthir.cmd demo-custom

./worthir demo-custom

Installed wheel: install the wheel, then use the global command.

python -m pip install worthir_eval-1.2.0-py3-none-any.whl
worthir demo-custom

Do not run both setup paths. Open reproduced/custom_task/comparison.md after the command finishes. The published wheel uses English terminal messages; the Chinese source branch provides Chinese launchers and documentation.

Use your own task

Prepare task.json, queries.csv, routes.csv, and outcomes.csv as shown in examples/custom_task/source/, then run:

.\worthir.cmd build-custom my_source my_task
.\worthir.cmd validate-task my_task
.\worthir.cmd evaluate my_task choices.csv --policy-id my-router

This path accepts any named higher-is-better effectiveness measure, arbitrary route prerequisites, fixed or query-dependent costs, and either cumulative or incremental cost input. The router receives queries.csv, the public route registry, lambda, and any costs declared as known at commitment time. Evaluator outcomes and costs measured only after execution remain separate.

For qrels and six-column TREC runs, use the shorter build-trec walkthrough. All input formats are described in docs/ADAPT_TO_NEW_TASK.md. For direct library use, see the worthir_eval Python example.

Recompute the paper results

This uses released query--route ledgers and frozen route selections. It does not redownload corpora or rerun retrieval models.

python paper_results/run.py

Open paper_results/reproduced/INDEX.md. The index names the exact paper version, caption, output, and reproduction level for every main-paper and appendix figure or table.

Rebuild the original retrieval routes

This is a separate, resource-intensive workflow. It checks licensed corpora and checkpoints, invokes a configured task adapter, and constructs new query--route ledgers through five explicit stages. Start with paper_results/full_replay/README.md and its task-specific resource estimates. Raw corpora, indexes, and model weights are not included in this repository.

The v1.2.0 release is the published artifact currently bound to the 2026-08-16 paper mapping. The earlier v1.0.0-ipmc2026 release remains the artifact submitted with the accepted IP&MC 2026 paper.

WorthIR-authored code is released under the MIT License. Third-party data and model terms are listed in NOTICE.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

worthir_eval-1.2.0.tar.gz (37.3 kB view details)

Uploaded Aug 24, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

worthir_eval-1.2.0-py3-none-any.whl (54.4 kB view details)

Uploaded Aug 24, 2026 Python 3

File details

Details for the file worthir_eval-1.2.0.tar.gz.

File metadata

Download URL: worthir_eval-1.2.0.tar.gz
Upload date: Aug 24, 2026
Size: 37.3 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for worthir_eval-1.2.0.tar.gz
Algorithm	Hash digest
SHA256	`ebcab11d2bf07a24876089118cff2dd2ef90ce05421eda94fcee39ec8235293d`
MD5	`bf132ecaddc7f1713a1295aab2917d0f`
BLAKE2b-256	`b058c079e287af650f2d29ec0c5e7b2ccbbe9a1ed365cb634c3ed078a45501c4`

See more details on using hashes here.

Provenance

The following attestation bundles were made for worthir_eval-1.2.0.tar.gz:

Publisher: publish.yml on HongTianShi/worthir-artifact

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: worthir_eval-1.2.0.tar.gz
- Subject digest: ebcab11d2bf07a24876089118cff2dd2ef90ce05421eda94fcee39ec8235293d
- Sigstore transparency entry: 2579309213
- Sigstore integration time: Aug 24, 2026
Source repository:
- Permalink: HongTianShi/worthir-artifact@25242918bb4f8d889715301703f30698f5d06c6b
- Branch / Tag: refs/tags/v1.2.0
- Owner: https://github.com/HongTianShi
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@25242918bb4f8d889715301703f30698f5d06c6b
- Trigger Event: release