📟 longeval-cli

Build dynamic test collections from search logs. longeval-cli turns raw search logs into a sequence of self-contained snapshots, each with its own queries, relevance labels (qrels), and documents. It is introduced in our paper Building Dynamic Test Collections: LongEval CLI and Scientific Retrieval Dataset (under review) to which we refer for the methodological background.

The pipeline

Each stage stores its results in the database or in files, so you can stop, inspect, and continue at any stage, and add a new snapshot without reprocessing the earlier ones.

Everything that depends on your search engine (parser, classifiers, document source) is a small Python file you pass on the command line, so the toolkit itself stays format-independent. The exported snapshots can be loaded with ir_datasets_longeval.

Installation

# from a clone of this repository
uv sync                      # base install
uv sync --extra clax         # click-model training and scoring (JAX, CLAX)
uv sync --extra postgres     # if your database is PostgreSQL
uv sync --extra tira         # for `retrieve`

The database defaults to a local SQLite file. Pass --database-url (or set LONGEVAL_DATABASE_URL) to use another file or PostgreSQL, e.g. postgresql+psycopg://user:password@host/database.

Quickstart

Run the first stages on the synthetic logs that ship with the repository:

export DB=sqlite:////tmp/minimal-example.sqlite3 E=examples/minimal-example

uv run longeval-cli init-db --database-url "$DB"
uv run longeval-cli import-queries $E/logs --parser $E/parser.py --database-url "$DB"
uv run longeval-cli import-clicks  $E/logs --parser $E/parser.py --database-url "$DB"
uv run longeval-cli classify --classifier $E/classifiers.py --filter $E/filter.yaml --database-url "$DB"
uv run longeval-cli database-status --database-url "$DB"

The tutorial continues from here through click-model training, qrels, documents, and the packaged snapshot, with the expected output of every step.

Documentation

The agent skills double as how-to guides for the four tasks you will do most:

• longeval-log-parser: write a parser for a new log format
• longeval-classifier: write or configure classifiers
• longeval-snapshot-filter: write a snapshot's filter YAML
• longeval-build-snapshot: build and release a snapshot end to end

Roadmap

• Query intent classification
• Query recency classification
• Topic generation
• Sampling instead of pooling
• Generalize filters.yml into a snapshot config file to reduce options needed

License

MIT, see LICENSE.