📟 longeval-cli

Build dynamic test collections from search logs. longeval-cli turns raw search logs into a sequence of self-contained snapshots, each with its own queries, relevance labels (qrels), and documents. It is introduced in our paper Building Dynamic Test Collections: LongEval CLI and Scientific Retrieval Dataset (under review) to which we refer for the methodological background.

The pipeline

Each stage stores its results in the database or in files, so you can stop, inspect, and continue at any stage, and add a new snapshot without reprocessing the earlier ones.

Everything that depends on your search engine (parser, classifiers, document source) is a small Python file you pass on the command line, so the toolkit itself stays format-independent. The exported snapshots can be loaded with ir_datasets_longeval.

Installation

pip install longeval-cli                 # base install
pip install "longeval-cli[clax]"         # click-model training and scoring (JAX, CLAX)
pip install "longeval-cli[postgres]"     # if your database is PostgreSQL
pip install "longeval-cli[tira]"         # for `retrieve`

Extras combine, e.g. pip install "longeval-cli[clax,postgres]"; with uv, use uv add longeval-cli or uv tool install longeval-cli. On Linux, the clax extra pulls in PyTorch, whose default wheels bring ~4 GB of CUDA libraries which are not really needed. To skip them, install the CPU build first: pip install torch --index-url https://download.pytorch.org/whl/cpu.

To run the examples (the quickstart below and the tutorial) or to develop, work from a clone of this repository instead:

git clone https://github.com/clef-longeval/longeval-cli.git && cd longeval-cli
uv sync                      # base install
uv sync --extra clax         # likewise --extra postgres / --extra tira

The database defaults to a local SQLite file. Pass --database-url (or set LONGEVAL_DATABASE_URL) to use another file or PostgreSQL, e.g. postgresql+psycopg://user:password@host/database.

Quickstart

Run the first stages on the synthetic logs that ship with the repository:

export DB=sqlite:////tmp/minimal-example.sqlite3 E=examples/minimal-example

uv run longeval-cli init-db --database-url "$DB"
uv run longeval-cli import-queries $E/logs --parser $E/parser.py --database-url "$DB"
uv run longeval-cli import-clicks  $E/logs --parser $E/parser.py --database-url "$DB"
uv run longeval-cli classify --classifier $E/classifiers.py --filter $E/filter.yaml --database-url "$DB"
uv run longeval-cli database-status --database-url "$DB"

The tutorial continues from here through click-model training, qrels, documents, and the packaged snapshot, with the expected output of every step.

Documentation

The agent skills double as how-to guides for the four tasks you will do most:

• longeval-log-parser: write a parser for a new log format
• longeval-classifier: write or configure classifiers
• longeval-snapshot-filter: write a snapshot's filter YAML
• longeval-build-snapshot: build and release a snapshot end to end

Roadmap

• Query intent classification
• Query recency classification
• Topic generation
• Sampling instead of pooling
• Generalize filters.yml into a snapshot config file to reduce options needed

License

MIT, see LICENSE.