tirex-tracker 0.2.20

Automatic resource and metadata tracking for information retrieval experiments.

TIREx Tracker

The TIREx tracker is a command line tool and API to automatically track resource usage, hardware specifications, and other metadata when running information retrieval experiments. It can be used easily in Python applications. For more information, refer to the project's readme.

Python API

First, install the TIREx tracker Python package from PyPI:

pip install tirex-tracker

Now, you can track the hardware metrics and metadata of your Python code by using the context manager:

from tirex_tracker import tracking

with tracking() as results:
    # Do something...

print(results)

Alternatively, you can track the hardware metrics and metadata of a Python function by using the function decorator:

from tirex_tracker import tracked

@tracked
def do_something():
    # Do something...

do_something()

print(do_something.results)

If you cannot use either the context manager or the function decorator from above, you can manually start and stop the tracking:

from tirex_tracker import start_tracking, stop_tracking

handle = start_tracking()
try:
    # Do something...
finally:
    results = stop_tracking(handle)

print(results)

Citation

If you find our work useful and reference or use it in a paper, please cite us.

@inproceedings{tirextracker2025,
    author = {Hagen, Tim and Fr{\"o}be, Maik and Merker, Jan Heinrich and Scells, Harrisen and Hagen, Matthias and Potthast, Martin},
    booktitle = {48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2025)},
    month = jul,
    publisher = {ACM},
    title = {{TIREx Tracker: The Information Retrieval Experiment Tracker}},
    year = {2025}
}

You can also use the CITATION.cff to generate a citation in other formats.

License

The TIREx tracker code is licensed under the MIT License. If you use the TIREx tracker in your experiments, we would appreciate you citing our paper.

Abstract

The reproducibility and transparency of retrieval experiments heavily depends on properly provided information on the experimental setup and conditions. But as manually curating such experiment metadata can be tedious, error-prone, and inconsistent, metadata should be systematically collected in an automatic way—similar to the collection of Python and git-specific settings in the ir_metadata reference implementation. To enable a platform-independent automatic metadata collection following the ir_metadata specification, we introduce the TIREx tracker: a tool realized via a lightweight C binary, pre-compiled with all dependencies for all major platforms to track hardware configurations, usage of power/CPUs/RAM/GPUs, and experiment/system versions. The TIREx tracker seamlessly integrates into Python, Java, or C/C++ workflows and can be easily incorporated in run submissions of shared tasks, which we showcase for the TIRA/TIREx platform.