Simple access to the TIRA API.
Project description
The TIRA Client
This is a python client for TIRA.io.
Setup REST Client to Access Non-Public Endpoints
To access non-public endpoints, you will need an authentication via an API key to ensure that you have the correct access credentials.
Please generate your API key online at tira.io/admin/api/keys and create a credentials file at ~/.tira/.tira-settings.json
with the following content:
{"api_key": "<YOUR-API-KEY>"}
Download The results of some Submission
You can download runs of published and unblinded submissions via:
from tira.rest_api_client import Client
tira = Client()
output = tira.get_run_output('<task>/<team>/<approach>', '<dataset>')
As an example, you can download all baseline BM25 runs submitted to TIREx via:
from tira.rest_api_client import Client
tira = Client()
datasets = ['antique-test-20230107-training', 'argsme-touche-2020-task-1-20230209-training', 'argsme-touche-2021-task-1-20230209-training', 'clueweb09-en-trec-web-2009-20230107-training', 'clueweb09-en-trec-web-2010-20230107-training', 'clueweb09-en-trec-web-2011-20230107-training', 'clueweb09-en-trec-web-2012-20230107-training', 'clueweb12-touche-2020-task-2-20230209-training', 'clueweb12-touche-2021-task-2-20230209-training', 'clueweb12-trec-web-2013-20230107-training', 'clueweb12-trec-web-2014-20230107-training', 'cord19-fulltext-trec-covid-20230107-training', 'cranfield-20230107-training', 'disks45-nocr-trec-robust-2004-20230209-training', 'disks45-nocr-trec7-20230209-training', 'disks45-nocr-trec8-20230209-training', 'gov-trec-web-2002-20230209-training', 'gov-trec-web-2003-20230209-training', 'gov-trec-web-2004-20230209-training', 'gov2-trec-tb-2004-20230209-training', 'gov2-trec-tb-2005-20230209-training', 'gov2-trec-tb-2006-20230209-training', 'medline-2004-trec-genomics-2004-20230107-training', 'medline-2004-trec-genomics-2005-20230107-training', 'medline-2017-trec-pm-2017-20230211-training', 'medline-2017-trec-pm-2018-20230211-training', 'msmarco-passage-trec-dl-2019-judged-20230107-training', 'msmarco-passage-trec-dl-2020-judged-20230107-training', 'nfcorpus-test-20230107-training', 'vaswani-20230107-training', 'wapo-v2-trec-core-2018-20230107-training']
for dataset in datasets:
output = tira.get_run_output('ir-benchmarks/tira-ir-starter/BM25 Re-Rank (tira-ir-starter-pyterrier)', dataset)
Export datasets
You can export datasets if you are the owner or if the dataset is public. Export a dataset via the cli:
tira-run --export-dataset '<task>/<tira-dataset>' --output-directory tira-dataset
Export a dataset via the python API:
from tira.rest_api_client import Client
tira = Client()
tira.download_dataset('<task>', '<tira-dataset>')
Running Jupyter Notebooks with TIRA
docker build -t tira/submission-base-image:1.0.0 -f Dockerfile .
Testing the model locally can be done using the following command:
tira-run \
--input-directory ${PWD}/input \
--output-directory ${PWD}/output \
--image tira/submission-base-image:1.0.0 \
--command 'tira-run-notebook --input $inputDataset --output $outputDir /workspace/template-notebook.ipynb'
Afterwards you can push the image to TIRA
docker push tira/submission-base-image:1.0.0
and set the command:
tira-run-notebook --input $inputDataset --output $outputDir /workspace/template-notebook.ipynb
Finally, if the actual processing in notebook is toggled via is_running_as_inference_server()
(as seen in the
template notebook)
and your notebook defines a function named predict
in the format
def predict(input_list: List) -> List:
you can start an inference server for your model with:
PORT=8001
docker run --rm -it --init \
-v "$PWD/logs:/workspace/logs" \
-p $PORT:$PORT \
tira/submission-base-image:1.0.0 \
tira-run-inference-server --notebook /workspace/template-notebook.ipynb --port $PORT
Exemplary request for a server running on localhost:8001
are
# POST (JSON list as payload)
curl -X POST -H "application/json" \
-d "[\"element 1\", \"element 2\", \"element 3\"]" \
localhost:8001
and
# GET (JSON object string(s) passed to the 'payload' parameter)
curl "localhost:8001?payload=\"element+1\"&payload=\"element+2\"&payload=\"element+3\""
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.