The TIRA Client
This is a python client for TIRA.io. Please find the documentation online.
Setup REST Client to Access Non-Public Endpoints
To get your API key, please navigate to the task for which you want to submit, click on "Submit" -> "Code Submission" or "Docker Submission" and the UI will show you your access token. With this, you can login:
tira-cli login --token YOUR-TOKEN-HERE
Admin-Only: Setup REST Client to Access Non-Public Endpoints
For admin-access to non-public endpoints, you will need an authentication via an API key to ensure that you have the correct access credentials. Please generate your admin API key online at tira.io/admin/api/keys and login your tira client:
tira-cli login --token YOUR-TOKEN-HERE
Admin-Only: Batch-Execution of jobs
To execute all approaches that have a positive review on one dataset on all datasets of a task, please run:
tira-cli admin batch-execution --task TASK
Admin-Only: Dataset submission
When preparing a dataset for tira-cli dataset-submission, the dataset card in README.md must define
tira_configs.default_upload_name. This is the default file name participants should upload for the dataset and the
dataset submission fails if it is missing.
tira_configs:
default_upload_name: predictions.jsonl
Download The results of some Submission
You can download runs of published and unblinded submissions via:
from tira.rest_api_client import Client
tira = Client()
output = tira.get_run_output('<task>/<team>/<approach>', '<dataset>')
As an example, you can download all baseline BM25 runs submitted to TIREx via:
from tira.rest_api_client import Client
from tira.tirex import TIREX_DATASETS
tira = Client()
for dataset in TIREX_DATASETS:
output = tira.get_run_output('ir-benchmarks/tira-ir-starter/BM25 Re-Rank (tira-ir-starter-pyterrier)', dataset)
Overview of public submissions
As an example, you can see all public software submissions submitted to TIREx via:
from tira.rest_api_client import Client
tira = Client()
submissions = tira.all_softwares("ir-benchmarks")
Export datasets
You can export datasets if you are the owner or if the dataset is public. Export a dataset via the cli:
tira-run --export-dataset '<task>/<tira-dataset>' --output-directory tira-dataset
Export a dataset via the python API:
from tira.rest_api_client import Client
tira = Client()
tira.download_dataset('<task>', '<tira-dataset>')
Running Jupyter Notebooks with TIRA
docker build -t tira/submission-base-image:1.0.0 -f Dockerfile .
Testing the model locally can be done using the following command:
tira-run \
--input-directory ${PWD}/input \
--output-directory ${PWD}/output \
--image tira/submission-base-image:1.0.0 \
--command 'tira-run-notebook --input $inputDataset --output $outputDir /workspace/template-notebook.ipynb'
Afterwards you can push the image to TIRA
docker push tira/submission-base-image:1.0.0
and set the command:
tira-run-notebook --input $inputDataset --output $outputDir /workspace/template-notebook.ipynb
Finally, if the actual processing in notebook is toggled via is_running_as_inference_server() (as seen in the
template notebook)
and your notebook defines a function named predict in the format
def predict(input_list: List) -> List:
you can start an inference server for your model with:
PORT=8001
docker run --rm -it --init \
-v "$PWD/logs:/workspace/logs" \
-p $PORT:$PORT \
tira/submission-base-image:1.0.0 \
tira-run-inference-server --notebook /workspace/template-notebook.ipynb --port $PORT
Exemplary request for a server running on localhost:8001 are
# POST (JSON list as payload)
curl -X POST -H "application/json" \
-d "[\"element 1\", \"element 2\", \"element 3\"]" \
localhost:8001
and
# GET (JSON object string(s) passed to the 'payload' parameter)
curl "localhost:8001?payload=\"element+1\"&payload=\"element+2\"&payload=\"element+3\""
3 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file tira-0.0.204.tar.gz.
File metadata
- Download URL: tira-0.0.204.tar.gz
- Upload date:
- Size: 1.3 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.14.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4d85daa51e92e548273122ff27339d03cdc4b216c99063099d71058a5ad2c5f8
|
|
| MD5 |
8401eefa08425b4ebfe0c88ffd6bb4aa
|
|
| BLAKE2b-256 |
23570991a5f50ad8c2f809e1224e4caf87fe8d13f91f4f8e8bb025e740170018
|
File details
Details for the file tira-0.0.204-py3-none-any.whl.
File metadata
- Download URL: tira-0.0.204-py3-none-any.whl
- Upload date:
- Size: 3.7 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.14.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8e037880e2ad457321ee11422a8de4c3b1149517dede47ffcbbbdf05a80872e7
|
|
| MD5 |
0055448fb1c16f379aa7774d0aa869bd
|
|
| BLAKE2b-256 |
c15337c451a613804082feb94ce76d86802bd2b8fc140b0090a638dad873320a
|
