orena-focus 0.3.0

Utilities for the ORena SAVE FOCUS challenge: Foreign Object Contextual Understanding for Safe Surgical AI

orena-focus

HeiCo-FOCUS dataset

LapChole-FOCUS dataset

Python utilities for the FOCUS datasets and challenge — Foreign Object Contextual Understanding for Surgery.

The library provides dataset loaders, preprocessing pipelines, answer-format handling, and an evaluation framework for working with the FOCUS surgical VQA datasets. It can be used independently for research on foreign-object understanding in minimally invasive surgery, and also serves as the official toolkit for the ORena SAVE FOCUS challenge at MICCAI 2026.

Challenge open for registration. Submit your results and compete on the leaderboard at orena-focus-challenge.org.

Retained foreign objects are a life-threatening and preventable surgical complication. FOCUS benchmarks vision-language models on clinically relevant VQA tasks around detecting, counting, and reasoning about foreign objects in endoscopic video.

Tracks

FOCUS offers three participation tracks, each requiring a different type of visual context:

Track	Track enum	Visual input	Max latency	Description
FRAME	Track.FRAME	Single frame	5 s	Answer questions from one extracted video frame. The simplest entry point — no temporal modelling required.
SEGMENT	Track.SEGMENT	<= 5min clip	15 s	Answer questions from a multi-second video segment surrounding the relevant event. Requires understanding of motion and temporal context.
PROCEDURE	Track.PROCEDURE	Up to full video	30 s	Answer questions that may require reasoning over an entire surgical procedure, including events that happened well before or after the queried moment.

Participants may enter any subset of tracks. Each track is evaluated independently with the same hierarchical capability taxonomy.

Installation

pip install orena-focus

Quick start

from focus import FocusDataset, DatasetSplit, Track

ds = FocusDataset("heico", DatasetSplit.TEST, Track.SEGMENT)

request, reference = ds[0]
print(request.question)        # "How many sponges are visible?"
print(reference.answer)        # "2"
print(reference.format.type)   # "number"

Data preparation

Download, preprocess, and split the dataset in one script — see examples/data_preparation.py for the full walkthrough.

from focus import download
from focus.preprocessing import VideoTimestampOverlayPreprocessor, FrameExtractorPreprocessor

download("heico")

VideoTimestampOverlayPreprocessor().process(dataset="heico")
FrameExtractorPreprocessor(stride=1).process(dataset="heico")

QA annotations are fetched automatically from HuggingFace when you construct a FocusDataset.

Inference & evaluation

See examples/inference.py for an end-to-end example with Qwen3-VL.

from focus import Evaluator, Response

responses = [Response(qID=req.qID, content=my_model(req)) for req, _ in ds]

results_df, summary_df = Evaluator().run(
    requests=ds.requests,
    references=ds.references,
    responses=responses,
)
print(summary_df)

Pre-evaluation score

The pre-evaluation phase uses a single headline number: the unweighted mean over ten buckets — the five capability groups, each scored independently for in-distribution and out-of-distribution questions (flat per-question accuracy within each bucket). Missing and timed-out answers count as incorrect. It is reported as the level="pre_evaluation", name="SCORE" row of summary_df, and can also be computed directly:

score, buckets_df = Evaluator().pre_evaluation_score(results_df)

Capability taxonomy

Five capability groups, each composed of leaf capabilities assigned to questions.

Answer formats

Format	Accepts	Returns
Binary	"yes" / "no"	bool
Number	Non-negative integer strings	int
Percentage	Numeric percentage strings	float
FOClass	Registered FO class names	str
OpenEnded	Free text (≤ 300 chars)	str
Matching	Regex-validated text	str
MultipleChoice	One of predefined options	str
Time	hh:mm:ss timestamps	timedelta

Datasets

FOCUS builds upon two datasets: HeiCo-FOCUS (30 videos) and LapChole-FOCUS (170 videos).

HeiCo-FOCUS

The QA annotations are publicly available on HuggingFace: orena-dkfz/heico-focus-vqa.

HeiCo-FOCUS is built on the HeiCo dataset. If you use this data, please cite the original publication:

Maier-Hein, L., et al. (2021). Heidelberg colorectal data set for surgical data science in the sensor operating room. https://doi.org/10.1038/s41597-021-00882-2

The HeiCo data is released under CC BY-NC-SA 4.0 — non-commercial use only, with attribution and share-alike conditions.

LapChole-FOCUS

The LapChole-FOCUS videos comprises unpublished laparoscopic cholecystectomy recordings. It is currently only available within the scope of the FOCUS-Challenge and will be publicly released after the Challenge has ended.

To access it, send a request on HuggingFace: orena-dkfz/lapchole-focus-vqa. To access and download the data you need to register yourself with HuggingFace.

Please find the data usage agreement at the dataset card or in the LICENSE file on HuggingFace.

License

The code is licensed under the permissive MIT license. The underlying data is licensed independently, see Dataset for the data licenses details.