MedICS Extension: Retinal Layer Segmentation

AI-based retinal layer segmentation for OCT volumes, built as a MedICS extension. It automatically detects the anatomical layers of the retina from OCT data and provides interactive tools to review, correct, quantify, and export the results.

Overview

This extension turns an OCT volume into a fully segmented retina. It runs a deep-learning model (ONNX, executed with ONNX Runtime) on each B-scan to detect up to 14 retinal boundaries, then reconstructs the boundaries into smooth layer curves, detects intraretinal fluid, and builds a 3D representation of the layers across the whole volume.

Everything runs inside a dedicated "OCT Analyzer" window:

• Load data from the MedICS workspace, from files on disk, or by dragging files onto the window (OCT, OCTA and existing segmentation data).
• Run AI segmentation with one click, optionally accelerated by a GPU (CUDA / DirectML / CoreML) with automatic CPU fallback.
• Review and edit the detected boundaries with a curve editor, layer visibility toggles, and a manual corrector.
• Interpolate the segmentation between sparse B-scans for a dense volume.
• Visualize the result as B-scans with overlaid curves, en-face views, and an interactive 3D surface reconstruction (VTK).
• Export the layer curves, fluid mask, and data back to the MedICS workspace or to files for downstream analysis.

The segmentation is computed locally — OCT/OCTA images never leave the machine.

Supported layer boundaries

The model detects the following anatomical boundaries in each B-scan:

Note: * labeled boundaries are not implemented yet.

Each boundary can be toggled on/off individually for display and export, and any boundary may be edited manually after segmentation.

Features

• One-click AI segmentation of OCT/OCTA volumes (ONNX Runtime).
• Up to 14 automatically detected retinal layer boundaries.
• Intraretinal fluid detection with an optional fluid volume mask.
• Interactive curve editor for manual refinement of any boundary.
• Layer-by-layer visibility control and "Select All" convenience.
• Interpolation between sparse B-scans (configurable step) with restart.
• 3D surface / en-face visualization of the segmented volume (VTK).
• Preprocessing controls: flattening (None / Fitting / RPE-BM), axis permute, axis flip, ROI (manual or auto), and scan resolution update.
• B-frame navigation across the volume (frame index, total frames).
• Sparse data mode for memory-efficient processing of large volumes.
• GPU acceleration via CUDA, DirectML, or CoreML, with CPU fallback and a device selector.
• Drag-and-drop data loading, plus loading from the MedICS workspace or files.
• Save results to files or transfer curves/masks back to the MedICS workspace.
• Data resolution helper for aligning volumes acquired at different scales.

Installation

From PyPI

pip install medics-ext-retinal-layer-segmentation

Usage

1. Install MedICS and this extension.
2. Launch MedICS: medics.
3. Open the extension from the Extensions menu (a valid, non-free MedICS token is required; free or invalid tokens are blocked with a warning dialog).
4. Load an OCT volume — pick it from the workspace, click the file buttons on the File tab, or drag it onto the window. Optionally load matching OCTA or existing segmentation data as well.
5. Optionally adjust preprocessing on the side panel (flatten, permute, flip, ROI, resolution).
6. Click Run AI Segmentation in the side panel and choose the compute device.
7. Review the detected boundaries, toggle layers on/off, and use the curve editor or corrector to refine any boundary if needed.
8. Use Save to WS to transfer oct_data, octa_data, and seg_data back into the MedICS workspace, or Save to write results to files.

Troubleshooting

GPU / CUDA not used, or segmentation crashes on Conv / cuDNN

AI segmentation uses ONNX Runtime with an optional NVIDIA CUDA provider. The stack only works when these three pieces match each other and your GPU architecture / driver:

Official compatibility matrix: ONNX Runtime CUDA Execution Provider

Typical symptoms

• Log: CUDA unavailable (onnxruntime build has no CUDA provider); using CPU → CPU-only onnxruntime is installed. On Linux/Windows with an NVIDIA GPU, install onnxruntime-gpu (and uninstall plain onnxruntime so they do not conflict).
• Log / crash: CUDNN_FE failure / CUDNN_BACKEND_API_FAILED / CUDNN_STATUS_EXECUTION_FAILED on a Conv node → CUDA loaded, but the cuDNN kernels do not support this GPU (common on older cards) or the cuDNN minor is incompatible with ORT.

Older GPUs (e.g. Pascal: GTX 10-series)

Cards such as the GTX 1080 (compute capability 6.1) are too old for current cuDNN 9 + recent onnxruntime-gpu Conv paths. NVIDIA dropped pre-Turing architectures in newer cuDNN releases, so even a working driver and CUDA toolkit may still fail at inference.

On this extension, that case is handled by automatic CPU fallback after a CUDA warm-up, so MedICS should not crash — but segmentation will run on CPU.

If you still want GPU on an older card, you must pick an older stack that still targets that architecture, for example:

1. Read the ORT CUDA EP page for the ORT version → required CUDA / cuDNN.

2. Confirm your driver supports that CUDA major (nvidia-smi).

3. Install a matching wheel, e.g. try an older GPU build that predates cuDNN Frontend-heavy Conv (community reports often cite ~onnxruntime-gpu==1.18.x for Pascal). Example:

   pip uninstall -y onnxruntime onnxruntime-gpu
   pip install 'onnxruntime-gpu==1.18.0'
   

4. Ensure CUDA/cuDNN libraries for that ORT major are on the library path (LD_LIBRARY_PATH on Linux, or use wheels that bundle them).

5. Verify:

   python -c "import onnxruntime as ort; print(ort.__version__); print(ort.get_available_providers())"
   

   You want CUDAExecutionProvider listed. Then re-run AI segmentation.

Newer GPUs / drivers (quick guide)

• Driver shows CUDA 12.x → prefer onnxruntime-gpu < 1.27 (CUDA 12 builds). ORT ≥ 1.27 defaults to CUDA 13 and needs a newer driver.
• Driver shows CUDA 13.x → onnxruntime-gpu>=1.27 with matching nvidia-cudnn-cu13 (this project pins cuDNN 9.14 on Windows to avoid flaky Conv engines on some Ada GPUs).
• When in doubt: match versions from the ORT docs table, then test providers and a short inference before processing a full volume.

Force CPU

Set MEDICS_EXT_DISABLE_GPU=1 to skip GPU entirely, or choose CPU in the extension device selector.