FrontVeg V2: Foreground-Aware Zero-Shot Plant Trait Segmentation in Trellised Crops

Abdoul Djalil Ousseini Hamza (ORCID), Herearii Metuarea (ORCID), Corentin Lothodé (ORCID), Morgane Roth (ORCID), Eric Duchêne (ORCID), Lionel Ley, David Alletru (ORCID), and David Rousseau^* (ORCID)

* Project Supervisor

FrontVeg V2

A napari plugin for automated plant organ segmentation and leaf area estimation in trellised crops using side view monocular RGB images.

Here are some examples of organ segmentation on different trellised crops:

Prerequisites

1. Create a virtual environment

   conda create -n <env_name> python=3.10 -y
   conda activate <env_name>
   pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
   

2. Request SAM3 Weights Access: Request access to the SAM3 checkpoint on Hugging Face. Once approved, download sam3.pt.

3. Download Depth-Anything V2 Weights: Download the Large model checkpoint (depth_anything_v2_vitl.pth) from the official Depth-Anything V2 repository.

4. Place your downloaded checkpoints inside the project folder:

   • Place sam3.pt into: checkpoints/sam3_ckpts/sam3.pt
   • Place depth checkpoints into: checkpoints/depthanything_ckpts/

Installation

Option 1: Recommended (Git Clone + Editable Install)

This option automatically sets up the relative paths for external/ submodules and checkpoints/.

# 1. Clone the repository 
git clone https://github.com/djaliloh/FrontVegV2.git
cd FrontVegV2

# 2. Clone and install external models in the external/ folder:
cd external

# Depth-Anything V2
git clone https://github.com/DepthAnything/Depth-Anything-V2.git

# SAM3
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .
cd ../..

# 3. Install in editable mode 
pip install -e .

Option 2: Direct PyPI Install (Coming Soon)

ℹ️ Note: Standalone installation via pip install will be released soon. In the meantime, please follow Option 1 to use the plugin.

Usage in Napari

1. Launch Napari: napari
2. Open an RGB image of a row crop.
3. Select Plugins > FrontVeg V2 Studio.
4. Set your prompt (e.g., "leaf") and adjust parameters (Tile Overlap, Sigma, etc.).
5. Click Run Complete Pipeline.

Acknowledgements

This work was supported by the French Programme et Équipements Prioritaires de Recherche (PEPR) AgroEcoNum (pepr-agroeconum.fr), the France 2030 program through the Agence Nationale de la Recherche (ANR), and the European Union’s Horizon Europe research and innovation programme (Grant No. 101094587, PHENET).

This research used computing resources from the GLiCID Computing Facility (Ligerien Group for Intensive Distributed Computing, doi:10.60487/glicid, Pays de la Loire, France), as well as HPC and storage resources provided by GENCI at IDRIS on the Jean Zay supercomputer’s H100 partition (Grant 2025-AD010115553R1).

The authors would like to thank Sirine Gharbi, Oumaima Karia, Justin Langlois, Paul, and Thomas for their valuable assistance in annotating the image dataset used in this study.

Contact

Imhorphen team, bioimaging research group,
IRHS, UMR 1345, INRAE,
42 Rue Georges Morel, 49070 Beaucouzé, France

• David Rousseau - Professor, david.rousseau@univ-angers.fr
• Corentin Lothode - Researcher Engineer, corentin.lothode@inrae.fr
• Herearii Metuarea - PhD student, herearii.metuarea@univ-angers.fr
• Abdoul Djalil Ousseini Hamza - Engineer, abdoul-djalil.ousseini-hamza@inrae.fr

Troubleshooting

• Missing Module 'triton': On Windows, install the Windows-compatible Triton build: pip install triton-windows.
• RuntimeError: mat1 and mat2 must have the same dtype: If running on GPUs older than NVIDIA Ampere (e.g., GTX 10xx, RTX 20xx), ensure autocast is set to float16 or float32 instead of bfloat16.
• Missing Checkpoints Error: Verify that sam3.pt exists at the expected path or set FRONTVEG_SAM3_CKPT manually.