unimi-crop-sensing 1.2

About The Project

unimi_crop_sensing was created with the aim of offering a set of simple and intuitive operations to interact with the ZED camera. It consists of a toolkit for processing and segmenting images and point clouds acquired via the ZED stereo camera. The project is designed for precision agriculture applications, allowing you to identify plants in 2D and 3D, generate bounding boxes and communicate with a cobot through WebSocket in a ROS environment.

Main features

• Green segmentation with Excess Green Index
• Plant clustering via KMeans
• 2D and 3D bounding box calculation on point cloud
• Save .ply, images, normal map
• WebSocket ROS (rosbridge) integration for communication on separate systems

Built With

• Stereolabs ZED SDK
• OpenCV
• NumPy
• scikit-image
• scikit-learn
• websocket-client

Getting Started

Prerequisites

Make sure you have:

• Python 3.9
• ZED SDK properly installed and working and a connected ZED camera
• ROS + rosbridge running if you use WebSocket
• The libraries listed in requirements.txt

Installation

You can use unimi_crop_sensing as a Python package installable via PyPI. Install everything with:

pip install unimi_crop_sensing

⚠️ Pyzed 5.0 requires numpy 2.x, this conflicts with other project features, so if you encounter errors related to numpy, make sure you install a compatible version:

pip install "numpy<2"

Usage

This is an example of a script that uses every function to obtain spatial coordinates and point clouds of each plant within its range

# This function is used to test the functionalities of the crop sensing module
def main():
    
    # Get the current pose of the cobot
    pose = cobot_manager.get_cobot_pose(linux_ip)

    # Initialize the ZED camera
    zed = zed_manager.zed_init(pose)
    
    # Capture the environment with the ZED camera
    image, depth_map, normal_map, point_cloud = zed_manager.get_zed_image(zed, save=True)

    # Filter the plants from the background
    mask = find_plant.filter_plants(image, save_mask=True)
    
    # Divide the plants into clusters
    masks, bounding_boxes = find_plant.segment_plants(mask, plants_number)
    find_plant.save_clustered_image(image, bounding_boxes)

    # Extract the 3D points from the clusters
    for m in masks:
        bbxpts = find_plant.get_3d_bbox(m, point_cloud)
        
    # Communicate the bounding boxes to the cobot (only if the cobot is operated in another machine)
    cobot_manager.send_cobot_map(linux_ip, bbxpts)

    # Create point cloud (this will create a .ply file by taking a video of the environment)
    zed.close()
    create_plc.record_and_save(plant_name='piantina1', frames=300)

Note: The pipeline.py file contains a ready-to-run example with all the necessary components to extract bounding boxes and send them to the Dobot cobot.

Contact

francescobassam.morgigno@studenti.unimi.it

Acknowledgments

• Stereolabs