A Python package to overlay YOLO detections on displays using a custom DLL.

These details have not been verified by PyPI

Project links

Homepage

Project description

YOLO Overlay

YOLO Overlay is a robust Python package that seamlessly integrates YOLOv8 object detection with a custom C DLL to overlay real-time detection boxes directly onto a Windows display. Designed for applications requiring on-screen visualization of object detections in real-time, YOLO Overlay is ideal for live monitoring systems, gaming enhancements, augmented reality applications, and more.

Disclaimer: YOLO Overlay is not affiliated with or endorsed by Ultralytics.

Introduction
Features
Architecture Overview
Installation
- Prerequisites
- Step-by-Step Installation
Quick Start Guide
Detailed Usage
Configuration
- Parameters
- Advanced Configuration
Customization
- Using a Custom DLL
- Customizing Detection Colors and Labels
Underlying Architecture
Development Guide
Troubleshooting
- Common Issues and Solutions
- Debugging Techniques
Frequently Asked Questions (FAQ)
Best Practices
Performance Optimization
Security Considerations
License
Acknowledgements
Contact
Appendix
- Source Code Overview
- Glossary

Introduction

YOLO Overlay bridges the gap between powerful object detection capabilities provided by YOLOv8 and real-time visualization on Windows displays. By leveraging a custom C DLL, the package efficiently renders detection boxes overlaid on the user's screen with minimal latency, ensuring a smooth and responsive user experience.

Key Objectives

Real-Time Visualization: Display object detections instantaneously as they are processed.
High Performance: Utilize efficient screen capturing and rendering techniques to maintain system responsiveness.
Flexibility: Support multiple monitors, custom configurations, and optional DLL customization.
Ease of Use: Provide a straightforward API for quick integration into various projects.

Features

YOLO Overlay boasts a range of features designed to provide flexibility, performance, and ease of use:

Real-Time Object Detection:
- Integrates YOLOv8 for state-of-the-art object detection.
- Displays detection boxes in real-time directly on the user's display.
Adjustable Parameters:
- Configure confidence thresholds to filter detections based on certainty.
- Set maximum bounding boxes to control overlay density.
Multi-Monitor Support:
- Automatically detects and supports multiple monitors with varying resolutions.
- Allows selection of target monitor for overlay.
Lightweight and Efficient:
- Utilizes ctypes for seamless interaction between Python and the C DLL.
- Minimal resource consumption ensures system performance remains unaffected.
Customizable Overlay:
- Option to use a custom DLL for tailored overlay behaviors and functionalities.
Thread-Safe Operations:
- Ensures smooth multi-threaded operations between detection processing and overlay rendering.
Comprehensive Logging:
- Debugging logs assist in monitoring the overlay's operations and troubleshooting issues.

Architecture Overview

Understanding the architecture of YOLO Overlay provides insights into how different components interact to deliver real-time object detection overlays.

1. Python Components

YOLO Overlay Package (yolo_overlay):
- overlay.py: Manages the lifecycle of the overlay, including initialization, starting, and stopping the overlay.
- detection.py: Handles screen capturing, performs YOLOv8 detections, and communicates detection data to the C DLL.
- utils.py: Contains utility functions for color conversion, label encoding, and bounding box sanitization.

2. C DLL Components

overlay-yolo.dll:
- Overlay Window Management: Creates and manages a transparent, topmost window for rendering detection boxes.
- Rendering Engine: Draws bounding boxes and labels based on detection data received from Python.
- Thread Management: Ensures thread-safe operations for concurrent access to detection data.
- Exported Functions: Provides functions like StartOverlay, StopOverlay, UpdateDetections, etc., for interaction with Python.

3. Inter-Process Communication

ctypes: Facilitates communication between Python and the C DLL, allowing Python to invoke DLL functions and pass detection data structures.

Data Flow

Screen Capture: Python captures the screen using mss.
Object Detection: YOLOv8 processes the captured image to identify objects.
Data Preparation: Detection results are formatted into DetectionBox structures.
Data Transmission: Python sends detection data to the C DLL via ctypes.
Overlay Rendering: The C DLL receives the data and renders detection boxes on the overlay window in real-time.

Installation

Setting up YOLO Overlay involves several steps, from installing the package to ensuring all dependencies are correctly configured.

Prerequisites

Before installing YOLO Overlay, ensure the following prerequisites are met:

Operating System: Windows (due to reliance on Windows APIs for overlay rendering).
Python Version: Python 3.8 or later.
YOLOv8 Model: A trained YOLOv8 .pt model file.
System Requirements:
- Adequate CPU and GPU resources for real-time object detection.
- Sufficient memory to handle detection processes and overlay rendering.

Step-by-Step Installation

1. Install the YOLO Overlay Package

Use pip to install the package from PyPI:

pip install yolo-overlay

Alternatively, install directly from the GitHub repository for the latest features:

pip install git+https://github.com/wuhplaptop/yolo_overlay.git

2. Verify DLL Availability

Default DLL (overlay-yolo.dll):
- Included in the package under the resources folder.
- Automatically loaded during package initialization.
Using a Custom DLL (Optional):
- Ensure your custom DLL exports the required functions:
  - StartOverlay()
  - StopOverlay()
  - UpdateDetections(DetectionBox*, int)
  - SetTargetMonitorRect(int, int, int, int)
  - SetMaxDetections(int)
- Place the custom DLL in a directory included in your system's PATH environment variable.
- Alternatively, specify the custom DLL's path during YOLO Overlay initialization.

3. Install Required Python Dependencies

YOLO Overlay automatically installs necessary dependencies during the package installation. These include:

ultralytics: Provides YOLOv8 functionalities.
mss: Facilitates high-performance screen capturing.
Pillow: Handles image processing.
screeninfo: Detects and manages multiple monitors.

Ensure that pip is up-to-date to avoid installation issues:

pip install --upgrade pip

4. Additional Dependencies (If Needed)

C++ Redistributable:
- The C DLL may require the Microsoft Visual C++ Redistributable. If you encounter DLL loading errors, download and install the latest C++ Redistributable.
CUDA (For GPU Acceleration):
- If you wish to leverage GPU acceleration for YOLOv8, ensure that CUDA is installed and properly configured on your system.

Quick Start Guide

Kickstart your journey with YOLO Overlay by following this quick start guide. This section provides a minimal example to get you up and running swiftly.

1. Prepare Your YOLOv8 Model

Ensure you have a trained YOLOv8 .pt model. If you don't have one, you can train a model using the Ultralytics YOLOv8 documentation.

2. Sample Script

Create a Python script (e.g., run_overlay.py) with the following content:

from yolo_overlay import YOLOOverlay

# Specify the path to your YOLOv8 model
model_path = "path/to/yolo_model.pt"

# Initialize YOLO Overlay
overlay = YOLOOverlay(
    model_path=model_path,
    dll_path=None,           # Optional: Specify a custom DLL path if needed
    max_detections=100,
    conf_threshold=0.5,
    monitor_index=0          # Index of the monitor to overlay on (0 for primary)
)

try:
    print("YOLO Overlay is running. Press Ctrl+C to stop.")
    while True:
        pass  # Keep the script running
except KeyboardInterrupt:
    overlay.stop()
    print("YOLO Overlay has been stopped.")

3. Run the Script

Execute the script using Python:

python run_overlay.py

Upon execution:

The overlay window will appear on the specified monitor, displaying detection boxes in real-time.
Press Ctrl+C in the terminal to gracefully stop the overlay.

4. Expected Output

Terminal:

[INFO] DLL loaded and functions configured successfully.
[DEBUG] Initializing YOLO model with path: path/to/yolo_model.pt
[INFO] YOLO model successfully loaded from: path/to/yolo_model.pt
[INFO] Selected monitor 0: Monitor(x=0, y=0, width=1920, height=1080)
[INFO] Set target monitor rectangle to: Left=0, Top=0, Right=1920, Bottom=1080
[INFO] Set maximum detections to: 100
[INFO] Overlay started successfully.
[INFO] Detection thread started.
YOLO Overlay is running. Press Ctrl+C to stop.

Overlay Window:
- A transparent window displaying bounding boxes and labels over detected objects in real-time.

Detailed Usage

Dive deeper into YOLO Overlay's functionalities with detailed usage instructions. This section covers initialization, running, stopping the overlay, and explains all available parameters.

Initializing YOLO Overlay

To begin using YOLO Overlay, initialize the YOLOOverlay class with the desired configurations.

from yolo_overlay import YOLOOverlay

# Initialize YOLO Overlay with custom parameters
overlay = YOLOOverlay(
    model_path="path/to/yolo_model.pt",  # Required: Path to YOLOv8 model
    dll_path="path/to/custom_overlay.dll",  # Optional: Custom DLL path
    max_detections=50,                      # Optional: Maximum detection boxes
    conf_threshold=0.6,                     # Optional: Confidence threshold
    monitor_index=1                          # Optional: Target monitor index
)

Running the Overlay

Once initialized, the overlay starts automatically and begins rendering detection boxes in real-time. Ensure that your script remains active to keep the overlay running.

try:
    print("YOLO Overlay is running. Press Ctrl+C to stop.")
    while True:
        pass  # Keep the script running
except KeyboardInterrupt:
    overlay.stop()
    print("YOLO Overlay has been stopped.")

Stopping the Overlay

To gracefully stop the overlay and release resources, invoke the stop() method.

overlay.stop()
print("YOLO Overlay has been stopped.")

Parameters

`YOLOOverlay` Class Initialization Parameters

model_path (str, required):
- Description: Path to the YOLOv8 .pt model file.
- Example: "models/yolov8n.pt"
dll_path (str, optional):
- Description: Path to a custom DLL. If None, the default overlay-yolo.dll included in the package is used.
- Example: "custom_dlls/my_overlay.dll"
max_detections (int, optional, default=100):
- Description: Maximum number of simultaneous detection boxes to display.
- Usage: Limits the number of detections rendered to prevent clutter.
- Example: max_detections=50
conf_threshold (float, optional, default=0.5):
- Description: Minimum confidence score required for detections to be displayed.
- Usage: Filters out low-confidence detections to enhance overlay quality.
- Example: conf_threshold=0.7
monitor_index (int, optional, default=0):
- Description: Index of the monitor to target for the overlay. 0 corresponds to the primary monitor.
- Usage: Selects the desired monitor in multi-monitor setups.
- Example: monitor_index=1

Configuration

YOLO Overlay provides several configuration options to tailor its behavior to specific requirements. This section elaborates on available parameters and advanced configuration settings.

Parameters

All configuration parameters can be set during the initialization of the YOLOOverlay class. Below is a detailed explanation of each parameter:

model_path (str):
- Purpose: Specifies the path to the YOLOv8 model used for object detection.
- Considerations: Ensure the model is compatible with YOLOv8 and is properly trained for your use case.
dll_path (str, optional):
- Purpose: Allows the use of a custom C DLL for overlay rendering.
- Usage: If not provided, the default overlay-yolo.dll is used. Useful for extending functionalities or integrating with other systems.
max_detections (int, optional):
- Purpose: Limits the number of detection boxes displayed to prevent visual clutter and manage performance.
- Default Value: 100
- Recommendation: Adjust based on the typical number of objects detected in your application.
conf_threshold (float, optional):
- Purpose: Sets the minimum confidence level for detections to be visualized.
- Default Value: 0.5
- Recommendation: Increase for higher precision or decrease to include more detections.
monitor_index (int, optional):
- Purpose: Selects which monitor the overlay should appear on, especially in multi-monitor setups.
- Default Value: 0 (primary monitor)
- Recommendation: Enumerate available monitors to choose the desired target.

Advanced Configuration

Beyond the basic parameters, YOLO Overlay offers advanced configuration options to fine-tune its operations.

1. Adjusting Detection Timeout

Description: Controls how long a detection remains active before being considered outdated.
Implementation: Modify the detectionTimeoutMs variable in the C DLL.
Default Value: 2000 milliseconds (2 seconds)
Use Case: Prevents stale detections from lingering on the screen.

2. Customizing Overlay Appearance

Transparent Color: Define the color key used for the transparent background.
- Default Value: Bright magenta (RGB(255, 0, 255))
- Modification: Change the transparentColor variable in the C DLL to use a different color key.
Label Display: Toggle the display of labels alongside bounding boxes.
- Default Value: Enabled (showLabels = 1)
- Modification: Set showLabels to 0 in the C DLL to hide labels.

3. Optimizing Overlay Performance

Reduce Maximum Detections: Lower max_detections to decrease rendering load.
Use Lightweight Models: Opt for smaller YOLOv8 variants (e.g., YOLOv8n) to enhance processing speed.
Adjust Frame Rate: Modify the detection loop's sleep time to control processing frequency.

Customization

YOLO Overlay is designed with flexibility in mind, allowing users to customize various aspects to suit their specific needs. This section explores customization options, including using a custom DLL and tailoring detection colors and labels.

Using a Custom DLL

If the default overlay behaviors provided by overlay-yolo.dll do not meet your requirements, you can integrate a custom C DLL to extend or modify functionalities.

Steps to Use a Custom DLL:

Develop Your Custom DLL:
- Ensure your DLL exports the necessary functions:
  - StartOverlay()
  - StopOverlay()
  - UpdateDetections(DetectionBox*, int)
  - SetTargetMonitorRect(int, int, int, int)
  - SetMaxDetections(int)
- Maintain compatibility with the data structures expected by the Python package.
Place the DLL:
- Save your custom DLL in a directory accessible by your system.
- Optionally, add the directory to your system's PATH environment variable.

Specify DLL Path During Initialization:

When initializing YOLOOverlay, provide the path to your custom DLL.

overlay = YOLOOverlay(
    model_path="path/to/yolo_model.pt",
    dll_path="path/to/custom_overlay.dll"
)

Ensure Compatibility:
- Verify that your custom DLL adheres to the required function signatures and data structures.
- Test the overlay to ensure detections are rendered correctly.

Customizing Detection Colors and Labels

Enhance the visual appeal and clarity of detection boxes by customizing colors and labels.

1. Customizing Detection Colors

Purpose: Differentiate object types or importance levels using distinct colors.

Implementation:

Modify the color attribute in the DetectionBox structure before sending it to the DLL.
Use the rgb_to_colorref utility function to convert RGB values to COLORREF format.

from yolo_overlay.utils import rgb_to_colorref

# Define custom colors
COLORS = {
    "person": rgb_to_colorref(0, 255, 0),     # Green
    "vehicle": rgb_to_colorref(0, 0, 255),    # Blue
    "animal": rgb_to_colorref(255, 0, 0),     # Red
    # Add more as needed
}

# Assign colors based on label
label = "person"
color = COLORS.get(label, rgb_to_colorref(255, 255, 255))  # Default to white

2. Customizing Detection Labels

Purpose: Display meaningful or localized labels for detected objects.

Implementation:

Modify the label attribute in the DetectionBox structure.
Use the encode_label utility to ensure labels are properly formatted.

from yolo_overlay.utils import encode_label

# Define custom labels
label = "Person"
encoded_label = encode_label(label)

# Assign to DetectionBox
detection_box.label = encoded_label

Underlying Architecture

A thorough understanding of YOLO Overlay's architecture facilitates effective utilization and potential customization. This section delves into the interactions between Python components, the C DLL, and the communication mechanisms that enable real-time overlay rendering.

Python Components

overlay.py:
- Functionality: Orchestrates the overlay lifecycle, including initialization, starting, and stopping the overlay.
- Key Components:
  - YOLOOverlay Class: Central class managing model loading, DLL interactions, monitor selection, and threading.
  - Thread Management: Initiates and manages a separate thread for processing detections to ensure non-blocking operations.
detection.py:
- Functionality: Handles screen capturing, performs object detection using YOLOv8, and communicates detection data to the C DLL.
- Key Components:
  - DetectionBox Structure: Defines the data structure for individual detections, mirroring the C DetectionBox.
  - process_detections Function: Core function that captures the screen, processes detections, and updates the overlay.
utils.py:
- Functionality: Provides utility functions for color conversion, label encoding, and bounding box sanitization.
- Key Functions:
  - rgb_to_colorref: Converts RGB values to COLORREF format for the C DLL.
  - encode_label: Encodes labels to a fixed byte length suitable for the C structure.
  - sanitize_bounding_box: Ensures bounding boxes are within monitor bounds and have valid dimensions.

C DLL Components

overlay-yolo.dll:
- Functionality: Manages the creation of the overlay window and renders detection boxes based on data received from Python.
- Key Components:
  - Window Management: Creates a transparent, topmost window using Windows APIs.
  - Rendering Engine: Draws bounding boxes and labels using GDI functions.
  - Thread Management: Runs the overlay in a separate thread to handle window messages and rendering.
  - Exported Functions: Exposes functions (StartOverlay, StopOverlay, UpdateDetections, etc.) for interaction with Python.
Key Data Structures:
- DetectionBox:
  - Fields: id, x, y, width, height, color, label, lastSeen, paused.
  - Purpose: Represents individual object detections with tracking and rendering information.

Inter-Process Communication

ctypes:
- Role: Bridges Python and the C DLL, allowing Python to call DLL functions and pass complex data structures.
- Usage:
  - Loading the DLL: Using ctypes.WinDLL to load overlay-yolo.dll.
  - Defining Function Signatures: Specifying argument and return types for DLL functions to ensure correct data transmission.
  - Passing Structures: Sending arrays of DetectionBox structures to the DLL for rendering.

Data Flow

Screen Capture and Detection:
- The Python detection.py module captures the screen using mss.
- The captured image is processed by YOLOv8 to detect objects.
- Detection results are formatted into DetectionBox structures.
Data Transmission:
- Detection data is sent to the C DLL via the UpdateDetections function using ctypes.
- The DLL receives the data and updates its internal detection array.
Overlay Rendering:
- The DLL's overlay window retrieves the latest detection data.
- Bounding boxes and labels are drawn over the target monitor in real-time.
- The overlay window is transparent, allowing seamless integration with the user's display.
Thread Management:
- Python manages detection processing in a separate thread to maintain performance.
- The DLL runs its own message loop in another thread to handle window events and rendering.

Development Guide

For developers interested in contributing to YOLO Overlay or customizing its functionalities, this section provides a comprehensive guide on setting up the development environment, building the C DLL, and contributing to the project.

Setting Up the Development Environment

Clone the Repository:

git clone https://github.com/wuhplaptop/yolo_overlay.git
cd yolo_overlay

Create a Virtual Environment:

python -m venv venv
venv\Scripts\activate  # On Windows

Install Python Dependencies:
```
pip install -r requirements.txt
```
Install Development Dependencies:
```
pip install -r dev-requirements.txt
```

Building the C DLL

To modify or build the C DLL (overlay-yolo.dll), follow these steps:

Navigate to the C Source Directory:
```
cd src/c_dll
```
Ensure Necessary Tools are Installed:
- Compiler: Visual Studio with C/C++ development tools.
- SDKs: Windows SDK for access to necessary headers and libraries.
Build the DLL:
- Using Visual Studio:
  - Open the solution file (YOLOOverlay.sln).
  - Configure the build (Release/Debug) and platform (x64).
  - Build the solution to generate overlay-yolo.dll.
- Using Command Line (MSVC):
```
cl /LD overlay.c /Feoverlay-yolo.dll
```
  Ensure environment variables for MSVC are set correctly.
Place the DLL:
- Copy the built overlay-yolo.dll to the resources folder or a directory accessible by the Python package.

Contributing to the Project

Contributions enhance the project's robustness and feature set. Here's how you can contribute:

Fork the Repository:
- Navigate to the YOLO Overlay GitHub repository.
- Click on the "Fork" button to create a personal copy.

Clone Your Fork:

git clone https://github.com/wuhplaptop/yolo_overlay.git
cd yolo_overlay

Create a New Branch:

git checkout -b feature/YourFeatureName

Implement Your Feature or Fix:
- Make changes to the Python modules, C DLL, or documentation as needed.
- Ensure adherence to the project's coding standards and guidelines.

Commit Your Changes:

git add .
git commit -m "Add [Your Feature]: Brief description"

Push to Your Fork:

git push origin feature/YourFeatureName

Submit a Pull Request:
- Navigate to your fork on GitHub.
- Click on "Compare & pull request."
- Provide a detailed description of your changes and their purpose.
- Submit the pull request for review.
Address Feedback:
- Collaborate with maintainers to refine your contribution.
- Make necessary adjustments based on feedback.

Code Standards and Guidelines

Python Code:
- Follow PEP 8 style guidelines.
- Write clear, concise, and well-documented code.
- Include docstrings for all public modules, classes, and functions.
C Code:
- Maintain consistent indentation and formatting.
- Use descriptive variable and function names.
- Comment complex logic and important sections for clarity.
Documentation:
- Ensure all new features or changes are reflected in the documentation.
- Maintain clarity and comprehensiveness in explanations.

Testing

Unit Tests:
- Write unit tests for new functionalities.
- Ensure existing tests pass after changes.
Integration Tests:
- Verify that Python and C components interact seamlessly.
- Test multi-monitor setups and various configurations.
Performance Tests:
- Assess the impact of changes on detection speed and overlay rendering.
- Optimize as necessary to maintain high performance.

Troubleshooting

Encountering issues is common during the development and usage of complex systems like YOLO Overlay. This section provides solutions to common problems and debugging techniques to help you resolve issues effectively.