group-cc-hook 0.1.1

A hook for torch.distributed ProcessGroup primitives with timeout detection

Group Collective Communication Hook

A monitoring solution for PyTorch distributed training that hooks into all ProcessGroup collective communication primitives and performs timeout detection in pure Python using multiprocessing to avoid GIL limitations.

Overview

This package provides a robust monitoring mechanism for distributed PyTorch training by:

• Hooking all ProcessGroup collective communication primitives
• Implementing timeout detection using a separate process to avoid Python GIL constraints
• Sending SIGUSR1 signal when timeout is detected
• No C++ compilation required - pure Python implementation with multiprocessing

Quick Start

Build from Source

# Install the package (no build step needed - pure Python)
pip install .

# Run tests
torchrun --nproc_per_node=4 test_all_reduce.py 

# Test basic hook functionality
torchrun --nproc_per_node=4 test_hook_simple.py

Installation

Install the package using pip:

pip install .

Usage

Add the following code to your training script:

from run_daemon import run_daemon, stop_daemon

# Start the monitoring daemon
run_daemon()
 
############################# Your Training Code START #########################

############################# Your Training Code END ###########################

# Stop the monitoring daemon
stop_daemon()

Configuration

You can configure the timeout threshold and other parameters by modifying the run_daemon function call.

from config import HookConfig
from run_daemon import run_daemon

# Default: timeout detection with SIGUSR1 signal
run_daemon()

# Custom configuration
config = HookConfig(
    timeout_seconds=600,
    signal_number=12,  # SIGUSR1 (default), change to another signal if needed
    send_signal_on_timeout=True, # Default: True, set to False if you want to handle timeouts in your code
    check_interval_ms=10 # Enabled by default
    
)
run_daemon(config)

Or just use the environment variables directly:

export PG_HOOK_TIMEOUT=600
export PG_HOOK_CHECK_INTERVAL=10
export PG_HOOK_SIGNAL=12
export PG_HOOK_SEND_SIGNAL=1

License

This project is licensed under the MIT License - see the LICENSE file for details.

Project Structure

├── work_monitor.py                                # Pure Python work monitoring and timeout detection
├── config.py                                      # Configuration management   
├── patch_all_collective_primi.py                  # Hooks for ProcessGroup primitives
├── run_daemon.py                                  # Daemon launcher script
├── setup.py                                       # Package setup configuration
├── test_all_reduce.py                             # AllReduce test script
├── test_hook_simple.py                            # Basic hook functionality test
├── README.md                                      # This file
└── doc/                                           # Documentation directory
    ├── DESIGN_ANALYSIS.md                         # Design analysis report
    ├── COMPATIBILITY.md                           # Version compatibility guide
    └── DEBUG_USAGE.md                             # Debug logging instructions

Documentation

• Design Analysis Report - Comprehensive analysis of design integrity, functionality, robustness, and version compatibility
• Version Compatibility Guide - Compatibility information for PyTorch, Python, CUDA, and other dependencies
• Debug Usage Guide - Detailed instructions for debug logging

Requirements

• Python: >= 3.7, < 3.13
• PyTorch: >= 1.8.0, < 3.0.0
• CUDA: Required for GPU support (if using NCCL backend)
• Operating System: Linux (or any OS supporting PyTorch distributed)

For detailed compatibility information, please refer to COMPATIBILITY.md.

Development

Running Tests

# Install development dependencies
pip install -r requirements-dev.txt

# Run unit tests
pytest tests/unit/

# Code quality checks
black --check .
flake8 .

Contributing

Contributions are welcome! Please feel free to submit Issues and Pull Requests.

Before submitting your code, please ensure:

• All tests pass
• Code follows the project style guidelines (Black, Flake8)
• Documentation is updated if necessary

License

See LICENSE file for details.

Changelog

See CHANGELOG.md for version history and changes.