octorun 0.1.0

A command-line tool for distributed parallel execution across multiple GPUs

🐙 OctoRun

Distributed Parallel Execution Made Simple

A powerful command-line tool for running Python scripts across multiple GPUs with intelligent task management and monitoring

---

📋 Overview

OctoRun is designed to help you run computationally intensive Python scripts across multiple GPUs efficiently. It automatically manages GPU allocation, chunks your workload, handles failures with retry mechanisms, and provides comprehensive monitoring and logging.

✨ Key Features

• 🔍 Automatic GPU Detection: Automatically detects and utilizes available GPUs
• 🧩 Intelligent Chunk Management: Divides work into chunks and distributes across GPUs
• 🔄 Failure Recovery: Automatic retry mechanism for failed chunks
• 📊 Comprehensive Logging: Detailed logging for monitoring and debugging
• ⚙️ Flexible Configuration: JSON-based configuration with CLI overrides
• 🎯 Kwargs Support: Pass custom arguments to your scripts via config or CLI
• 💾 Memory Monitoring: Monitor GPU memory usage and thresholds
• 🔒 Lock Management: Prevent duplicate processing of chunks

🚀 Installation

Via pip

pip install octorun

From source

git clone https://github.com/HarborYuan/OctoRun.git
cd OctoRun
pip install -e .

⚡ Quick Start

1️⃣ Create Configuration octorun save_config --script ./your_script.py	2️⃣ Run Your Script octorun run [--config config.json]
3️⃣ Monitor GPU Usage octorun list_gpus [--detailed]	4️⃣ View Logs tail -f logs/session_*.log

⚙️ Configuration

📄 Basic Configuration

The configuration file (config.json) contains the following options:

{
    "script_path": "./your_script.py",
    "gpus": "auto",
    "total_chunks": 128,
    "log_dir": "./logs",
    "chunk_lock_dir": "./logs/locks",
    "monitor_interval": 60,
    "restart_failed": false,
    "max_retries": 3,
    "memory_threshold": 90,
    "kwargs": {
        "batch_size": 32,
        "learning_rate": 0.001
    }
}

🔧 Configuration Options

Option	Description	Default
script_path	Path to your Python script	-
gpus	GPU configuration ("auto" or list of GPU IDs)	"auto"
total_chunks	Number of chunks to divide work into	128
log_dir	Directory for log files	"./logs"
chunk_lock_dir	Directory for chunk lock files	"./logs/locks"
monitor_interval	Monitoring interval in seconds	60
restart_failed	Whether to restart failed processes	false
max_retries	Maximum retries for failed chunks	3
memory_threshold	Memory threshold percentage	90
kwargs	Custom arguments to pass to script	{}

🎯 Using Kwargs

OctoRun supports passing additional keyword arguments to your scripts through both the configuration file and command line interface.

📋 Configuration File

Add kwargs to your config.json:

{
    "script_path": "./train_model.py",
    "gpus": "auto",
    "total_chunks": 128,
    "kwargs": {
        "batch_size": 64,
        "learning_rate": 0.01,
        "model_type": "transformer",
        "epochs": 10,
        "output_dir": "./results"
    }
}

🖥️ Command Line Interface

Override or add kwargs via command line:

# Override config kwargs
octorun run --config config.json --kwargs '{"batch_size": 128, "learning_rate": 0.005}'

# Add new kwargs
octorun run --config config.json --kwargs '{"model_type": "bert", "max_length": 512}'

🎯 Priority

CLI kwargs > Config file kwargs

CLI kwargs override config file kwargs for the same keys while preserving other config kwargs

🔧 Script Implementation

Your script must accept the required OctoRun arguments plus any custom kwargs:

import argparse

def main():
    parser = argparse.ArgumentParser()
    
    # 🔧 Required OctoRun arguments
    parser.add_argument('--gpu_id', type=int, required=True)
    parser.add_argument('--chunk_id', type=int, required=True)
    parser.add_argument('--total_chunks', type=int, required=True)
    
    # 🎯 Your custom arguments
    parser.add_argument('--batch_size', type=int, default=32)
    parser.add_argument('--learning_rate', type=float, default=0.001)
    parser.add_argument('--model_type', type=str, default='default')
    parser.add_argument('--epochs', type=int, default=1)
    parser.add_argument('--output_dir', type=str, default='./output')
    
    args = parser.parse_args()
    
    # ✨ Use the arguments in your script
    print(f"🚀 Processing chunk {args.chunk_id}/{args.total_chunks} on GPU {args.gpu_id}")
    print(f"🎯 Training with batch_size={args.batch_size}, lr={args.learning_rate}")
    
    # Your processing logic here
    ...

if __name__ == "__main__":
    main()

🎮 Commands

🚀 run

Run your script with the specified configuration:

octorun run --config config.json [--kwargs '{"key": "value"}']

💾 save_config

Generate a default configuration file:

octorun save_config [--script ./your_script.py]

🔍 list_gpus

List available GPUs:

octorun list_gpus [--detailed]

📚 Examples

🤖 Example 1: Machine Learning Training

Click to expand

Config file (ml_config.json):

{
    "script_path": "./train_model.py",
    "total_chunks": 64,
    "kwargs": {
        "batch_size": 32,
        "learning_rate": 0.001,
        "model_type": "resnet50",
        "epochs": 100,
        "dataset_path": "/data/imagenet"
    }
}

Command:

octorun run --config ml_config.json --kwargs '{"batch_size": 64, "learning_rate": 0.01}'

📊 Example 2: Data Processing

Click to expand

octorun run --config config.json --kwargs '{"input_dir": "/data/raw", "output_dir": "/data/processed", "compression": "gzip"}'

📊 Monitoring and Logging

OctoRun provides comprehensive logging:

Log Type	Location	Description
📋 Session logs	logs/session_TIMESTAMP.log	Overall session information
🧩 Chunk logs	logs/chunk_N.log	Individual chunk processing logs
🔒 Lock files	logs/locks/	Chunk completion tracking

📊 Real-time Monitoring

# Monitor session progress
tail -f logs/session_*.log

# Monitor specific chunk
tail -f logs/chunk_42.log

# Monitor GPU usage
watch -n 1 'octorun list_gpus --detailed'

🛠️ Error Handling

• 🔄 Automatic retry mechanism for failed chunks
• 📊 Configurable maximum retry attempts
• 💾 Memory threshold monitoring
• 📝 Comprehensive error logging

Robust error handling ensures your jobs complete successfully

📋 Requirements

• 🐍 Python ≥ 3.10
• 🎮 NVIDIA GPUs with CUDA support
• 🔧 nvidia-smi tool available in PATH

🤝 Contributing

We welcome contributions! Here's how to get started:

1. 🍴 Fork the repository
2. 🌿 Create a feature branch
3. ✨ Make your changes
4. 🧪 Add tests
5. 📤 Submit a pull request

📄 License

This project is licensed under the MIT License.

👨‍💻 Author

Haobo Yuan - [e-mail]

🙏 Acknowledgements

The project is highly relied on AI tools for code generation and documentation, enhancing productivity and code quality.

---

Made with ❤️ and 🤖 AI assistance

Star ⭐ this repo if you find it useful!