idol-cli 0.1.0

IDOL - Incremental DAG Optimization for Learning CLI

IDOL - Incremental DAG Optimization for Learning

Transform debug traces into validated golden datasets for ML training pipelines.

Overview

IDOL is a CLI tool that processes debug trace JSON files to create high-quality labeled datasets through:

1. Automatic candidate generation using task-specific heuristics
2. Human review interface for label validation and correction
3. Dataset freezing with validation and optional train/holdout splits

Features

• 🎯 Deterministic ID generation ensures idempotent operations
• 📝 JSONL format enables git-friendly diffs and streaming processing
• ⚡ Memory-efficient parsing of large JSON files
• 🔍 8 task-specific heuristics for automatic labeling
• ✅ Interactive review with accept/negate/edit options
• ❄️ Dataset validation before freezing
• 📊 Train/holdout splits for ML evaluation

Installation

# Clone the repository
git clone <repository-url>
cd auto-taskopt

# Install dependencies
pip install -e .

# Or install from requirements
pip install -r requirements.txt

Quick Start

# 1. Harvest candidates from debug traces
idol harvest examples/20250912_121325/

# 2. Review candidates for a specific task
idol review --task gpu_hw_analysis

# 3. Freeze validated dataset
idol freeze --task gpu_hw_analysis

# Check status
idol status

Usage

Harvest: Generate Candidates

Process debug trace JSON files to extract task executions and generate candidates:

# Process a single file
idol harvest trace.json

# Process all JSON files in a directory
idol harvest examples/

# Verbose output
idol harvest examples/ -v

The harvest command:

• Parses debug traces to extract task executions
• Generates stable SHA1-based IDs
• Applies task-specific heuristics for auto-labeling
• Saves candidates to rca_gold/candidates/<task>.jsonl

Review: Validate Labels

Interactively review and correct auto-generated labels:

# Review all pending candidates for a task
idol review --task gpu_hw_analysis

# Review only first 10 items
idol review --task logs_analysis --max-items 10

Review options:

• [a] Accept auto label
• [n] Negate (mark as 'no_issue')
• [e] Edit (provide custom JSON)
• [s] Skip
• [q] Quit

Freeze: Create Final Datasets

Merge candidates and overrides into validated datasets:

# Freeze a specific task
idol freeze --task gpu_hw_analysis

# Freeze all tasks
idol freeze

# Create train/holdout split (80/20)
idol freeze --task final_analysis --holdout

# Skip validation
idol freeze --task network_analysis --no-validate

Output files:

• rca_gold/frozen/<task>.json - Main dataset
• rca_gold/frozen/<task>.holdout.json - Holdout set (if requested)

Additional Commands

# Show current status
idol status

# Clean all generated data
idol clean

Supported Tasks

IDOL supports 8 task types with specialized heuristics:

1. job_info - Job success/failure status
2. gpu_hw_analysis - GPU hardware failure detection (XIDs/SXIDs)
3. logs_analysis - Software errors and numerical instability
4. health_checks - System health status
5. scheduler_analysis - Scheduler termination analysis
6. network_analysis - Network/InfiniBand issues
7. storage_analysis - Storage I/O and metadata spikes
8. final_analysis - Root cause determination with confidence

Data Format

Candidate Record (JSONL)

{
  "id": "abc123def456",
  "task": "gpu_hw_analysis",
  "input": {
    "job_id": 4824587,
    "attempt": 0,
    "key_findings": "XID 48 hardware failure on node-001",
    "tool_calls": [{"name": "job_summary", "args": {}}]
  },
  "auto": {
    "result": "confirmed_hw_failure",
    "evidence": "XIDs: [48]"
  },
  "status": "pending"
}

Frozen Dataset (JSON)

[
  {
    "id": "abc123def456",
    "input": {
      "job_id": 4824587,
      "attempt": 0,
      "key_findings": "...",
      "tool_calls": []
    },
    "gold": {
      "result": "confirmed_hw_failure",
      "evidence": "Manual verification"
    }
  }
]

Project Structure

auto-taskopt/
├── src/idol/
│   ├── cli.py          # CLI commands
│   ├── harvester.py    # Candidate generation
│   ├── reviewer.py     # Human review interface
│   ├── freezer.py      # Dataset finalization
│   ├── heuristics.py   # Task-specific rules
│   ├── models.py       # Data models
│   └── utils.py        # Utilities
├── tests/              # Test suite
├── examples/           # Sample debug traces
└── rca_gold/          # Generated data
    ├── candidates/     # JSONL candidates
    ├── overrides/      # Human corrections
    └── frozen/         # Final datasets

Development

Running Tests

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=src/idol --cov-report=term-missing

# Run specific test module
pytest tests/test_heuristics.py -v

Code Quality

# Lint code
ruff check src/ --fix

# Type checking
mypy src/idol/ --strict

Performance

• Harvest: Process 100 traces in <30 seconds
• Review: Display each candidate in <100ms
• Freeze: Validate 10,000 records in <5 seconds

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit a pull request

License

MIT License - see LICENSE file for details