logsage 0.1.7

LLM-based library for AI training job failure attribution and recommending auto-resume policy

LogSage

LogSage is an LLM-powered toolkit that analyzes AI training job logs, attributes root causes, and recommends an auto-resume policy to reduce wasted GPU time. It provides:

• Error extraction and de-duplication from SLURM job logs
• Root-cause attribution (e.g., hardware, configuration, memory, communication)
• Action recommendations (restart immediately vs stop) with justification
• Optional node isolation hints for hardware-related failures

Table of Contents

• Description
• Quickstart
• Components
• Configuration
• Testing
• Roadmap and Project Status
• Additional Resources
• Contributing

Description

• Problem: Training jobs fail for many reasons (hardware, networking, configuration, data). Manual root-cause analysis is slow and wastes GPU hours.
• Solution: LogSage uses log parsing + NVIDIA NIM to extract error patterns, attribute likely causes, and recommend whether to restart or stop, with an explanation.
• How it works (high level):
  1. Receive job logs (client-provided)
  2. Extract and cluster error lines; remove noise
  3. Attribute errors with an LLM using structured prompts and heuristics
  4. Recommend restart/stop and, when applicable, suggest temporal isolation of suspect nodes
• Benefits: Faster triage, increased data center availability, reduced GPU downtime, and improved error extraction coverage by leveraging LLMs.

Quickstart

Prerequisites:

• Python >= 3.9
• Poetry (recommended) or pip

Setup (using Poetry):

make install
# or
poetry install --with dev --all-extras

Install via pip:

python3 -m venv myenv
source myenv/bin/activate
pip install -U logsage --index-url=https://urm.nvidia.com/artifactory/api/pypi/nv-shared-pypi-local/simple --extra-index-url https://pypi.org/simple

Components

• logsage/auto_resume_policy/
  • CLI utilities to analyze local log files
  • Recommendations engine that suggests job auto-resume policies: 'STOP', 'RESTART', 'TEMPORAL-ISOLATION+RESTART'

Configuration

Configuration is managed via logsage/auto_resume_policy/config.py (Pydantic settings). Key variables:

• NVIDIA_API_KEY (required in production): API key for NVIDIA AI Endpoints (required for LLM calls)
• FAST_API_ROOT_PATH (optional): Root path when running behind a proxy
• DEBUG (optional): true/false (default true locally; set false in prod)

Testing

Run the test suite:

make test
# or
poetry run pytest

Coverage reports are configured in pyproject.toml.

Roadmap and Project Status

• Streamed/async log ingestion paths
• Integration with log collector like loki
• Expanded attribution categories and guardrails
• Improve test coverage and add end-to-end examples

Additional Resources

• Auto-Resume-Policy API & details: logsage/auto_resume_policy/README.md
• Fetcher & deployment details: logsage/fetcher/README.md
• Project configuration and developer tooling: pyproject.toml, Makefile

Internal dashboards (if applicable):

• Grafana (LogSage): https://grafana.nvidia.com/d/aeutclepcu41sf/logsage?orgId=290
• Kibana/ES example index (sandbox): https://gpuwa.nvidia.com/elasticsearch/df-sandbox-ohazai-logsage-test2-202508/_search

Contributing

For setup, development guidelines, and versioning information, see the Contributing Guide.