trainwatcher 0.3.0

Lightweight training awareness system for machine learning workflows with email/Telegram notifications.

TrainWatcher

A lightweight training awareness system for machine learning workflows.

TrainWatcher notifies developers when ML training completes or fails and provides a minimal run summary. Notifications can be delivered via email or Telegram (and a hosted cloud email backend is available). It is designed to be easy to integrate, framework-agnostic, and reliable.

Developed and managed by Sambit Mahapatra.

Install

pip install trainwatcher

Quickstart

from trainwatcher import monitor

monitor.start()

try:
    # Your training loop here.
    monitor.log(epoch=1, loss=0.42, val_acc=0.81)
    monitor.end()
except Exception as exc:
    monitor.fail(exc)
    raise

Convenience context manager:

from trainwatcher import monitor

with monitor.watch():
    # Your training loop here.
    pass

Single-call wrapper:

from trainwatcher import watch

watch(train_model, interpretation="rule")

Zero-config hosted interpretation:

from trainwatcher import watch

watch(train_model, interpretation="hybrid")

If the user already registered an email with add_email(...), hybrid uses the TrainWatcher hosted backend automatically. No user LLM key is required.

If you log both training and validation metrics, TrainWatcher can also add a deterministic observation and next-step suggestions to successful summaries.

from trainwatcher import monitor

monitor.start()

for epoch in range(1, 5):
    monitor.log(
        epoch=epoch,
        loss=1.0 / epoch,
        val_loss=0.7 + (epoch * 0.05),
    )

monitor.end()

Example completed summary:

Training Completed

Runtime: 2m
Final Loss: 0.25
Epochs: 4

Observation: Training loss improved while validation loss worsened.

Suggestions:
- Enable early stopping to stop near the best validation checkpoint.
- Increase regularization such as dropout or weight decay.
- Reduce the number of training epochs or add more data augmentation.

Configuration

TrainWatcher reads configuration from trainwatcher_config.yaml by default. You can also set TRAINWATCHER_CONFIG to an absolute path or pass a config dict to monitor.end(config=...) and monitor.fail(config=...).

Example config (YAML):

notifications:
  email: true
  telegram: false

email:
  host: smtp.example.com
  port: 587
  username: [e-mail]
  password: app_password
  sender: [e-mail]
  recipient: [e-mail]
  use_tls: true
  subject: TrainWatcher Notification

logging:
  enabled: true
  path: trainwatcher_run.json

interpretation:
  mode: hybrid    # rule | llm | hybrid
  fallback: rule  # rule | byok | none
  hosted:
    enabled: true

Advanced BYOK fallback example:

interpretation:
  mode: hybrid
  fallback: byok
  hosted:
    enabled: true
  byok:
    api_key: YOUR_OPTIONAL_KEY
    base_url: https://openrouter.ai/api/v1
    model: openai/gpt-oss-20b
    max_tokens: 300
    temperature: 0.2
    timeout_seconds: 20

An example config is available at examples/trainwatcher_config.yaml.

Cloud Notifications (Resend + Cloudflare)

If you want users to receive email without SMTP setup, use the TrainWatcher Cloud backend.

1. Deploy the Cloudflare Worker in cloudflare/ and set RESEND_API_KEY and RESEND_FROM.
2. If you self-host, set the backend URL locally:

export TRAINWATCHER_BASE_URL="https://your-worker.workers.dev"

3. Register an email once:

from trainwatcher import add_email

add_email("[e-mail]")

After that, monitor.end() and monitor.fail() will send notifications automatically if credentials exist. If you use the hosted TrainWatcher backend, no base URL is required. You can inspect the active backend URL with:

from trainwatcher import get_base_url
print(get_base_url())

Cloud backend enforces a default limit of 10 emails per user per day.

To remove the email:

from trainwatcher import delete_email

delete_email()

CLI

You can also register/delete emails via the CLI:

trainwatcher add-email [e-mail]
trainwatcher delete-email
trainwatcher help

Mid-Run Notifications

Manual milestone:

from trainwatcher import monitor

monitor.notify("Model loaded on GPU")

Time-based heartbeat:

monitor.heartbeat(interval_seconds=900)  # every 15 minutes
# ...
monitor.stop_heartbeat()

Step-based pings:

for epoch in range(100):
    train()
    monitor.step(notify_every=10)

Configure defaults once:

monitor.configure(
    heartbeat_interval=900,
    step_notify_every=10,
    heartbeat_message="Training still running",
)

Optional per-channel minimum intervals:

limits:
  email_min_interval_seconds: 1800
  telegram_min_interval_seconds: 300
  cloud_min_interval_seconds: 1800

Phase-02 Interpretation

TrainWatcher Phase-02 adds a deterministic interpretation layer on top of the original notification flow.

Current Phase-02 core:

• normalized metric aliases (loss, train_loss, val_loss, acc, val_acc, lr, etc.)
• rule-based analysis for overfitting, plateau, diverging training, and normal convergence
• short implementation-oriented suggestions
• duck-typed best-model extraction for sklearn-like search objects
• hosted LLM interpretation for hybrid and llm modes
• optional Markdown report rendering for advanced integrations

The rule engine stays lightweight and framework-agnostic. It does not depend on PyTorch, TensorFlow, scikit-learn, or transformers. It only consumes the metrics you log through TrainWatcher.

Advanced/internal modules available now:

• trainwatcher.best_model
• trainwatcher.llm
• trainwatcher.metrics
• trainwatcher.prompts
• trainwatcher.rules
• trainwatcher.suggestions
• trainwatcher.report

Rule-based interpretation always runs. Hosted LLM interpretation is available through config-driven llm or hybrid modes and uses the TrainWatcher backend by default. If the hosted LLM path fails or hits quota, TrainWatcher falls back to the rule-based summary. Advanced users can optionally configure BYOK fallback with the byok config block or TRAINWATCHER_LLM_* environment variables.

Best-model capture is available through the monitor today:

from trainwatcher import monitor

monitor.start()
monitor.set_best_model(grid_search)
monitor.end()

Environment Variables

These environment variables override config values when set.

• TRAINWATCHER_CONFIG
• TRAINWATCHER_BASE_URL
• TRAINWATCHER_API_KEY
• TRAINWATCHER_CREDENTIALS_PATH
• TRAINWATCHER_DISABLE_PROXY
• TRAINWATCHER_NOTIFICATIONS_EMAIL
• TRAINWATCHER_NOTIFICATIONS_TELEGRAM
• TRAINWATCHER_EMAIL_HOST
• TRAINWATCHER_EMAIL_PORT
• TRAINWATCHER_EMAIL_USERNAME
• TRAINWATCHER_EMAIL_PASSWORD
• TRAINWATCHER_EMAIL_SENDER
• TRAINWATCHER_EMAIL_RECIPIENT
• TRAINWATCHER_EMAIL_USE_TLS
• TRAINWATCHER_EMAIL_SUBJECT
• TRAINWATCHER_TELEGRAM_BOT_TOKEN
• TRAINWATCHER_TELEGRAM_CHAT_ID
• TRAINWATCHER_LOGGING_ENABLED
• TRAINWATCHER_LOGGING_PATH
• TRAINWATCHER_INTERPRETATION_MODE
• TRAINWATCHER_INTERPRETATION_FALLBACK
• TRAINWATCHER_LLM_API_KEY
• TRAINWATCHER_LLM_BASE_URL
• TRAINWATCHER_LLM_MODEL
• TRAINWATCHER_LLM_MAX_TOKENS
• TRAINWATCHER_LLM_TEMPERATURE
• TRAINWATCHER_LLM_TIMEOUT_SECONDS

The TRAINWATCHER_LLM_* variables are for optional BYOK fallback. Normal hosted interpretation does not require them.

Logging

Set logging.enabled: true in the config to write a run summary JSON file. The default path is trainwatcher_run.json relative to the config file location.

The JSON run log is an internal artifact for debugging or integrations. Normal usage does not require reading it directly.

Testing

Install dev dependencies and run tests:

pip install -e .[dev]
pytest

Vision

TrainWatcher aims to become an intelligent training awareness layer that lets developers manage and monitor long-running training processes without constant supervision. The long-term goal is an intelligent training companion that integrates into broader ML tooling ecosystems and improves developer productivity.

Mission

Provide a lightweight, reliable, easy-to-integrate system that captures training lifecycle events and delivers concise summaries through notification channels such as email or messaging services. The mission prioritizes:

• Simplicity of integration
• Reliability of notifications
• Minimal disruption to existing workflows

Problem Statement

Long-running ML training jobs often require repeated manual checks for completion or failure. This interrupts workflow and delays awareness of errors. Existing enterprise platforms are too heavy for many developers, and there is a gap for a simple, developer-friendly tool focused on basic training awareness.

Value Proposition

TrainWatcher allows developers to step away during training without losing awareness of outcomes. Minimal integration and reliable notifications reduce monitoring fatigue and improve experimentation efficiency.

License

Licensed under the GNU Lesser General Public License v3.0 (LGPL-3.0). See LICENSE.