telegram-log-service

telegram-log-service — a server that receives ML training logs via HTTP and sends real-time alerts through a Telegram bot. Designed to work with messenger-logger-callback.

Architecture

Training Script                      Telegram Log Service                 Telegram
┌──────────────────┐   HTTP POST    ┌─────────────────────┐             ┌──────────┐
│ MessengerLogger  │ ─────────────> │ /api/logs handler   │             │          │
│ or Callback      │   /api/logs    │   ↓                 │   Bot API   │ Telegram │
│ + heartbeat      │                │ global_state        │ ──────────> │ Users    │
└──────────────────┘                │   ↓                 │             │          │
                                    │ alerting → bot      │             └──────────┘
                                    │ staleness_checker   │
                                    └─────────────────────┘

Flow:

1. Training scripts send JSON events (logs, status updates, heartbeats) to POST /api/logs.
2. The web handler updates in-memory run state and triggers alerts when appropriate.
3. The Telegram bot sends alerts to subscribed users and responds to commands.
4. A background staleness checker detects crashed/stalled runs.

Prerequisites

• Python 3.8+
• A Telegram bot token (create one via @BotFather)

Installation

From source (pip)

git clone https://github.com/Riko0/telegram_log_service.git
cd telegram_log_service
pip install .

Configure

cp .env.example .env
# Edit .env and fill in your TELEGRAM_BOT_TOKEN and ADMIN_TELEGRAM_NAME

Run

After installing, the telegram-log-service command is available system-wide:

telegram-log-service

Or using the Python module:

python -m telegram_log_service

Docker

# From the telegram_log_service directory:
chmod +x deploy/docker/build_docker.sh deploy/scripts/startup.sh
./deploy/docker/build_docker.sh

The Docker image installs the package via pip install . and runs telegram-log-service as the entry point. Pass your .env file via --env-file.

Configuration

All settings are via environment variables (or .env file). See .env.example for a complete template.

API

POST /api/logs

Receives training events. Requires Authorization: Bearer <token> header if WEB_AUTH_TOKEN is set.

Required fields:

Optional fields:

Any other top-level keys are stored as run metadata.

GET /health

Returns server status:

{"status": "ok", "active_runs": 3}

Bot Commands

User Commands

Admin Commands

Alerts

The bot sends alerts to subscribed users when:

If ClearML is detected, alerts include a direct link to the ClearML dashboard.

Heartbeat

When the client library sends heartbeat events (every ~60 seconds by default), the server uses a shorter stall threshold (HEARTBEAT_STALL_THRESHOLD_SECONDS, default 300s) for faster crash detection. Runs without heartbeats use the standard STALL_ALERT_THRESHOLD_SECONDS (default 1800s). This is fully backwards-compatible -- old clients work the same as before.

Data Persistence

• Whitelist, subscribers, user info are saved to JSON files and survive restarts.
• Training run data is saved to training_data.json on every meaningful event (not heartbeats) and restored on startup.

Related Projects

• messenger-logger-callback — the client library that sends training logs to this service. pip install messenger-logger-callback

License

MIT