inferyx-monitoring 1.0.35

Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI

inferyx-monitoring

Monitor batch jobs from a CSV file, poll the Inferyx API, and send email (and optional Teams / Google Chat) alerts.

Version: 1.0.35 · PyPI: inferyx-monitoring · CLI: inferyx-monitoring

This page is the full guide: Install, Upgrade, and Whats new. Everything is on this one page — no external doc links.

---

Contents

• Install
• .env configuration
• Upgrade
• Admin UI
• Nginx (manual)
• Whats new

---

Install

First-time setup on a Linux server. Default paths use /opt/pipeline-monitor and user inferyx.

Quick install (script)

Copy both scripts to the server (from git repo server/), or use them from pip after install:

sudo bash install_inferyx_monitoring.sh

With Admin UI (nginx is configured manually — see Nginx):

sudo bash install_inferyx_monitoring.sh --admin --admin-ui

Pin a version:

sudo bash ./install_inferyx_monitoring.sh --pin 1.0.32

After pip install, scripts are also at (run with bash or chmod +x first):

/opt/pipeline-monitor/.venv/share/inferyx-monitoring/scripts/

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/scripts/install_inferyx_monitoring.sh

The script creates the user, venv, installs from PyPI, runs --init-config, and enables systemd.

Manual install (step by step)

Step 1 — Python and package

sudo apt update && sudo apt install -y python3 python3-venv python3-pip
id inferyx || sudo useradd --system --home-dir /opt/pipeline-monitor --shell /usr/sbin/nologin inferyx
sudo mkdir -p /opt/pipeline-monitor && sudo chown inferyx:inferyx /opt/pipeline-monitor
sudo -u inferyx python3 -m venv /opt/pipeline-monitor/.venv
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade pip inferyx-monitoring

Step 2 — Config files

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --init-config --work-dir /opt/pipeline-monitor

Creates /opt/pipeline-monitor/.env and /opt/pipeline-monitor/batch_file.csv if missing. See .env configuration for what is auto-updated vs manual.

Step 3 — Edit .env (required secrets)

sudo -u inferyx vi /opt/pipeline-monitor/.env
sudo chmod 600 /opt/pipeline-monitor/.env

Required variables

Variable	Description
PIPELINE_SMTP_HOST, PIPELINE_SMTP_PORT, PIPELINE_SMTP_USERNAME, PIPELINE_SMTP_PASSWORD	SMTP
PIPELINE_FROM_NAME	Sender name
PIPELINE_MAIL_TO	Alert recipients
PIPELINE_API_BASE_URL	API base URL (no name= in the URL)
PIPELINE_API_TOKEN, PIPELINE_API_TOKEN_HEADER	API auth
PIPELINE_DEVOPS_EMAIL	Recipient for no_data alerts

Recommended scheduling

PIPELINE_CHECK_MODE=schedule_windows
PIPELINE_CHECK_WINDOW_MINUTES=10
PIPELINE_SCHEDULE_GRACE_MINUTES=5
PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false

Step 4 — Edit batch_file.csv

One batch per row. Last column: Active or Suspended. Use 24-hour times.

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
my_batch,Daily,9:00:00,"10 mins",,Active

Step 5 — Test run

Run as one line (do not split with \):

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --once --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv

Step 6 — systemd service

sudo tee /etc/systemd/system/inferyx-monitoring.service <<'EOF'
[Unit]
Description=Inferyx Pipeline Batch Monitor
After=network-online.target

[Service]
Type=simple
User=inferyx
Group=inferyx
WorkingDirectory=/opt/pipeline-monitor
ExecStart=/opt/pipeline-monitor/.venv/bin/inferyx-monitoring --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv
Restart=always
RestartSec=10
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now inferyx-monitoring.service

Paths reference

Item	Path
Install directory	/opt/pipeline-monitor
Config	/opt/pipeline-monitor/.env
Batch list	/opt/pipeline-monitor/batch_file.csv
Log	/opt/pipeline-monitor/pipeline_script.log
Python venv	/opt/pipeline-monitor/.venv
Service user	inferyx
systemd unit	inferyx-monitoring.service

---

.env configuration

What happens to /opt/pipeline-monitor/.env on first install, --init-config, and every service start (including after pip upgrade).

You must set manually (first time)

These are never filled in automatically — edit before going to production:

Variable	Purpose
PIPELINE_SMTP_HOST, PIPELINE_SMTP_PORT, PIPELINE_SMTP_USERNAME, PIPELINE_SMTP_PASSWORD	SMTP server and auth
PIPELINE_FROM_NAME	Sender display name
PIPELINE_MAIL_TO	Alert recipients
PIPELINE_API_BASE_URL	Inferyx API URL (no name= in URL)
PIPELINE_API_TOKEN, PIPELINE_API_TOKEN_HEADER	API authentication
PIPELINE_DEVOPS_EMAIL	no_data alert recipient

Optional but usually set once: Teams/GChat webhooks (PIPELINE_TEAMS_*, PIPELINE_GCHAT_*), mail templates, scheduling tuning.

Updated automatically (safe)

On start or --init-config, the monitor only adds what is missing — it does not change your existing values:

Action	When
Create .env from template	File does not exist
Append missing keys from the package template	New version introduces new settings
Retire legacy PIPELINE_MAIL_BODY_* keys	Commented out; structured email layout used instead
Remove duplicate .env keys	Keeps one value per key
Create batch_file.csv from template	File does not exist
Copy jfl_batch.csv → batch_file.csv	Legacy name only; original kept
Add Status column to CSV	Column missing; existing rows get Active

Never changed automatically

Item	Notes
SMTP password, API token	Your secrets stay as-is
PIPELINE_MAIL_TO, subjects, templates you already set	Not overwritten
Batch names and schedule rows in CSV	Rows are not replaced
/etc/pipeline-monitor/auth.policy	Admin OAuth — always manual

After upgrade — what to check

1. Restart the service (migration runs on start):

   sudo systemctl restart inferyx-monitoring.service
   

2. Review .env for new keys appended at the bottom (marked with package version comment).
3. If upgrading from 1.0.14 or older, confirm PIPELINE_CHECK_MODE=schedule_windows is present.
4. For Admin UI, edit auth.policy manually when OAuth settings change — not updated by pip.

---

Upgrade

Latest version: 1.0.35

Quick upgrade (script)

sudo bash ./upgrade_inferyx_monitoring.sh
sudo bash ./upgrade_inferyx_monitoring.sh --mode admin
sudo bash ./upgrade_inferyx_monitoring.sh --mode full
sudo bash ./upgrade_inferyx_monitoring.sh --mode ui
sudo bash ./upgrade_inferyx_monitoring.sh --mode full --pin 1.0.32

From an existing install:

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/scripts/upgrade_inferyx_monitoring.sh --mode full

Mode	What it does
monitor	pip install --upgrade inferyx-monitoring + restart monitor
admin	Upgrade [admin] + restart monitor and admin API
full	Upgrade [admin] + refresh UI + restart monitor and admin API
ui	Refresh Admin UI static files only

Scripts do not change nginx. Reload nginx manually after UI update if needed — see Nginx.

Legacy CSV rename (from 1.0.14 or older): add --legacy-migrate.

Manual upgrade

Pick the row that matches what you want to update, run the commands, then restart the service.

Monitor only

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring
sudo systemctl restart inferyx-monitoring.service

Monitor + Admin API

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade 'inferyx-monitoring[admin]'
sudo systemctl restart inferyx-monitoring.service
sudo systemctl restart inferyx-monitoring-admin.service

Full stack (monitor + admin API + admin UI)

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade 'inferyx-monitoring[admin]'
sudo /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin
sudo systemctl restart inferyx-monitoring.service
sudo systemctl restart inferyx-monitoring-admin.service

Then reload nginx manually if UI files changed — Nginx.

Admin UI files only

Use this when the Python package is already up to date and you only need new static UI files.

From pip (after upgrading the package):

sudo /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin

From deploy tarball (copy inferyx-monitoring-deploy-VERSION.tar.gz to the server):

tar -xzf /tmp/inferyx-monitoring-deploy-VERSION.tar.gz -C /tmp
cd /tmp/inferyx-monitoring-deploy-VERSION
sudo ./scripts/install_admin_ui_deploy.sh

Reload nginx manually after static files update — Nginx.

Upgrade from 1.0.14 or older

sudo -u inferyx mv /opt/pipeline-monitor/jfl_batch.csv /opt/pipeline-monitor/batch_file.csv
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring
sudo systemctl restart inferyx-monitoring.service

Migration usually adds scheduling keys automatically on restart. If missing, add:

PIPELINE_CHECK_MODE=schedule_windows
PIPELINE_CHECK_WINDOW_MINUTES=10

Update systemd ExecStart to use --csv-file /opt/pipeline-monitor/batch_file.csv if it still points at jfl_batch.csv. See .env configuration.

Common problems after upgrade

Problem	Fix
unrecognized arguments:	Run CLI commands as one line
No email	Check SMTP settings in .env
API errors	Remove name= from API URL; set PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false
Service crash on 1.0.25	Upgrade to 1.0.26 or newer
Alert crash NameError: signature	Upgrade to 1.0.21 or newer
inferyx-monitoring-admin --help starts server	Upgrade to 1.0.28 or newer

---

Admin UI

Optional web UI to edit .env and batch CSV. Requires OAuth. Servers do not need Node.js or npm — UI files are pre-built in the pip package.

First-time Admin UI setup

1. Install admin extra

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install 'inferyx-monitoring[admin]'

2. Auth policy

sudo mkdir -p /etc/pipeline-monitor
sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/config/auth.policy.example /etc/pipeline-monitor/auth.policy
sudo chmod 600 /etc/pipeline-monitor/auth.policy
sudo vi /etc/pipeline-monitor/auth.policy

Set OAuth providers and ui.enabled: true.

Production URL paths (must match nginx and OAuth console):

Setting	Example value
ui.public_base_url	https://monitor.example.com
ui.ui_base_path	/monitoring/admin/
OAuth redirect_uri	https://monitor.example.com/monitoring/api/auth/callback/google

Browser URLs:

• UI: https://monitor.example.com/monitoring/admin/
• API: https://monitor.example.com/monitoring/api/ (nginx → 127.0.0.1:8090)

auth.policy is not updated by pip upgrade — edit manually when paths change.

3. Install UI static files

sudo /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin

4. Admin API service

sudo tee /etc/systemd/system/inferyx-monitoring-admin.service <<'EOF'
[Unit]
Description=Inferyx Pipeline Monitor Admin API
After=network-online.target

[Service]
Type=simple
User=inferyx
Group=inferyx
WorkingDirectory=/opt/pipeline-monitor
ExecStart=/opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin --host 127.0.0.1 --port 8090
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now inferyx-monitoring-admin.service

5. Nginx — configure manually: Nginx

Admin CLI

Use the venv path (sudo does not see the venv automatically):

/opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin --help
/opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin

---

Nginx (manual)

Install and upgrade scripts do not edit or reload nginx. Configure once (or after you change paths/TLS), then reload yourself.

1. Copy the example site file

sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/config/nginx-pipeline-monitor-admin.conf.example \
  /etc/nginx/sites-available/pipeline-monitor-admin

2. Edit for your server

sudo vi /etc/nginx/sites-available/pipeline-monitor-admin

Set server_name, SSL certificate paths, and paths if you changed the web root.

Paths must match auth.policy (ui_base_path and OAuth redirect_uri):

URL path	Purpose
/monitoring/admin/	Angular UI (static files)
/monitoring/api/	Admin API (proxy to port 8090)

Example snippet:

location /monitoring/api/ {
    proxy_pass http://127.0.0.1:8090/api/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

location /monitoring/admin/ {
    alias /var/www/pipeline-monitor-admin/;
    try_files $uri $uri/ /monitoring/admin/index.html;
}

location = /monitoring {
    return 302 /monitoring/admin/;
}

3. Enable and reload (you run this)

sudo ln -sf /etc/nginx/sites-available/pipeline-monitor-admin /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Run step 3 again after Admin UI static files are updated (inferyx-monitoring-admin-install-ui or deploy tarball).

---

Whats new

1.0.35

• Admin UI at /monitoring/admin/ (not /admin/) — auth.policy, nginx example, UI build default
• OAuth redirect_uri: /monitoring/api/auth/callback/...
• Login error redirect follows ui_base_path

Upgrade: update auth.policy + nginx + OAuth console, reinstall UI, reload nginx.

1.0.34

• Nginx is manual only — install/upgrade scripts no longer reload or copy nginx config
• README: .env auto vs manual config on install and upgrade

1.0.33

• Fix inferyx-monitoring-admin-install-ui — was looking in wrong directory for bundled UI files

Upgrade: pip install --upgrade 'inferyx-monitoring[admin]==1.0.33' then run install-ui again.

1.0.32

• Server install script: install_inferyx_monitoring.sh
• Server upgrade script: upgrade_inferyx_monitoring.sh (modes: monitor, admin, full, ui)
• Scripts ship in pip package under share/inferyx-monitoring/scripts/

Upgrade: sudo ./upgrade_inferyx_monitoring.sh --mode full or see Upgrade above.

1.0.31

• Single-page PyPI documentation: install, upgrade, and whats new on this README only
• Pip package no longer bundles separate doc files

Upgrade: pip install --upgrade inferyx-monitoring then restart services (see Upgrade above).

1.0.30

• Pre-built Admin UI in pip wheel — no npm on production servers
• inferyx-monitoring-admin-install-ui command
• Deploy tarball inferyx-monitoring-deploy-VERSION.tar.gz for UI-only updates

Upgrade: pip install --upgrade 'inferyx-monitoring[admin]' then run inferyx-monitoring-admin-install-ui if you use the Admin UI.

1.0.29

• Release notes moved to this README (PyPI cannot host separate doc files)

1.0.28

• inferyx-monitoring-admin --help and --version work without starting the server
• Nginx and systemd examples for Admin UI

1.0.27

• Admin web UI for .env and batch CSV (OAuth, audit log)
• Safer login error messages; plain-text mail status (no icons)

1.0.26

• Hotfix for startup crash on 1.0.25 (CHAT_CONFIG import error) — upgrade immediately if on 1.0.25

1.0.25

• Microsoft Teams and Google Chat webhook alerts
• Test with: inferyx-monitoring --test-chat-alerts --work-dir /opt/pipeline-monitor

1.0.21

• Fix alert email crash (NameError: signature)

1.0.20

• Auto-migration on start: adds missing .env keys, migrates jfl_batch.csv to batch_file.csv

1.0.15

• Default batch file renamed to batch_file.csv
• --init-config command
• PIPELINE_CHECK_MODE=schedule_windows recommended