inferyx-monitoring 1.0.37

Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI

inferyx-monitoring

Version: 1.0.37 · PyPI: inferyx-monitoring · CLI: inferyx-monitoring

---

Contents

1. Overview
2. What's new
3. Installation
4. Upgrade
5. Properties
6. Batch CSV format

---

1. Overview

inferyx-monitoring watches batch jobs listed in a CSV file, queries the Inferyx API for each batch, and sends email alerts (and optional Teams / Google Chat messages) when problems are detected.

Alert	When it fires
failed	Batch execution failed
running	Still running past expected end + grace
missed	Did not start within schedule window + grace
no_data	No API record (email to DevOps)

Two parts on the server

Part	What it is	Required?
Monitor	Polls batches, sends alerts	Yes
Admin portal	Web UI + backend API (OAuth, edit .env / CSV)	Optional

Admin portal = UI + backend together. Always install and upgrade them as one unit. No Node.js on the server — UI is pre-built static files under /var/www/.

Browser  →  https://<host>/monitoring/admin/  →  nginx  →  /var/www/pipeline-monitor-admin/  (static HTML/JS)
Browser  →  https://<host>/monitoring/api/    →  nginx  →  127.0.0.1:8090  (inferyx-monitoring-admin)

Paths

Item	Path
Monitor install	/opt/pipeline-monitor
Monitor venv	/opt/pipeline-monitor/.venv
Monitor config	/opt/pipeline-monitor/.env
Batch CSV	/opt/pipeline-monitor/batch_file.csv
Monitor service	inferyx-monitoring.service
Admin backend service	inferyx-monitoring-admin.service (port 8090)
Admin static files	/var/www/pipeline-monitor-admin/
Admin auth policy	/etc/pipeline-monitor/auth.policy
Public UI URL	https://<host>/monitoring/admin/
Public API URL	https://<host>/monitoring/api/

Commands below use: VENV=/opt/pipeline-monitor/.venv/bin

---

2. What's new

1.0.37

• Clearer install/upgrade: Admin portal (UI + backend) as one deployment; static files under /var/www/.

1.0.36

• PyPI documentation restructured: overview, install, upgrade, properties, CSV format only.

1.0.35

• Admin UI URL path: /monitoring/admin/ (was /admin/).
• OAuth callback: /monitoring/api/auth/callback/....
• Nginx and auth.policy must use the new paths.

1.0.34

• Nginx configuration is manual only (scripts do not reload nginx).
• .env migration rules documented (auto vs manual).

1.0.33

• Fix inferyx-monitoring-admin-install-ui (wrong path to bundled UI files).

1.0.32

• Server install/upgrade scripts in pip package (share/inferyx-monitoring/scripts/).

1.0.30

• Pre-built Admin UI in pip wheel; inferyx-monitoring-admin-install-ui command.

1.0.27

• Admin web UI; auth.policy for OAuth and paths.

1.0.25

• Teams and Google Chat webhook alerts.

1.0.20

• Auto .env migration on start; retires PIPELINE_MAIL_BODY_* keys.

1.0.15

• Default batch file batch_file.csv (was jfl_batch.csv).
• PIPELINE_CHECK_MODE=schedule_windows recommended.
• Status column in CSV.
• --init-config command.

---

3. Installation

Choose A (monitor only) or A then B (monitor + admin portal).

---

A. Install monitor (required)

Step	Action
1	Create user, venv, install package
2	Create .env and batch_file.csv
3	Edit .env and CSV
4	Test once
5	Enable systemd service

Step 1 — User, venv, package

sudo apt update && sudo apt install -y python3 python3-venv python3-pip
id inferyx || sudo useradd --system --home-dir /opt/pipeline-monitor --shell /usr/sbin/nologin inferyx
sudo mkdir -p /opt/pipeline-monitor && sudo chown inferyx:inferyx /opt/pipeline-monitor
sudo -u inferyx python3 -m venv /opt/pipeline-monitor/.venv
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade pip 'inferyx-monitoring==1.0.37'

Step 2 — Config files

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --init-config --work-dir /opt/pipeline-monitor

Step 3 — Edit config (see properties and CSV)

sudo -u inferyx vi /opt/pipeline-monitor/.env
sudo -u inferyx vi /opt/pipeline-monitor/batch_file.csv
sudo chmod 600 /opt/pipeline-monitor/.env

Step 4 — Test

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --once \
  --work-dir /opt/pipeline-monitor \
  --env-file /opt/pipeline-monitor/.env \
  --csv-file /opt/pipeline-monitor/batch_file.csv

Step 5 — systemd

sudo tee /etc/systemd/system/inferyx-monitoring.service <<'EOF'
[Unit]
Description=Inferyx Pipeline Batch Monitor
After=network-online.target

[Service]
Type=simple
User=inferyx
Group=inferyx
WorkingDirectory=/opt/pipeline-monitor
ExecStart=/opt/pipeline-monitor/.venv/bin/inferyx-monitoring --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv
Restart=always
RestartSec=10
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now inferyx-monitoring.service

Monitor install done.

---

B. Install admin portal (optional — UI + backend together)

Do this after section A. The admin portal has 3 pieces — install all of them:

#	Piece	Where it runs
1	Backend API	pip [admin] + inferyx-monitoring-admin.service → 127.0.0.1:8090
2	UI static pages	/var/www/pipeline-monitor-admin/ (copied from pip package)
3	Nginx	Public URLs /monitoring/admin/ + /monitoring/api/

Plus one config file: /etc/pipeline-monitor/auth.policy (OAuth).

Step 1 — Backend (pip)

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install 'inferyx-monitoring[admin]==1.0.37'

Step 2 — Auth policy (manual — edit OAuth and URLs)

sudo mkdir -p /etc/pipeline-monitor
sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/config/auth.policy.example \
  /etc/pipeline-monitor/auth.policy
sudo chmod 600 /etc/pipeline-monitor/auth.policy
sudo vi /etc/pipeline-monitor/auth.policy

Minimum settings:

• ui.enabled: true
• ui.ui_base_path: /monitoring/admin/
• auth.google.redirect_uri: https://<your-host>/monitoring/api/auth/callback/google
• Register that redirect URI in Google / AWS console

See auth.policy properties.

Step 3 — UI static files → /var/www/

sudo /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui \
  --target /var/www/pipeline-monitor-admin

Step 4 — Backend systemd service

sudo tee /etc/systemd/system/inferyx-monitoring-admin.service <<'EOF'
[Unit]
Description=Inferyx Pipeline Monitor Admin API
After=network-online.target

[Service]
Type=simple
User=inferyx
Group=inferyx
WorkingDirectory=/opt/pipeline-monitor
ExecStart=/opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin --host 127.0.0.1 --port 8090
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now inferyx-monitoring-admin.service

Step 5 — Nginx (manual — one-time setup)

sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/config/nginx-pipeline-monitor-admin.conf.example \
  /etc/nginx/sites-available/pipeline-monitor-admin
sudo vi /etc/nginx/sites-available/pipeline-monitor-admin   # server_name, SSL certs
sudo ln -sf /etc/nginx/sites-available/pipeline-monitor-admin /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Step 6 — Verify

Check	URL
UI	https://<host>/monitoring/admin/
API health	https://<host>/monitoring/api/health

Admin portal install done.

---

C. Install script (optional shortcut)

sudo bash install_inferyx_monitoring.sh                      # monitor only
sudo bash install_inferyx_monitoring.sh --admin --admin-ui   # monitor + admin portal

After pip install, scripts are at: /opt/pipeline-monitor/.venv/share/inferyx-monitoring/scripts/

Nginx is still configured manually (step B.5 above).

---

4. Upgrade

4.1 Choose your upgrade

I want to upgrade…	Go to
Monitor only (no admin portal)	4.2 Monitor only
Admin portal (backend + UI static pages)	4.3 Admin portal
Everything (monitor + admin portal)	4.4 Full upgrade
Very old install (before 1.0.15)	4.5 Legacy

Rule: Admin portal = backend pip package + /var/www/ static files. Upgrade both together. Nginx config rarely changes unless paths change.

4.2 Upgrade — monitor only

Use when you do not use the admin portal.

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring
sudo systemctl restart inferyx-monitoring.service

On restart, missing .env keys are added automatically (secrets are not changed). Review /opt/pipeline-monitor/.env after upgrade.

4.3 Upgrade — admin portal (UI + backend)

Use when the monitor is already fine and you only need a new admin portal version.

Step	Command
1. Upgrade backend (pip)	sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade 'inferyx-monitoring[admin]'
2. Copy UI to /var/www/	sudo /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin
3. Restart backend	sudo systemctl restart inferyx-monitoring-admin.service
4. Reload nginx	sudo nginx -t && sudo systemctl reload nginx

Copy-paste:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade 'inferyx-monitoring[admin]==1.0.37'
sudo /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin
sudo systemctl restart inferyx-monitoring-admin.service
sudo nginx -t && sudo systemctl reload nginx

Edit /etc/pipeline-monitor/auth.policy or OAuth console only if release notes say URL paths changed.

4.4 Upgrade — everything

Use when you want monitor + admin portal on the latest version.

Step	Command
1. Upgrade pip (includes admin backend)	pip install --upgrade 'inferyx-monitoring[admin]'
2. Copy UI to /var/www/	inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin
3. Restart monitor	systemctl restart inferyx-monitoring.service
4. Restart admin backend	systemctl restart inferyx-monitoring-admin.service
5. Reload nginx	nginx -t && systemctl reload nginx

Copy-paste:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade 'inferyx-monitoring[admin]==1.0.37'
sudo /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-admin-install-ui --target /var/www/pipeline-monitor-admin
sudo systemctl restart inferyx-monitoring.service
sudo systemctl restart inferyx-monitoring-admin.service
sudo nginx -t && sudo systemctl reload nginx

4.5 Upgrade — from 1.0.14 or older

sudo -u inferyx mv /opt/pipeline-monitor/jfl_batch.csv /opt/pipeline-monitor/batch_file.csv

Then run 4.4 Full upgrade or 4.2 Monitor only.

Update systemd ExecStart if it still has --csv-file .../jfl_batch.csv.

4.6 What changes automatically vs manually

Auto on service start	Manual
Missing .env keys appended	SMTP password, API token
PIPELINE_MAIL_BODY_* retired	auth.policy
Status column added to CSV if missing	Nginx (unless paths change)
jfl_batch.csv copied to batch_file.csv	OAuth redirect URIs in Google/AWS

4.7 Upgrade script (optional)

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/scripts/upgrade_inferyx_monitoring.sh --mode monitor
sudo bash .../upgrade_inferyx_monitoring.sh --mode admin   # admin portal (pip + /var/www/ + restart admin)
sudo bash .../upgrade_inferyx_monitoring.sh --mode full      # monitor + admin portal

--mode	Upgrades
monitor	Monitor pip only + restart monitor
admin	Admin portal: pip [admin] + /var/www/ UI + restart admin API
full	Monitor + admin portal (both services restarted)
ui	/var/www/ UI files only + restart admin API

---

5. Properties

5.1 .env file (/opt/pipeline-monitor/.env)

Required (set manually on first install)

Property	Description
PIPELINE_SMTP_HOST	SMTP server hostname
PIPELINE_SMTP_PORT	SMTP port (e.g. 587)
PIPELINE_SMTP_USERNAME	SMTP username
PIPELINE_SMTP_PASSWORD	SMTP password
PIPELINE_FROM_NAME	Sender display name
PIPELINE_MAIL_TO	Alert email recipients (comma-separated)
PIPELINE_API_BASE_URL	Inferyx API URL — no name= in URL
PIPELINE_API_TOKEN	API authentication token
PIPELINE_API_TOKEN_HEADER	Header name (e.g. token or Authorization)
PIPELINE_DEVOPS_EMAIL	Recipient for no_data alerts

Optional — scheduling and API

Property	Default	Description	Since
PIPELINE_CHECK_MODE	schedule_windows	schedule_windows (recommended) or full_window	1.0.15
PIPELINE_CHECK_WINDOW_MINUTES	10	Minutes around start/end to poll API	1.0.15
PIPELINE_CHECK_INTERVAL	60	Seconds between checks (legacy mode)	original
PIPELINE_SCHEDULE_GRACE_MINUTES	5	Grace after expected start/end	original
PIPELINE_POST_RUN_GRACE_MINUTES	60	Grace after run completes	original
PIPELINE_API_FILTER_BY_SCHEDULE_DATE	false	Add schedule date to API query	original
PIPELINE_API_RETRY_COUNT	3	API retry attempts	original
PIPELINE_API_RETRY_BACKOFF_SEC	5	Seconds between API retries	original
PIPELINE_ALERT_COOLDOWN_MINUTES	60	Minutes before repeat alert	original
PIPELINE_FAILED_ALERT_ONCE_PER_DAY	true	Limit failed alerts per day	original
PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS	true	Limit all alert types per day	original
PIPELINE_MAIL_CC	(empty)	CC recipients	original

Optional — Teams / Google Chat

Property	Default	Description	Since
PIPELINE_TEAMS_ENABLED	false	Enable Teams webhooks	1.0.25
PIPELINE_TEAMS_WEBHOOK_URL	(empty)	Teams incoming webhook URL	1.0.25
PIPELINE_GCHAT_ENABLED	false	Enable Google Chat webhooks	1.0.25
PIPELINE_GCHAT_WEBHOOK_URL	(empty)	Google Chat webhook URL	1.0.25
PIPELINE_CHAT_ALERT_NO_DATA	false	Also send no_data to chat (default: email only)	1.0.25
PIPELINE_CHAT_GREETING	(built-in)	Chat message greeting	1.0.25
PIPELINE_CHAT_SIGNATURE	(built-in)	Chat message signature	1.0.25

Test chat: inferyx-monitoring --test-chat-alerts --work-dir /opt/pipeline-monitor

Optional — email subjects

Property	Description	Since
PIPELINE_MAIL_SUBJECT_DEFAULT	Default subject template	original
PIPELINE_MAIL_SUBJECT_NO_DATA	Subject for no_data	original
PIPELINE_MAIL_SUBJECT_FAILED	Subject for failed	original
PIPELINE_MAIL_SUBJECT_RUNNING	Subject for running	original
PIPELINE_MAIL_SUBJECT_MISSED	Subject for missed	original

Placeholders: {batch_name}, {issue_type_upper}, etc.

Optional — email layout (structured HTML)

Property	Description	Since
PIPELINE_MAIL_GREETING	Email greeting line	1.0.17
PIPELINE_MAIL_ENVIRONMENT	Environment label in email	original
PIPELINE_MAIL_HEADING_BATCH_DETAILS	Section heading	1.0.19
PIPELINE_MAIL_HEADING_ADDITIONAL_INFO	Section heading	1.0.19
PIPELINE_MAIL_HEADING_ALERT_SUMMARY	Section heading	1.0.19
PIPELINE_MAIL_HEADING_ACTION_REQUIRED	Section heading	1.0.19
PIPELINE_MAIL_HEADING_CURRENT_STATUS	Section heading	1.0.19
PIPELINE_MAIL_INTRO_RUNNING	Body section (running)	1.0.19
PIPELINE_MAIL_SUMMARY_RUNNING	Body section (running)	1.0.19
PIPELINE_MAIL_ACTION_RUNNING	Body section (running)	1.0.19
PIPELINE_MAIL_STATUS_RUNNING	Body section (running)	1.0.19
PIPELINE_MAIL_SIGNATURE	Email signature HTML	original
PIPELINE_MAIL_FOOTER_NOTE	Footer note HTML	original

Similar PIPELINE_MAIL_INTRO_*, SUMMARY_*, ACTION_*, STATUS_* exist for failed, missed, no_data (built-in defaults if omitted).

Retired (do not add — removed on upgrade)

Property	Replaced by	Since retired
PIPELINE_MAIL_BODY_DEFAULT	Structured sections above	1.0.20
PIPELINE_MAIL_BODY_NO_DATA	Structured sections above	1.0.20
PIPELINE_MAIL_BODY_FAILED	Structured sections above	1.0.20
PIPELINE_MAIL_BODY_RUNNING	Structured sections above	1.0.20
PIPELINE_MAIL_BODY_MISSED	Structured sections above	1.0.20

5.2 auth.policy (/etc/pipeline-monitor/auth.policy)

Admin UI only. Not updated by pip. JSON format. Copy from share/inferyx-monitoring/config/auth.policy.example.

ui section

Property	Example	Description	Since
enabled	true	Turn Admin UI on/off	1.0.27
session_timeout_minutes	60	Login session length	1.0.27
public_base_url	https://monitor.example.com	Public site URL (no trailing path)	1.0.27
ui_base_path	/monitoring/admin/	UI path under public URL	1.0.35 (was /admin/)

Browser UI URL = public_base_url + ui_base_path → https://monitor.example.com/monitoring/admin/

auth section (OAuth)

Property	Description	Since
auth.google.enabled	Enable Google login	1.0.27
auth.google.client_id	Google OAuth client ID	1.0.27
auth.google.client_secret_path	File containing client secret	1.0.27
auth.google.redirect_uri	https://<host>/monitoring/api/auth/callback/google	1.0.35 path
auth.google.allowed_domains	Allowed email domains	1.0.27
auth.aws_idc.*	AWS IAM Identity Center OAuth (same pattern)	1.0.27

Register redirect_uri in Google / AWS OAuth console.

paths section

Property	Default	Description	Since
work_dir	/opt/pipeline-monitor	Monitor install directory	1.0.27
env_file	/opt/pipeline-monitor/.env	Path to .env	1.0.27
csv_file	/opt/pipeline-monitor/batch_file.csv	Path to batch CSV	1.0.27
audit_log	/var/log/pipeline-monitor/admin-audit.log	Admin audit log	1.0.27
session_secret_path	/etc/pipeline-monitor/session_secret	Session signing secret file	1.0.27

branding section

Property	Description	Since
app_name	UI title	1.0.27
logo_url	Logo image URL	1.0.27
footer_text	Footer text	1.0.27
primary_color	Theme color (hex)	1.0.27
favicon_url	Favicon URL	1.0.27

security section

Property	Default	Description	Since
restart_service_on_save	true	Restart monitor after config save from UI	1.0.27
service_name	inferyx-monitoring	systemd service to restart	1.0.27
rate_limit_per_minute	120	API rate limit per IP	1.0.27

---

6. Batch CSV format

6.1 Current format (batch_file.csv)

File: /opt/pipeline-monitor/batch_file.csv (since 1.0.15; was jfl_batch.csv)

Example:

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
daily_report,Daily,9:00:00,"10 mins",,Active
monthly_close,Monthly,6:00:00,"45 mins",1,Active
adhoc_job,Once,14:30:00,"5 mins",,Suspended

Use 24-hour times. Quote values that contain commas.

6.2 Column reference

Column	Required	Description	Since
Name	Yes	Batch name as known to Inferyx API	original
Frequency	Yes	Daily, Weekly, Monthly, or Once	original
ExpectedStartTime	Yes	Expected start time (HH:MM:SS)	original
AvgExecutionTime	Yes	Typical duration (e.g. "10 mins", "1 hour")	original
ExpectedDayOfMonth	No	Day of month for monthly jobs (1–31); empty for Daily/Weekly	original
Status	Yes*	Active or Suspended — only Active rows are monitored	1.0.15

*If Status column is missing, upgrade/migration adds it and sets existing rows to Active.

6.3 Legacy format (before 1.0.15)

Item	Old	Current
Filename	jfl_batch.csv	batch_file.csv
Columns	Same except no Status column	Includes Status
systemd --csv-file	May point to jfl_batch.csv	Use batch_file.csv

Migration on start copies jfl_batch.csv → batch_file.csv (original kept) and adds Status if missing.

6.4 Frequency values

Value	Meaning
Daily	Runs every day at ExpectedStartTime
Weekly	Runs weekly at ExpectedStartTime
Monthly	Runs on ExpectedDayOfMonth at ExpectedStartTime
Once	Single scheduled run

6.5 Status values

Value	Meaning
Active	Monitored
Suspended	Ignored (not polled)

Also accepted by Admin UI: Inactive, Disabled (treated as not active).