inferyx-monitoring 1.0.60

Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI

inferyx-monitoring

Batch pipeline monitor with email alerts and Admin UI.

Install directory: /opt/pipeline-monitor
Admin URL: https://<your-host>/monitoring/admin/ (nginx required)
PyPI: inferyx-monitoring

---

Quick reference

Task	Command
Fresh install	See Install from scratch
Upgrade	pip install --upgrade → release.sh upgrade --skip-pip
Uninstall	release.sh uninstall --skip-pip
Remove everything	release.sh uninstall --purge --skip-pip
Service status	sudo systemctl status inferyx-monitoring
Logs	sudo journalctl -u inferyx-monitoring -n 50

Maintainer build docs: docs/DEVELOPMENT.md (not for server operators).

---

Prerequisites

Requirement	Notes
OS	Ubuntu 22.04+ (or Debian with apt)
Python	3.10+ (python3, python3-venv)
User	inferyx system user (created by install script)
nginx	Reverse proxy for /monitoring/admin/ and /monitoring/api/
Network	SMTP, Inferyx API, Google OAuth and/or AWS IDC SAML

---

Install from scratch

Step 1 — Install Python package

Replace VERSION with the target release (e.g. 1.0.56):

export VERSION=1.0.56
export IM_HOME=/opt/pipeline-monitor

sudo mkdir -p "$IM_HOME"
sudo useradd --system --home-dir "$IM_HOME" --shell /usr/sbin/nologin inferyx 2>/dev/null || true

sudo -u inferyx python3 -m venv "$IM_HOME/.venv"
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade pip
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install "inferyx-monitoring==${VERSION}"

Or from a local wheel:

sudo -u inferyx "$IM_HOME/.venv/bin/pip" install /path/to/inferyx_monitoring-${VERSION}-py3-none-any.whl

Step 2 — Run install script

sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" install --skip-pip

This creates:

Path	Purpose
/opt/pipeline-monitor/.env	SMTP, API, alert settings
/opt/pipeline-monitor/auth.policy	Admin UI, OAuth, paths
/opt/pipeline-monitor/batch_file.csv	Batch schedules
/opt/pipeline-monitor/secrets/	session_secret, OAuth secrets
/opt/pipeline-monitor/www/	Admin UI static files
/opt/pipeline-monitor/logs/	Monitor log
inferyx-monitoring.service	systemd unit

Step 3 — Configure

sudo nano /opt/pipeline-monitor/.env
sudo nano /opt/pipeline-monitor/auth.policy
sudo nano /opt/pipeline-monitor/batch_file.csv

Required after install:

1. .env — SMTP, PIPELINE_API_BASE_URL, PIPELINE_API_TOKEN_HEADER, alert toggles (PIPELINE_ALERT_*)
2. batch_file.csv — batch rows with a real Token per batch (not in .env)
3. auth.policy — ui.public_base_url, login providers, paths under /opt/pipeline-monitor

Secrets (never commit):

sudo mkdir -p /opt/pipeline-monitor/secrets
sudo openssl rand -hex 32 | sudo tee /opt/pipeline-monitor/secrets/session_secret
sudo chmod 600 /opt/pipeline-monitor/secrets/*
sudo chown -R inferyx:inferyx /opt/pipeline-monitor

.env templates: inferyx_pipeline_monitor/data/.env.example (in package: .venv/.../data/)

AWS IAM Identity Center (SAML 2.0)

Choose SAML 2.0 (not OAuth) when creating the AWS IDC application.

AWS console field	auth.policy key	Example
Application ACS URL	acs_url	https://devops.inferyx.com/monitoring/api/auth/callback/aws_idc
Application SAML audience	entity_id	https://devops.inferyx.com/monitoring
IAM Identity Center SAML issuer URL	idp_entity_id	from IDC metadata
IAM Identity Center sign-in URL	idp_sso_url	from IDC metadata
IDC certificate	idp_certificate_path	/opt/pipeline-monitor/secrets/aws_idc_idp.crt

"aws_idc": {
  "enabled": true,
  "entity_id": "https://devops.inferyx.com/monitoring",
  "acs_url": "https://devops.inferyx.com/monitoring/api/auth/callback/aws_idc",
  "idp_entity_id": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_sso_url": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_certificate_path": "/opt/pipeline-monitor/secrets/aws_idc_idp.crt",
  "allowed_domains": ["inferyx.com"]
}

SP metadata (optional download for AWS setup):

https://devops.inferyx.com/monitoring/api/auth/saml/aws_idc/metadata

Google Workspace stays on OAuth 2.0 (auth.google).

---

Step 4 — nginx

sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/nginx.conf.example \
        /etc/nginx/sites-available/pipeline-monitor
sudo ln -sf /etc/nginx/sites-available/pipeline-monitor /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Edit server_name and SSL paths in the nginx file for production.

Step 5 — Verify

sudo systemctl status inferyx-monitoring
curl -s http://127.0.0.1:8090/api/health
curl -s http://127.0.0.1:8090/api/ui/status

Open https://<your-host>/monitoring/admin/ in a browser.

---

Upgrade

export VERSION=1.0.56   # new version
export IM_HOME=/opt/pipeline-monitor

sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade "inferyx-monitoring==${VERSION}"
sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip

Upgrade without service restart:

sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip --no-restart

The upgrade script:

• Deploys new Admin UI to /opt/pipeline-monitor/www/
• Merges new .env keys (existing values kept)
• Migrates batch_file.csv columns when needed (adds Token / Status; drops legacy Alert* CSV columns)
• Updates systemd unit
• Restarts inferyx-monitoring service

Your config is never overwritten: .env secrets, batch_file.csv rows, and auth.policy values stay as-is unless you edit them.

Post-upgrade checklist (1.0.56+)

1. Open Admin UI → Batch Data (or edit CSV) and set Token for each Active batch
2. In Application Settings / .env, set alert toggles as needed (PIPELINE_ALERT_MISSED, _FAILED, _RUNNING, _NO_DATA)
3. Remove unused PIPELINE_API_TOKEN= from .env if present (no longer used)
4. Confirm PIPELINE_API_TOKEN_HEADER is still set (e.g. token)
5. sudo systemctl status inferyx-monitoring and check logs if API calls fail

---

Uninstall / remove

Stop service, keep config

Keeps /opt/pipeline-monitor (.env, CSV, secrets, logs):

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --skip-pip

Remove everything

Deletes /opt/pipeline-monitor entirely:

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --purge --skip-pip

Manual cleanup (optional)

# nginx
sudo rm -f /etc/nginx/sites-enabled/pipeline-monitor
sudo rm -f /etc/nginx/sites-available/pipeline-monitor
sudo nginx -t && sudo systemctl reload nginx

# service user (only if nothing else uses it)
sudo userdel inferyx 2>/dev/null || true

Reinstall from scratch

After --purge, repeat Install from scratch.

---

Configuration files

File	Template	Description
.env	data/.env.example	SMTP, API base URL / token header, alert toggles
batch_file.csv	data/batch_file.csv.example	Batch names, schedules, per-batch Token
auth.policy	release/auth.policy.example	Admin UI, OAuth/SAML, file paths

All paths in auth.policy should point under /opt/pipeline-monitor/ on the server.

---

Batch CSV

Required columns: Name, Frequency, Token
Optional: ExpectedStartTime, AvgExecutionTime, ExpectedDayOfMonth, Status (Active / Suspended)

API auth uses the Token column per batch (header name from PIPELINE_API_TOKEN_HEADER in .env). There is no global PIPELINE_API_TOKEN.

Frequency: Daily, Hourly, Monthly, Monday–Sunday, OnDemand, Continuous.
Use a weekday name for weekly jobs (do not use Weekly). For multiple runs per day, set several times in ExpectedStartTime.

Alert types are enabled/disabled globally in .env (Application Settings), not in the CSV:

Variable	Controls	Default
PIPELINE_ALERT_MISSED	Did not start (missed)	true
PIPELINE_ALERT_FAILED	Failed status	true
PIPELINE_ALERT_RUNNING	Running too long / stuck	true
PIPELINE_ALERT_NO_DATA	API/auth lookup failure	true

Scheduled (Daily, Hourly, Weekly, …)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
payroll_job,Daily,9:00:00,"30 mins",,YOUR_TOKEN,Active

Alerts: missed, failed, stuck, no_data (when enabled in .env)

On-demand (runtime trigger)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
manual_export,OnDemand,,"30 mins",,YOUR_TOKEN,Active

Use OnDemand (or Adhoc, Manual, Unscheduled). Leave start time empty.
AvgExecutionTime enables stuck alerts (API start + avg + grace).
Polling uses CHECK_WINDOW_MINUTES windows, not 24/7.

Continuous (always-on)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
stream_job,Continuous,,,,YOUR_TOKEN,Active

Alerts: failed and no_data only (when enabled in .env).

---

Alert tuning (.env)

Key variables (full list in .env.example):

Variable	Default	Purpose
PIPELINE_CHECK_MODE	schedule_windows	Poll in short windows vs continuous
PIPELINE_CHECK_WINDOW_MINUTES	10	Poll window length
PIPELINE_SCHEDULE_GRACE_MINUTES	5	Delay before missed/stuck alerts
PIPELINE_ALERT_MISSED	true	Enable missed (did not start) alerts
PIPELINE_ALERT_FAILED	true	Enable failed-status alerts
PIPELINE_ALERT_RUNNING	true	Enable stuck / running-too-long alerts
PIPELINE_ALERT_NO_DATA	true	Enable API/auth error alerts
PIPELINE_AUDIT_RETENTION_DAYS	30	Keep admin audit JSONL entries for N days (0 = forever)
PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS	true	One new email per issue per day
PIPELINE_ALERT_REPLY_TO_THREAD	false	Follow-ups as reply-all in same thread

Test email threading:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --test-mail-thread

---

Troubleshooting

Symptom	Fix
No such file ... release.sh	Run pip install inferyx-monitoring first
Service restart loop	Check .env — journalctl -u inferyx-monitoring -n 50
nginx 404 on /monitoring/admin/	Use nginx.conf.example; check ui_base_path in auth.policy
Admin UI "Unavailable"	curl http://127.0.0.1:8090/api/ui/status
Missed vs no_data	Empty API after schedule → missed; API error → no_data
Batch API 401 / no_data	Set Token on that CSV row; check PIPELINE_API_TOKEN_HEADER
Alerts not sending	Check PIPELINE_ALERT_MISSED / _FAILED / _RUNNING / _NO_DATA in .env
OAuth / SAML login fails	Google: match redirect_uri. AWS IDC: use SAML 2.0; match acs_url / entity_id; check IdP cert file

See CHANGELOG.md.

---

Service commands

sudo systemctl start inferyx-monitoring
sudo systemctl stop inferyx-monitoring
sudo systemctl restart inferyx-monitoring
sudo systemctl status inferyx-monitoring
sudo journalctl -u inferyx-monitoring -f

Or via package CLI:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl status
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl restart