inferyx-monitoring

Batch pipeline monitor with email alerts and Admin UI.

Install directory: /opt/pipeline-monitor
Admin URL: https://<your-host>/monitoring/admin/ (nginx required)
PyPI: inferyx-monitoring

Quick reference

Task	Command
Fresh install	See Install from scratch
Upgrade	`pip install --upgrade` → `release.sh upgrade --skip-pip`
Uninstall	`release.sh uninstall --skip-pip`
Remove everything	`release.sh uninstall --purge --skip-pip`
Service status	`sudo systemctl status inferyx-monitoring`
Logs	`sudo journalctl -u inferyx-monitoring -n 50`

Maintainer build docs: docs/DEVELOPMENT.md (not for server operators).

Prerequisites

Requirement	Notes
OS	Ubuntu 22.04+ (or Debian with `apt`)
Python	3.10+ (`python3`, `python3-venv`)
User	`inferyx` system user (created by install script)
nginx	Reverse proxy for `/monitoring/admin/` and `/monitoring/api/`
Network	SMTP, Inferyx API, Google OAuth and/or AWS IDC SAML

Install from scratch

Step 1 — Install Python package

Replace VERSION with the target release (e.g. 1.0.56):

export VERSION=1.0.56
export IM_HOME=/opt/pipeline-monitor

sudo mkdir -p "$IM_HOME"
sudo useradd --system --home-dir "$IM_HOME" --shell /usr/sbin/nologin inferyx 2>/dev/null || true

sudo -u inferyx python3 -m venv "$IM_HOME/.venv"
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade pip
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install "inferyx-monitoring==${VERSION}"

Or from a local wheel:

sudo -u inferyx "$IM_HOME/.venv/bin/pip" install /path/to/inferyx_monitoring-${VERSION}-py3-none-any.whl

Step 2 — Run install script

sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" install --skip-pip

This creates:

Path	Purpose
`/opt/pipeline-monitor/.env`	SMTP, API, alert settings
`/opt/pipeline-monitor/auth.policy`	Admin UI, OAuth, paths
`/opt/pipeline-monitor/batch_file.csv`	Batch schedules
`/opt/pipeline-monitor/secrets/`	session_secret, OAuth secrets
`/opt/pipeline-monitor/www/`	Admin UI static files
`/opt/pipeline-monitor/logs/`	Monitor log
`inferyx-monitoring.service`	systemd unit

Step 3 — Configure

sudo nano /opt/pipeline-monitor/.env
sudo nano /opt/pipeline-monitor/auth.policy
sudo nano /opt/pipeline-monitor/batch_file.csv

Required after install:

.env — SMTP, PIPELINE_API_BASE_URL, PIPELINE_API_TOKEN_HEADER, alert toggles (PIPELINE_ALERT_*)
batch_file.csv — batch rows with a real Token per batch (not in .env)
auth.policy — ui.public_base_url, login providers, paths under /opt/pipeline-monitor

Secrets (never commit):

sudo mkdir -p /opt/pipeline-monitor/secrets
sudo openssl rand -hex 32 | sudo tee /opt/pipeline-monitor/secrets/session_secret
sudo chmod 600 /opt/pipeline-monitor/secrets/*
sudo chown -R inferyx:inferyx /opt/pipeline-monitor

.env templates: inferyx_pipeline_monitor/data/.env.example (in package: .venv/.../data/)

AWS IAM Identity Center (SAML 2.0)

Choose SAML 2.0 (not OAuth) when creating the AWS IDC application.

AWS console field	`auth.policy` key	Example
Application ACS URL	`acs_url`	`https://devops.inferyx.com/monitoring/api/auth/callback/aws_idc`
Application SAML audience	`entity_id`	`https://devops.inferyx.com/monitoring`
IAM Identity Center SAML issuer URL	`idp_entity_id`	from IDC metadata
IAM Identity Center sign-in URL	`idp_sso_url`	from IDC metadata
IDC certificate	`idp_certificate_path`	`/opt/pipeline-monitor/secrets/aws_idc_idp.crt`

"aws_idc": {
  "enabled": true,
  "entity_id": "https://devops.inferyx.com/monitoring",
  "acs_url": "https://devops.inferyx.com/monitoring/api/auth/callback/aws_idc",
  "idp_entity_id": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_sso_url": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_certificate_path": "/opt/pipeline-monitor/secrets/aws_idc_idp.crt",
  "allowed_domains": ["inferyx.com"]
}

SP metadata (optional download for AWS setup):

https://devops.inferyx.com/monitoring/api/auth/saml/aws_idc/metadata

Google Workspace stays on OAuth 2.0 (auth.google).

Step 4 — nginx

sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/nginx.conf.example \
        /etc/nginx/sites-available/pipeline-monitor
sudo ln -sf /etc/nginx/sites-available/pipeline-monitor /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Edit server_name and SSL paths in the nginx file for production.

Step 5 — Verify

sudo systemctl status inferyx-monitoring
curl -s http://127.0.0.1:8090/api/health
curl -s http://127.0.0.1:8090/api/ui/status

Open https://<your-host>/monitoring/admin/ in a browser.

Upgrade

export VERSION=1.0.56   # new version
export IM_HOME=/opt/pipeline-monitor

sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade "inferyx-monitoring==${VERSION}"
sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip

Upgrade without service restart:

sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip --no-restart

The upgrade script:

Deploys new Admin UI to /opt/pipeline-monitor/www/
Merges new .env keys (existing values kept)
Migrates batch_file.csv columns when needed (adds Token / Status; drops legacy Alert* CSV columns)
Updates systemd unit
Restarts inferyx-monitoring service

Your config is never overwritten: .env secrets, batch_file.csv rows, and auth.policy values stay as-is unless you edit them.

Post-upgrade checklist (1.0.56+)

Open Admin UI → Batch Data (or edit CSV) and set Token for each Active batch
In Application Settings / .env, set alert toggles as needed (PIPELINE_ALERT_MISSED, _FAILED, _RUNNING, _NO_DATA)
Remove unused PIPELINE_API_TOKEN= from .env if present (no longer used)
Confirm PIPELINE_API_TOKEN_HEADER is still set (e.g. token)
sudo systemctl status inferyx-monitoring and check logs if API calls fail

Uninstall / remove

Stop service, keep config

Keeps /opt/pipeline-monitor (.env, CSV, secrets, logs):

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --skip-pip

Remove everything

Deletes /opt/pipeline-monitor entirely:

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --purge --skip-pip

Manual cleanup (optional)

# nginx
sudo rm -f /etc/nginx/sites-enabled/pipeline-monitor
sudo rm -f /etc/nginx/sites-available/pipeline-monitor
sudo nginx -t && sudo systemctl reload nginx

# service user (only if nothing else uses it)
sudo userdel inferyx 2>/dev/null || true

Reinstall from scratch

After --purge, repeat Install from scratch.

Configuration files

File	Template	Description
`.env`	`data/.env.example`	SMTP, API base URL / token header, alert toggles
`batch_file.csv`	`data/batch_file.csv.example`	Batch names, schedules, per-batch Token
`auth.policy`	`release/auth.policy.example`	Admin UI, OAuth/SAML, file paths

All paths in auth.policy should point under /opt/pipeline-monitor/ on the server.

Batch CSV

Required columns: Name, Frequency, Token
Optional: ExpectedStartTime, AvgExecutionTime, ExpectedDayOfMonth, Status (Active / Suspended)

API auth uses the Token column per batch (header name from PIPELINE_API_TOKEN_HEADER in .env). There is no global PIPELINE_API_TOKEN.

Frequency: Daily, Hourly, Monthly, Monday–Sunday, OnDemand, Continuous.
Use a weekday name for weekly jobs (do not use Weekly). For multiple runs per day, set several times in ExpectedStartTime.

Alert types are enabled/disabled globally in .env (Application Settings), not in the CSV:

Variable	Controls	Default
`PIPELINE_ALERT_MISSED`	Did not start (missed)	`true`
`PIPELINE_ALERT_FAILED`	Failed status	`true`
`PIPELINE_ALERT_RUNNING`	Running too long / stuck	`true`
`PIPELINE_ALERT_NO_DATA`	API/auth lookup failure	`true`

Scheduled (Daily, Hourly, Weekly, …)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
payroll_job,Daily,9:00:00,"30 mins",,YOUR_TOKEN,Active

Alerts: missed, failed, stuck, no_data (when enabled in .env)

On-demand (runtime trigger)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
manual_export,OnDemand,,"30 mins",,YOUR_TOKEN,Active

Use OnDemand (or Adhoc, Manual, Unscheduled). Leave start time empty.
AvgExecutionTime enables stuck alerts (API start + avg + grace).
Polling uses CHECK_WINDOW_MINUTES windows, not 24/7.

Continuous (always-on)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
stream_job,Continuous,,,,YOUR_TOKEN,Active

Alerts: failed and no_data only (when enabled in .env).

Alert tuning (.env)

Key variables (full list in .env.example):

Variable	Default	Purpose
`PIPELINE_CHECK_MODE`	`schedule_windows`	Poll in short windows vs continuous
`PIPELINE_CHECK_WINDOW_MINUTES`	`10`	Poll window length
`PIPELINE_SCHEDULE_GRACE_MINUTES`	`5`	Delay before missed/stuck alerts
`PIPELINE_ALERT_MISSED`	`true`	Enable missed (did not start) alerts
`PIPELINE_ALERT_FAILED`	`true`	Enable failed-status alerts
`PIPELINE_ALERT_RUNNING`	`true`	Enable stuck / running-too-long alerts
`PIPELINE_ALERT_NO_DATA`	`true`	Enable API/auth error alerts
`PIPELINE_AUDIT_RETENTION_DAYS`	`30`	Keep admin audit JSONL entries for N days (`0` = forever)
`PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS`	`true`	One new email per issue per day
`PIPELINE_ALERT_REPLY_TO_THREAD`	`false`	Follow-ups as reply-all in same thread

Test email threading:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --test-mail-thread

Troubleshooting

Symptom	Fix
`No such file ... release.sh`	Run `pip install inferyx-monitoring` first
Service restart loop	Check `.env` — `journalctl -u inferyx-monitoring -n 50`
nginx 404 on `/monitoring/admin/`	Use `nginx.conf.example`; check `ui_base_path` in `auth.policy`
Admin UI "Unavailable"	`curl http://127.0.0.1:8090/api/ui/status`
Missed vs no_data	Empty API after schedule → missed; API error → no_data
Batch API 401 / no_data	Set Token on that CSV row; check `PIPELINE_API_TOKEN_HEADER`
Alerts not sending	Check `PIPELINE_ALERT_MISSED` / `_FAILED` / `_RUNNING` / `_NO_DATA` in `.env`
OAuth / SAML login fails	Google: match `redirect_uri`. AWS IDC: use SAML 2.0; match `acs_url` / `entity_id`; check IdP cert file

See CHANGELOG.md.

Service commands

sudo systemctl start inferyx-monitoring
sudo systemctl stop inferyx-monitoring
sudo systemctl restart inferyx-monitoring
sudo systemctl status inferyx-monitoring
sudo journalctl -u inferyx-monitoring -f

Or via package CLI:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl status
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl restart

Metadata

Release files for inferyx-monitoring 1.0.66

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for inferyx-monitoring 1.0.66
File	Size	Uploaded
inferyx_monitoring-1.0.66.tar.gz	210.0 kB	Aug 29, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for inferyx-monitoring 1.0.66
File	Interpreter	ABI	Platform	Reset
inferyx_monitoring-1.0.66-py3-none-any.whl 218.9 kB Aug 29, 2026	Python 3	none	any	Details

Total release size: 428.9 kB

Release files / inferyx_monitoring-1.0.66.tar.gz

Download URL	inferyx_monitoring-1.0.66.tar.gz
Size	210.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`9663824314131cacca1d2ce65e3acd364951473c9d25222ea880307adb5dddfb`
BLAKE2b-256 checksum How to use checksums	`65ae79bc7cdad7b8a3e988c4220167ab614a3c10714446e27d37cfbf15295a2d`
Upload date	Aug 29, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.10.12`

Release files / inferyx_monitoring-1.0.66-py3-none-any.whl

Download URL	inferyx_monitoring-1.0.66-py3-none-any.whl
Size	218.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`2b111dfe67be7143db0ceb60f058a586315d044a76b65b6268c7beb4951e9409`
BLAKE2b-256 checksum How to use checksums	`ccf8d61e9fbb2c21d2d084ed7d4db15a25f2aabf27c3fb0365c7660c7a922883`
Upload date	Aug 29, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.10.12`