inferyx-monitoring

Batch pipeline monitor with email alerts and Admin UI.

Install directory: /opt/pipeline-monitor
Admin URL: https://<your-host>/monitoring/admin/ (nginx required)
PyPI: inferyx-monitoring

Quick reference

Deployment runbook (upgrade, rollback, health checks): docs/DEPLOYMENT.md
Maintainer build docs: docs/BUILD.md (not for server operators).
Visual steps (git only, not in the PyPI package): docs/html/index.html. Build is project setup, git check-in, then pip check-in. Deploy is install, upgrade, rollback, then uninstall. Property descriptions: docs/html/config.html.

Prerequisites

Install from scratch

Step 1 — Install Python package

Replace VERSION with the target release (e.g. 1.0.56):

export VERSION=1.0.56
export IM_HOME=/opt/pipeline-monitor

sudo mkdir -p "$IM_HOME"
sudo useradd --system --home-dir "$IM_HOME" --shell /usr/sbin/nologin inferyx 2>/dev/null || true

sudo -u inferyx python3 -m venv "$IM_HOME/.venv"
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade pip
sudo -u inferyx "$IM_HOME/.venv/bin/pip" install "inferyx-monitoring==${VERSION}"

Or from a local wheel:

sudo -u inferyx "$IM_HOME/.venv/bin/pip" install /path/to/inferyx_monitoring-${VERSION}-py3-none-any.whl

Step 2 — Run install script

sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" install --skip-pip

This creates:

Step 3 — Configure

sudo nano /opt/pipeline-monitor/.env
sudo nano /opt/pipeline-monitor/auth.policy
sudo nano /opt/pipeline-monitor/batch_file.csv

Required after install:

1. .env — SMTP, PIPELINE_API_BASE_URL, PIPELINE_API_TOKEN_HEADER, alert toggles (PIPELINE_ALERT_*)
2. batch_file.csv — batch rows with a real Token per batch (not in .env)
3. auth.policy — ui.public_base_url, login providers, paths under /opt/pipeline-monitor

Secrets (never commit):

sudo mkdir -p /opt/pipeline-monitor/secrets
sudo openssl rand -hex 32 | sudo tee /opt/pipeline-monitor/secrets/session_secret
sudo chmod 600 /opt/pipeline-monitor/secrets/*
sudo chown -R inferyx:inferyx /opt/pipeline-monitor

.env templates: inferyx_pipeline_monitor/data/.env.example (in package: .venv/.../data/)

AWS IAM Identity Center (SAML 2.0)

Choose SAML 2.0 (not OAuth) when creating the AWS IDC application.

"aws_idc": {
  "enabled": true,
  "entity_id": "https://devops.inferyx.com/monitoring",
  "acs_url": "https://devops.inferyx.com/monitoring/api/auth/callback/aws_idc",
  "idp_entity_id": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_sso_url": "https://portal.sso.<region>.amazonaws.com/saml/assertion/<id>",
  "idp_certificate_path": "/opt/pipeline-monitor/secrets/aws_idc_idp.crt",
  "allowed_domains": ["inferyx.com"]
}

SP metadata (optional download for AWS setup):

https://devops.inferyx.com/monitoring/api/auth/saml/aws_idc/metadata

Google Workspace stays on OAuth 2.0 (auth.google).

Step 4 — nginx

sudo cp /opt/pipeline-monitor/.venv/share/inferyx-monitoring/nginx.conf.example \
        /etc/nginx/sites-available/pipeline-monitor
sudo ln -sf /etc/nginx/sites-available/pipeline-monitor /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Edit server_name and SSL paths in the nginx file for production.

Step 5 — Verify

sudo systemctl status inferyx-monitoring
curl -s http://127.0.0.1:8090/api/health
curl -s http://127.0.0.1:8090/api/ui/status

Open https://<your-host>/monitoring/admin/ in a browser.

Upgrade

export VERSION=1.0.56   # new version
export IM_HOME=/opt/pipeline-monitor

sudo -u inferyx "$IM_HOME/.venv/bin/pip" install --upgrade "inferyx-monitoring==${VERSION}"
sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip

Upgrade without service restart:

sudo bash "$IM_HOME/.venv/share/inferyx-monitoring/release.sh" upgrade --skip-pip --no-restart

The upgrade script:

• Deploys new Admin UI to /opt/pipeline-monitor/www/
• Merges new .env keys (existing values kept)
• Migrates batch_file.csv columns when needed (adds Token / Status; drops legacy Alert* CSV columns)
• Updates systemd unit
• Restarts inferyx-monitoring service

Your config is never overwritten: .env secrets, batch_file.csv rows, and auth.policy values stay as-is unless you edit them.

Post-upgrade checklist (1.0.56+)

1. Open Admin UI → Batch Data (or edit CSV) and set Token for each Active batch
2. In Application Settings / .env, set alert toggles as needed (PIPELINE_ALERT_MISSED, _FAILED, _RUNNING, _NO_DATA)
3. Remove unused PIPELINE_API_TOKEN= from .env if present (no longer used)
4. Confirm PIPELINE_API_TOKEN_HEADER is still set (e.g. token)
5. sudo systemctl status inferyx-monitoring and check logs if API calls fail

Uninstall / remove

Stop service, keep config

Keeps /opt/pipeline-monitor (.env, CSV, secrets, logs):

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --skip-pip

Remove everything

Deletes /opt/pipeline-monitor entirely:

sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh uninstall --purge --skip-pip

Manual cleanup (optional)

# nginx
sudo rm -f /etc/nginx/sites-enabled/pipeline-monitor
sudo rm -f /etc/nginx/sites-available/pipeline-monitor
sudo nginx -t && sudo systemctl reload nginx

# service user (only if nothing else uses it)
sudo userdel inferyx 2>/dev/null || true

Reinstall from scratch

After --purge, repeat Install from scratch.

Configuration files

All paths in auth.policy should point under /opt/pipeline-monitor/ on the server.

Batch CSV

Required columns: Name, Frequency, Token
Optional: ExpectedStartTime, AvgExecutionTime, ExpectedDayOfMonth, Status (Active / Suspended)

API auth uses the Token column per batch (header name from PIPELINE_API_TOKEN_HEADER in .env). There is no global PIPELINE_API_TOKEN.

Frequency: Daily, Hourly, Monthly, Monday–Sunday, OnDemand, Continuous.
Use a weekday name for weekly jobs (do not use Weekly). For multiple runs per day, set several times in ExpectedStartTime.

Alert types are enabled/disabled globally in .env (Application Settings), not in the CSV:

Scheduled (Daily, Hourly, Weekly, …)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
payroll_job,Daily,9:00:00,"30 mins",,YOUR_TOKEN,Active

Alerts: missed, failed, stuck, no_data (when enabled in .env)

On-demand (runtime trigger)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
manual_export,OnDemand,,"30 mins",,YOUR_TOKEN,Active

Use OnDemand (or Adhoc, Manual, Unscheduled). Leave start time empty.
AvgExecutionTime enables stuck alerts (API start + avg + grace).
Polling uses CHECK_WINDOW_MINUTES windows, not 24/7.

Continuous (always-on)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Token,Status
stream_job,Continuous,,,,YOUR_TOKEN,Active

Alerts: failed and no_data only (when enabled in .env).

Alert tuning (.env)

Key variables (full list in .env.example):

Test email threading:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --test-mail-thread

Troubleshooting

See CHANGELOG.md.

Service commands

sudo systemctl start inferyx-monitoring
sudo systemctl stop inferyx-monitoring
sudo systemctl restart inferyx-monitoring
sudo systemctl status inferyx-monitoring
sudo journalctl -u inferyx-monitoring -f

Or via package CLI:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl status
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring-ctl restart