inferyx-monitoring 1.0.23

Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI

inferyx-monitoring — Server deployment

Monitor batch jobs from CSV, poll status via API, and send email alerts for failures, missed runs, long-running jobs, and missing API data.

Package: inferyx-monitoring
CLI: inferyx-monitoring
Current version: 1.0.20 (PyPI)

---

What's new in 1.0.20

• Auto config migration on upgrade — each start (and --init-config) merges missing .env keys only; keeps SMTP/API/subjects/batch rows.
• Retires legacy plain PIPELINE_MAIL_BODY_* lines so structured emails apply automatically.
• Copies jfl_batch.csv → batch_file.csv if needed; adds Status column when missing.

What's new in 1.0.19

• Structured alert emails — greeting, intro, Batch Details table, Additional Information, Alert Summary, Action Required, Current Status, sign-off, and footer note.
• Fully configurable from .env — PIPELINE_MAIL_GREETING, PIPELINE_MAIL_ENVIRONMENT, section headings, per-alert intro/summary/action/status, PIPELINE_MAIL_SIGNATURE, PIPELINE_MAIL_FOOTER_NOTE.

What's new in 1.0.18

• Styled HTML alert emails — tables, status badges, formatted sign-off.

What's new in 1.0.17

• Email bodies — greeting, Expected Start, Grace Time, Alert After, and closing block.
• New .env keys: PIPELINE_MAIL_GREETING, placeholders {grace_time} and {alert_after_time}.

What's new in 1.0.16

• Updated deployment docs — full upgrade guide on PyPI (this README).
• Upgrade paths — separate steps for 1.0.15 → 1.0.16 (pip only) vs 1.0.14 or older (pip + manual config).
• No code changes — documentation release only.

What's new in 1.0.15

• batch_file.csv — default batch list filename (replaces jfl_batch.csv).
• --init-config — creates .env and batch_file.csv only; removes leftover .example files.
• Example templates — dummy batch names, localhost API default, {batch_name} email subjects.
• PIPELINE_CHECK_MODE=schedule_windows, Status column, API URL fixes.

Check modes

PIPELINE_CHECK_MODE	Behavior
schedule_windows	Recommended. START window at ExpectedStartTime + grace; END window at ExpectedStartTime + AvgExecutionTime + grace. No API calls outside those windows.
full_window	Legacy: poll from expected start until end of day.

Example — Daily, 7:00, "10 mins", Active, grace 5, window 10:

Window	Time	Alerts
START	7:05 – 7:15	missed, no_data, failed
END	7:15 – 7:25	long-running, failed

---

Paths

Item	Path
Install directory	/opt/pipeline-monitor
Config	/opt/pipeline-monitor/.env
Batch list	/opt/pipeline-monitor/batch_file.csv
Log file	/opt/pipeline-monitor/pipeline_script.log
Python venv	/opt/pipeline-monitor/.venv
Service user	inferyx
systemd unit	inferyx-monitoring.service

---

1. Install

sudo apt update
sudo apt install -y python3 python3-venv python3-pip

Service user — check first; create only if missing:

id inferyx

If the command returns no such user, create the user:

sudo useradd --system --home-dir /opt/pipeline-monitor --shell /usr/sbin/nologin inferyx

If the user already exists, skip the command above and continue.

sudo mkdir -p /opt/pipeline-monitor
sudo chown inferyx:inferyx /opt/pipeline-monitor

sudo -u inferyx python3 -m venv /opt/pipeline-monitor/.venv
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade pip "inferyx-monitoring==1.0.16"

Fresh install pulls 1.0.16 from PyPI. To install whatever is latest later: pip install inferyx-monitoring.

---

2. Create config files (first time)

cd /opt/pipeline-monitor
sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --init-config --work-dir /opt/pipeline-monitor

Creates .env and batch_file.csv if missing.
On every run, auto-migrates existing config: adds missing .env keys, retires old mail-body overrides, migrates CSV safely.
Never overwrites SMTP/API secrets, custom subjects, or batch rows.

---

3. Configure .env

sudo -u inferyx vi /opt/pipeline-monitor/.env
sudo chmod 600 /opt/pipeline-monitor/.env

Required

Variable	Description
PIPELINE_SMTP_HOST	SMTP server (e.g. smtp.office365.com)
PIPELINE_SMTP_PORT	SMTP port (usually 587)
PIPELINE_SMTP_USERNAME	SMTP login / sender email
PIPELINE_FROM_NAME	Display name in From header
PIPELINE_SMTP_PASSWORD	SMTP password
PIPELINE_MAIL_TO	Primary alert recipients (comma-separated)
PIPELINE_API_BASE_URL	API path only — no name= in URL
PIPELINE_API_TOKEN	API authentication token
PIPELINE_API_TOKEN_HEADER	Header name for token (usually token)
PIPELINE_DEVOPS_EMAIL	Recipient for no_data and script-failure alerts

API example

PIPELINE_API_BASE_URL=http://localhost:8080/framework/metadata/getBaseEntityStatusByCriteria
PIPELINE_API_TOKEN=your_inferyx_api_token
PIPELINE_API_TOKEN_HEADER=token
PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false

Replace localhost with your Inferyx host. Use the base path only — the monitor adds name=<batch> for each row in batch_file.csv.
Keep PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false unless your API supports startDate / endDate.

Email templates (optional)

Variable	Purpose
PIPELINE_MAIL_CC	CC recipients
PIPELINE_MAIL_GREETING	Opening line (default: Hello Team,)
PIPELINE_MAIL_ENVIRONMENT	Shown in Additional Information (default: Production)
PIPELINE_MAIL_HEADING_*	Section titles: BATCH_DETAILS, ADDITIONAL_INFO, ALERT_SUMMARY, ACTION_REQUIRED, CURRENT_STATUS
PIPELINE_MAIL_INTRO_*	Intro paragraph per alert: RUNNING, FAILED, MISSED, NO_DATA, DEFAULT
PIPELINE_MAIL_SUMMARY_*	Alert Summary section (supports {batch_name}, etc.)
PIPELINE_MAIL_ACTION_*	Action Required section (HTML <ul> supported)
PIPELINE_MAIL_STATUS_*	Current Status section
PIPELINE_MAIL_SIGNATURE	Sign-off block
PIPELINE_MAIL_FOOTER_NOTE	Automated notification / ITSM footer
PIPELINE_MAIL_BODY_*	Optional full HTML layout override per alert type

Dynamic fields in tables: {batch_name}, {status}, {expected_start_time}, {expected_end_time}, {grace_time}, {alert_after_time}, {frequency}, {avg_time}, {current_time}, {delay}, {environment}.

Omit PIPELINE_MAIL_BODY_* to use the built-in structured layout (recommended). Customize sections via PIPELINE_MAIL_INTRO_RUNNING, etc. See .env.example.

Example subjects (dummy names — real batch name comes from {batch_name} at runtime):

PIPELINE_MAIL_SUBJECT_DEFAULT=Inferyx | Batch Alert | {batch_name} | {issue_type_upper}
PIPELINE_MAIL_SUBJECT_FAILED=Inferyx | Batch Failed: {batch_name}

Scheduling (optional)

Variable	Default	Meaning
PIPELINE_CHECK_MODE	full_window	schedule_windows = check only at start/end windows; full_window = poll all day
PIPELINE_CHECK_WINDOW_MINUTES	10	Minutes each start/end check window stays open
PIPELINE_CHECK_INTERVAL	60	Seconds between service poll cycles
PIPELINE_SCHEDULE_GRACE_MINUTES	5	Minutes after expected start/end before alerts
PIPELINE_POST_RUN_GRACE_MINUTES	60	Used only when CHECK_MODE=full_window
PIPELINE_ALERT_COOLDOWN_MINUTES	60	Cooldown between repeat alerts
PIPELINE_FAILED_ALERT_ONCE_PER_DAY	true	One failed alert per day per batch
PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS	true	One alert per issue type per day

Recommended production settings:

PIPELINE_CHECK_MODE=schedule_windows
PIPELINE_CHECK_WINDOW_MINUTES=10
PIPELINE_SCHEDULE_GRACE_MINUTES=5

---

4. Configure batch_file.csv

sudo -u inferyx vi /opt/pipeline-monitor/batch_file.csv

Rules:

• One batch per line — each row must end with Active or Suspended on its own line.
• Do not glue rows (bad: Activebatch_other or Statusbatch_other).
• Quote values with commas or spaces: "09:00,10:00", "3 mins".
• Use 24-hour time: 0:30:00, 16:30:00.

Column	Required	Description
Name	Yes	Batch name as in the Inferyx API
Frequency	Yes	Daily, Hourly, Monthly, weekday name, etc.
ExpectedStartTime	Scheduled	9:30:00 or "09:00,10:00,11:00"
AvgExecutionTime	Recommended	"3 mins", "10 mins", "1 Hr"
ExpectedDayOfMonth	Monthly	Day 1–31
Status	Optional	Active or Suspended (default: Active)

Example (replace dummy names with your Inferyx batch names)

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
dummy_batch_name,Daily,9:00:00,"10 mins",,Active
dummy_batch_other,Daily,12:00:00,"5 mins",,Suspended

Upgrading from jfl_batch.csv? See §7 Upgrade.

---

5. Test

Run as one single line (do not use \ line breaks — they cause unrecognized arguments errors):

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --once --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv

tail -100 /opt/pipeline-monitor/pipeline_script.log

---

6. Start service (systemd)

sudo tee /etc/systemd/system/inferyx-monitoring.service <<'EOF'
[Unit]
Description=Inferyx Pipeline Batch Monitor
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=inferyx
Group=inferyx
WorkingDirectory=/opt/pipeline-monitor
ExecStart=/opt/pipeline-monitor/.venv/bin/inferyx-monitoring --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv
Restart=always
RestartSec=10
Environment=PYTHONUNBUFFERED=1
Environment=PIPELINE_ENV_FILE=/opt/pipeline-monitor/.env
Environment=PIPELINE_LOG_FILE=/opt/pipeline-monitor/pipeline_script.log
Environment=PIPELINE_CSV_FILE=/opt/pipeline-monitor/batch_file.csv

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now inferyx-monitoring.service
sudo systemctl status inferyx-monitoring.service

Logs:

sudo journalctl -u inferyx-monitoring.service -f

---

7. Upgrade

Check installed version:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip show inferyx-monitoring | grep Version

Installed version	What to do
1.0.20	Nothing — you are current.
1.0.16 – 1.0.19	§7.1 only (pip install + restart). Migration runs automatically.
1.0.14 or older	§7.1 + review §7.2 if paths still use jfl_batch.csv.

7.1 Upgrade the package

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring
sudo systemctl restart inferyx-monitoring.service

Auto migration (1.0.20+) runs on every service start — no manual .env edit required:

Action	Safe?
Add missing keys to .env (e.g. new PIPELINE_MAIL_INTRO_RUNNING)	Yes — existing keys kept
Retire legacy PIPELINE_MAIL_BODY_* (plain HTML)	Yes — enables structured emails
Copy jfl_batch.csv → batch_file.csv if only legacy file exists	Yes — original kept
Add Status column to CSV if missing (default Active)	Yes — rows preserved
Overwrite SMTP/API token, subjects, batch names	Never

Optional — see migration output:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --init-config --work-dir /opt/pipeline-monitor

7.2 Manual changes (1.0.14 or older only)

Skip §7.2–7.5 if you are already on 1.0.15 or 1.0.16.

#	Item	Action
1	Batch CSV filename	Default is now batch_file.csv (was jfl_batch.csv). Choose one: rename your file, or keep the old name and point the service at it (step 2).
2	systemd unit	Update ExecStart, --csv-file, and PIPELINE_CSV_FILE if you renamed the CSV (see step 3).
3	.env (recommended)	Add or confirm these lines (your SMTP/API secrets stay as they are):

PIPELINE_CHECK_MODE=schedule_windows
PIPELINE_CHECK_WINDOW_MINUTES=10
PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false

Optional — only if you want custom email subjects (use {batch_name} for the real batch name at runtime):

PIPELINE_MAIL_SUBJECT_DEFAULT=Inferyx | Batch Alert | {batch_name} | {issue_type_upper}
PIPELINE_MAIL_SUBJECT_NO_DATA=Inferyx | Batch No Data: {batch_name}
PIPELINE_MAIL_SUBJECT_FAILED=Inferyx | Batch Failed: {batch_name}
PIPELINE_MAIL_SUBJECT_RUNNING=Inferyx | Batch Running: {batch_name}
PIPELINE_MAIL_SUBJECT_MISSED=Inferyx | Batch Missed: {batch_name}

#	Item	Action
4	API URL	Must be base path only — no name= in PIPELINE_API_BASE_URL. Example: http://<your-inferyx-host>:8080/framework/metadata/getBaseEntityStatusByCriteria
5	CSV Status column	Add Active or Suspended as the last column per row if missing.
6	Restart service	Required after pip upgrade and any systemd edits.

7.3 Option A — Rename CSV to batch_file.csv (recommended)

sudo -u inferyx mv /opt/pipeline-monitor/jfl_batch.csv /opt/pipeline-monitor/batch_file.csv

Update systemd (if it still references jfl_batch.csv):

sudo sed -i 's|jfl_batch.csv|batch_file.csv|g' /etc/systemd/system/inferyx-monitoring.service
sudo systemctl daemon-reload
sudo systemctl restart inferyx-monitoring.service

7.4 Option B — Keep jfl_batch.csv (no rename)

Edit the service file and set the old path explicitly:

sudo vi /etc/systemd/system/inferyx-monitoring.service

Ensure these lines use jfl_batch.csv:

ExecStart=.../inferyx-monitoring --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/jfl_batch.csv
Environment=PIPELINE_CSV_FILE=/opt/pipeline-monitor/jfl_batch.csv

Then:

sudo systemctl daemon-reload
sudo systemctl restart inferyx-monitoring.service

7.5 Test after upgrade

Run as one single line:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/inferyx-monitoring --once --work-dir /opt/pipeline-monitor --env-file /opt/pipeline-monitor/.env --csv-file /opt/pipeline-monitor/batch_file.csv

If you kept jfl_batch.csv, change --csv-file to that path.

tail -50 /opt/pipeline-monitor/pipeline_script.log
sudo systemctl status inferyx-monitoring.service

7.6 What is not changed by upgrade

File / setting	Changed by pip upgrade?
/opt/pipeline-monitor/.env	No
Your batch CSV (batch_file.csv or jfl_batch.csv)	No
systemd unit file	No — edit manually if paths changed
Log file	No

7.7 Future upgrades (1.0.17+)

When a newer release is published after 1.0.16:

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring
sudo systemctl restart inferyx-monitoring.service

Check What's new at the top of this README and on PyPI for version-specific steps.

---

Troubleshooting

Symptom	Fix
unrecognized arguments:	Run the test command as one line — no \ at end of lines
API error / empty response	PIPELINE_API_FILTER_BY_SCHEDULE_DATE=false; remove name= from PIPELINE_API_BASE_URL
Wrong batch in API URL	Base URL must not contain a fixed batch name
CSV errors / wrong batches	One batch per line; fix glued rows like Activebatch_...
No email	Check SMTP settings in .env
Skip a batch	Status=Suspended in CSV
Still using jfl_batch.csv	Set PIPELINE_CSV_FILE or --csv-file to the old path, or rename to batch_file.csv