inferyx-monitoring 1.0.51

Monitor batch pipelines via API and email alerts — install and deploy on Linux servers from PyPI

inferyx-monitoring

Version: 1.0.50 · PyPI: inferyx-monitoring

Batch monitor + admin UI. One package, one service, one install directory (/opt/pipeline-monitor).

---

Quick reference

Task	Command
Build (maintainer)	./scripts/build.sh
Publish (maintainer)	./scripts/build.sh --release yes --publish
Install (server)	pip install → release.sh install --skip-pip
Upgrade (server)	pip install --upgrade → release.sh upgrade --skip-pip

Browser URL: https://<host>/monitoring/admin/ (nginx required)

---

Project layout

Monitor/
├── README.md
├── CHANGELOG.md
├── VERSION
├── pipeline_status_mailer.py         ← monitor + alerts
├── auth.policy                       ← gitignored (same file local + server)
├── .env                              ← gitignored
├── batch_file.csv                    ← gitignored
├── secrets/                          ← gitignored (session_secret, etc.)
├── scripts/build.sh                  ← build UI + wheel (not in PyPI)
├── release/
│   ├── release.sh                    ← server install | upgrade
│   └── *.example                     ← auth.policy, nginx, systemd templates
├── admin-ui/                         ← Angular source → admin_static in wheel
└── inferyx_pipeline_monitor/         ← Python package
    ├── settings.py                   ← loads .env
    ├── admin/                        ← Admin API
    └── data/
        ├── .env.example              ← .env template
        └── batch_file.csv.example    ← CSV template

Config files (one set for local and server)

What	Template (in git)	Runtime (gitignored)
SMTP, API, alerts	inferyx_pipeline_monitor/data/.env.example	.env
Batch schedules	inferyx_pipeline_monitor/data/batch_file.csv.example	batch_file.csv
Admin UI / OAuth	release/auth.policy.example	auth.policy

On a server, paths in auth.policy point to /opt/pipeline-monitor/... (default in the example).
For local dev, copy the same template and use relative paths (see below).

There is no config/ folder and no development/ folder. Settings load from .env via inferyx_pipeline_monitor/settings.py.

---

Build (maintainer)

./scripts/build.sh
./scripts/build.sh --release yes --publish
make clean

---

Install (server)

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install inferyx-monitoring==1.0.50
sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh install --skip-pip

Edit /opt/pipeline-monitor/.env and /opt/pipeline-monitor/auth.policy.

---

Upgrade (server)

sudo -u inferyx /opt/pipeline-monitor/.venv/bin/pip install --upgrade inferyx-monitoring==1.0.50
sudo bash /opt/pipeline-monitor/.venv/share/inferyx-monitoring/release.sh upgrade --skip-pip

---

Local development

Same config files as production — only paths and URLs differ.

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

cp inferyx_pipeline_monitor/data/.env.example .env
cp inferyx_pipeline_monitor/data/batch_file.csv.example batch_file.csv
cp release/auth.policy.example auth.policy
mkdir -p secrets logs
echo "change-me" > secrets/session_secret

# Edit auth.policy for local:
#   paths.work_dir = "."
#   paths.env_file = ".env"
#   paths.csv_file = "batch_file.csv"
#   paths.session_secret_path = "secrets/session_secret"
#   ui.public_base_url = "http://localhost:4200"
#   auth.google.client_secret_path = "<inline secret or file path>"
#   auth.google.redirect_uri = "http://localhost:4200/api/auth/callback/google"

export PIPELINE_ADMIN_POLICY=auth.policy
./scripts/build.sh
inferyx-monitoring-service --work-dir . --with-admin --admin-port 8091

Admin UI hot reload: cd admin-ui && npm ci && npm start → http://localhost:4200/

---

Batch CSV — scheduled vs runtime / on-demand

Required columns: Name, Frequency.
Optional: ExpectedStartTime, AvgExecutionTime, ExpectedDayOfMonth, Status (Active / Suspended).

Fixed schedule (Daily, Hourly, Weekly, …)

Use when the batch runs at a known time:

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
my_daily_job,Daily,9:00:00,"30 mins",,Active
my_monthly_job,Monthly,2:00:00,"1 hr",15,Active

Alerts: missed (did not start on time), failed, stuck/running (past expected end + grace), no_data (API error).

Runtime / on-demand (no fixed time)

Use when the batch is triggered manually or by an external event — not on a fixed clock:

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
user_triggered_job,OnDemand,,"30 mins",,Active

Frequency must be one of: OnDemand, Adhoc, Manual, Unscheduled, Anytime (case-insensitive).

Leave ExpectedStartTime and ExpectedDayOfMonth empty.
Set AvgExecutionTime if you want stuck alerts (e.g. "30 mins" — alert if still running 30 min + grace after actual start from API).

Alert	Applies?	Why
Failed	Yes	API status = failed
no_data	Yes	API/token/network error
Stuck / running too long	Only if AvgExecutionTime is set	Uses API actual start + avg time + grace (checked in END window)
Running (immediate)	No	Normal while batch is executing
Missed	No	No schedule to miss

Polling (default PIPELINE_CHECK_MODE=schedule_windows): OnDemand is not polled every loop.

Phase	When API is called
Idle (no run)	Discovery poll every PIPELINE_CHECK_WINDOW_MINUTES (default 10 min)
Run detected	START window: actual start + grace → + CHECK_WINDOW_MINUTES
Run active	END window: expected end + grace → + CHECK_WINDOW_MINUTES (stuck check)
Between windows	No API call
After run finished	Back to idle discovery

Set PIPELINE_CHECK_MODE=full_window to poll OnDemand continuously (old behaviour).

Tip: Set Status=Suspended for batches you are not monitoring right now.

Always-on / streaming jobs

For jobs that should run continuously (not slot-based):

Name,Frequency,ExpectedStartTime,AvgExecutionTime,ExpectedDayOfMonth,Status
stream_processor,Continuous,,,,Active

Frequency: Continuous, Realtime, Always, etc.
Alerts: failed and no_data only (no missed / schedule-based stuck).

---

Troubleshooting

Symptom	Fix
No such file ... release.sh	Run pip install inferyx-monitoring first
nginx 404 on /monitoring/admin/	Use release/nginx.conf.example
Admin UI "Unavailable"	curl http://127.0.0.1:8090/api/ui/status
Service restart loop	Check .env — journalctl -u inferyx-monitoring -n 50
Missed vs no_data	Empty API after schedule → missed; API error → no_data

See CHANGELOG.md.

---

Scheduling / tuning (.env)

When to poll

Variable	Default	Used when	Meaning
PIPELINE_CHECK_MODE	schedule_windows	Always	schedule_windows — poll only in short windows after expected start + grace, and after expected end + grace. full_window — poll continuously from expected start until the slot ends.
PIPELINE_CHECK_INTERVAL	60	Always	Seconds between monitor loop runs.
PIPELINE_CHECK_WINDOW_MINUTES	10	schedule_windows only	Length of each poll window. Example: start 09:00 + 5 min grace → poll 09:05–09:15.
PIPELINE_POST_RUN_GRACE_MINUTES	60	full_window only	How long to keep polling after expected end. Not used in schedule_windows.

When to alert

Variable	Default	Meaning
PIPELINE_SCHEDULE_GRACE_MINUTES	5	Minutes after expected start before a missed alert, and after expected end before a stuck/running alert. Example: start 09:00 → missed earliest 09:05.

API retries

Variable	Default	Meaning
PIPELINE_API_RETRY_COUNT	3	Retries per batch per loop when the API call fails.
PIPELINE_API_RETRY_BACKOFF_SEC	5	Seconds between API retries.

Email alert frequency (hourly, daily, weekly — all schedules)

Variable	Default	Used when	Meaning
PIPELINE_ALERT_ONCE_PER_DAY_ALL_SCENARIOS	true	Always	At most one new email per issue type (missed, failed, running, no_data) per batch per day.
PIPELINE_ALERT_REPLY_TO_THREAD	false	Optional	When true, first alert opens a new thread; later alerts for the same ongoing issue are reply-all (same conversation). Applies to all frequencies.
PIPELINE_ALERT_COOLDOWN_MINUTES	60	REPLY=true or ALL_SCENARIOS=false	Minimum minutes between repeat alerts. Ignored with defaults (schedule_windows + once-per-day + no reply) — once-per-day already prevents repeats. Needed for reply-all follow-ups across slots (e.g. hourly) or when once-per-day is off.
PIPELINE_FAILED_ALERT_ONCE_PER_DAY	true	ALL_SCENARIOS=false only	Failed-only once-per-day. Ignored when ALL_SCENARIOS=true (default).

Typical flow with defaults (schedule_windows): batch expected 09:00 → poll only 09:05–09:15 and after expected end → grace 5 min → if still not running, missed alert (one email per day) → if still running past expected end + 5 min grace, stuck alert.

Minimal config for most users: keep defaults; only set PIPELINE_ALERT_REPLY_TO_THREAD=true if you want follow-up reply-all emails instead of silence after the first alert.

Test reply-all threading (Office365/Gmail):

inferyx-monitoring --test-mail-thread
# or: python3 pipeline_status_mailer.py --test-mail-thread

Sends two emails: a new alert, then a reply-all follow-up. Both should appear in one conversation in inbox and Sent.