inferyx-monitoring 1.0.9

Monitor batch pipelines via API and send email alerts for failures and missed runs

Pipeline Batch Monitor

Monitor batch jobs from CSV, check status via API, and email alerts for failures, missed runs, long-running jobs, and missing API data.

Version: see VERSION file.

Quick start (development)

cp .env.example .env          # edit with real SMTP/API credentials
./pipeline-monitor setup      # create .venv + dependencies
./pipeline-monitor validate   # check .env, CSV, Python
./pipeline-monitor run-once   # single test cycle
./pipeline-monitor run        # continuous local run

Same commands via Make: make setup, make validate, make run-once, make build.

Package workflow (build → deploy → use)

One CLI drives the full lifecycle:

./pipeline-monitor <command>

Command	Purpose
setup	Create .venv, install requirements.txt
validate	Validate .env, jfl_batch.csv, syntax
run	Run locally (foreground, continuous)
run-once	One monitoring cycle (cron/debug)
build	Create dist/pipeline-status-mailer-<version>.tar.gz + dist/latest.tar.gz
install	Production install from source (+ systemd, needs sudo)
deploy	Install from tarball (needs sudo)
status	systemctl status
logs	Follow journal or app log

1. Build release package

On your build machine (with .env configured for validation only — secrets are not bundled by default):

./pipeline-monitor validate
./pipeline-monitor build

Output:

• dist/pipeline-status-mailer-1.0.0.tar.gz
• dist/latest.tar.gz (copy of latest build)
• MANIFEST.txt inside the package (file list + SHA256)

Private pipeline (bundle .env — handle tarball as secret):

./pipeline-monitor build --with-env

2. Copy to target server

scp dist/latest.tar.gz user@prod-server:/tmp/

3. Deploy on target server

ssh user@prod-server
tar -xzf /tmp/latest.tar.gz -C /tmp
cd /tmp/pipeline-status-mailer-*
cp .env.example .env && vi .env
sudo ./pipeline-monitor install --user inferyx --group inferyx

One-step deploy (from dev machine with .env present, copies .env into package during deploy):

sudo ./pipeline-monitor deploy dist/latest.tar.gz --user inferyx --group inferyx

On server after install:

sudo ./pipeline-monitor status
sudo ./pipeline-monitor logs
/opt/pipeline-status-mailer/pipeline-monitor run-once

See package/INSTALL.md inside the release tarball for server-side notes.

Default production paths

Item	Path
Application	/opt/pipeline-status-mailer
Python venv	/home/inferyx/monitoring
Config	/opt/pipeline-status-mailer/.env
Batches	/opt/pipeline-status-mailer/jfl_batch.csv
systemd unit	pipeline-status-mailer.service

Batch CSV (jfl_batch.csv)

Column	Required	Description
Name	Yes	Batch name for API
Frequency	Yes	Daily, Hourly, Monthly, Continuous, …
ExpectedStartTime	Scheduled	e.g. 9:30, 7:20:00, or 09:00,12:00
AvgExecutionTime	Recommended	e.g. 10 mins, 1 Hr
ExpectedDayOfMonth	Monthly	Day 1–31 (default 1)
Status	Optional	Active (monitor) or Suspended (skip). Defaults to Active if omitted

Monthly example:

batch_monthly_finance_close,Monthly,9:30,10 mins,1

Configuration

Required in .env (see .env.example):

• SMTP: PIPELINE_SMTP_*, PIPELINE_MAIL_TO, PIPELINE_FROM_NAME
• API: PIPELINE_API_BASE_URL, PIPELINE_API_TOKEN, PIPELINE_API_TOKEN_HEADER
• PIPELINE_DEVOPS_EMAIL

Optional tuning:

Variable	Default	Meaning
PIPELINE_CHECK_INTERVAL	60	Seconds between poll cycles
PIPELINE_SCHEDULE_GRACE_MINUTES	5	Delay after expected start before missed/no-data alerts; delay after expected end before long-running alerts. Polling starts at expected start.
PIPELINE_POST_RUN_GRACE_MINUTES	60	How long after expected end the monitor keeps checking that batch

Email templates (.env)

Subjects, bodies, and signature are configured via .env (see .env.example). Omit any variable to keep the built-in default.

Variable	Alert type
PIPELINE_MAIL_SUBJECT_*	DEFAULT, NO_DATA, FAILED, RUNNING, MISSED
PIPELINE_MAIL_BODY_*	Same keys as subjects
PIPELINE_MAIL_SIGNATURE	HTML footer appended to every alert (or use {signature} in a body template)

Placeholders: {batch_name}, {issue_type}, {issue_type_upper}, {status}, {error}, {error_line}, {expected_start_time}, {expected_end_time}, {frequency}, {avg_time}, {current_time}, {signature}

Project layout

Monitor/
  pipeline-monitor      # Main CLI (build / deploy / run)
  pipeline_status_mailer.py
  config.py
  jfl_batch.csv
  .env.example
  VERSION
  Makefile
  scripts/
    lib/common.sh
    build_release.sh
    deploy_release.sh
    install_pipeline_status_mailer.sh
    validate_env.sh
    setup_python_env.sh
    run_with_venv.sh
  package/INSTALL.md
  dist/                 # Created by build

Publish to PyPI (pip install)

Package name: inferyx-monitoring

Guide	Audience
docs/SERVER_DEPLOYMENT.md	Deploy on target server from scratch (pip install + systemd)
docs/PYPI_PUBLISHING.md	Build and upload to PyPI

Quick publish:

pip install build twine
make pip-build          # creates dist/*.whl and dist/*.tar.gz
make pip-check
make pip-test-upload    # optional: TestPyPI first
make pip-upload         # production PyPI (needs token)

Install on a production server: see docs/SERVER_DEPLOYMENT.md.

pip upgrade never changes your live config

File	Shipped in pip package	On pip upgrade
.env.example	Yes (template)	Updated in package only
jfl_batch.csv.example	Yes (template)	Updated in package only
.env	No (you create locally)	Never overwritten
jfl_batch.csv	No (you create locally)	Never overwritten

Upgrade code only:

pip install --upgrade inferyx-monitoring
# .env and jfl_batch.csv in /opt/pipeline-monitor stay as-is
sudo systemctl restart inferyx-monitoring.service

Makefile shortcuts

make setup validate build
make install    # sudo install from source
make deploy     # sudo deploy dist/latest.tar.gz
make status logs clean
make pip-build pip-upload   # PyPI release