promcsv

Turn Prometheus metrics into CSV files - and deliver them to S3 or SFTP.

pip install promcsv

What it does

promcsv turns Prometheus metrics into CSV files for systems that do not speak Prometheus. It scrapes a configured set of /metrics endpoints on a fixed interval and writes one CSV file per target per cycle - target outputs are never combined. Files land atomically in a ready/ directory, are optionally gzip-compressed and uploaded to S3 or SFTP, and are cleaned up by a retention sweep so disk usage stays bounded.

It is built to run unattended for months as a systemd service, but it runs just as well in the foreground or from cron.

Supported features

• One CSV per target per cycle with a fixed 5-column schema (timestamp,metric,type,labels,value) that never changes shape
• Wall-clock-aligned scraping (a 5m interval fires at :00, :05, :10, ...), targets scraped concurrently, one failing target never affects the others
• Optional gzip compression (.csv.gz), applied once at write time
• Upload to S3 (including S3-compatible stores like MinIO; SSE/KMS and storage class supported) or SFTP (key or password auth, strict host-key verification), or no upload - consumers pick files up from ready/
• Flat or per-target remote layout, one config switch
• Atomic file handling everywhere - no partial files, local or remote
• Automatic retention cleanup; upload outages self-recover by draining the backlog once the endpoint returns
• status.json health file for external monitoring and alerting
• systemd-native: Type=notify watchdog, hardened unit, journald logging
• Strict config validation with did-you-mean hints (--validate-config)
• Small footprint: one RPM or wheel; only requests and PyYAML required (boto3/paramiko only for the upload target you actually use)

Install

Two options - pick one, do not mix them on one host:

• Option 1 - RPM (Rocky/RHEL 8+): recommended for servers; one command installs everything, including the systemd service.
• Option 2 - pip (any Linux): from PyPI or a wheel file; systemd setup is a short manual step.

Option 1 - install from RPM (Rocky/RHEL 8+, recommended)

Download the RPM from the latest release, then:

dnf install ./python3.12-promcsv-<version>-1.el8.noarch.rpm
vi /etc/promcsv/config.yaml
promcsv -c /etc/promcsv/config.yaml --validate-config
systemctl enable --now promcsv

For S3 or SFTP upload, also install the extra package (it has no python3.12 RPM): python3.12 -m pip install boto3 (S3) or paramiko (SFTP).

Option 2 - install with pip (any Linux)

Prerequisite: Python 3.12 or newer. On Rocky/RHEL 8:

dnf install python3.12 python3.12-pip

Install as root (system-wide), straight from PyPI:

sudo python3.12 -m pip install promcsv
# S3 / SFTP upload extras:
sudo python3.12 -m pip install boto3      # when upload.target is s3
sudo python3.12 -m pip install paramiko   # when upload.target is sftp

Offline hosts: download the wheel from the latest release and sudo python3.12 -m pip install ./promcsv-<version>-py3-none-any.whl.

Note: on hosts with a hardened root umask (0027/0077) pip creates unreadable package directories and other users then get ModuleNotFoundError - install with sudo sh -c 'umask 022 && python3.12 -m pip install promcsv'.

Create the configuration from the built-in example and validate it:

install -d -m 0755 /etc/promcsv
promcsv --print-config > /etc/promcsv/config.yaml    # then edit
promcsv -c /etc/promcsv/config.yaml --validate-config

The tool is now fully usable from the command line (promcsv -c /etc/promcsv/config.yaml, or --once from cron). To run it as a systemd service, complete the one-time setup below - the RPM path does all of this automatically.

Run as a systemd service (pip installs)

One-time host setup:

# service user and directories
useradd -r -s /sbin/nologin promcsv
install -d -o promcsv -g promcsv /var/data/promcsv
chgrp promcsv /etc/promcsv /etc/promcsv/config.yaml
chmod 0750 /etc/promcsv; chmod 0640 /etc/promcsv/config.yaml

# verify the service user can load the package (catches permission problems)
sudo -u promcsv /usr/bin/python3.12 -c "import promcsv"   # must print nothing

# unit, log rotation and credentials template
promcsv --print-unit > /usr/lib/systemd/system/promcsv.service
promcsv --print-logrotate > /etc/logrotate.d/promcsv
promcsv --print-env > /etc/promcsv/env && chmod 0600 /etc/promcsv/env
systemctl daemon-reload
systemctl enable --now promcsv

The printed unit's ExecStart= assumes /usr/local/bin/promcsv; if command -v promcsv shows a different path (e.g. a venv), edit ExecStart= in the installed unit accordingly.

CLI

promcsv -c /etc/promcsv/config.yaml [--validate-config | --once]

• -c, --config FILE - path to the YAML config (required).
• --validate-config - load and validate the config, print a report, and exit 0 (valid) or 2 (invalid). Read-only; touches nothing.
• --once - run a single scrape cycle and exit: 0 if at least one target was scraped successfully, 1 otherwise. Cron fallback / smoke test.
• --print-config - print the example configuration and exit.
• --print-unit - print the systemd unit and exit. The printed ExecStart= assumes /usr/local/bin/promcsv; if command -v promcsv shows a different path (e.g. a venv), edit ExecStart= accordingly.
• --print-logrotate - print the logrotate snippet and exit.
• --print-env - print the /etc/promcsv/env template and exit.
• --version - print version and exit.

The --print-* flags need no configuration file and are mutually exclusive with each other and with --once/--validate-config.

Exit codes:

The unit sets RestartPreventExitStatus=2: a broken config exits 2 and is not restarted, so systemd does not loop on an error no retry can fix. Runtime failures exit 1 and are restarted after RestartSec.

Documentation

• Usage guide - the CSV format specification for consumers, the directory lifecycle, S3 and SFTP upload setup, monitoring and alerting, and operational best practices.
• Annotated example configuration - every parameter documented in place (also available via promcsv --print-config).

Build

Build the RPM (Rocky/RHEL 8+)

One-time build-host prerequisites:

dnf install rpm-build python3.12-devel python3.12-pip \
            python3.12-setuptools python3.12-wheel systemd
python3.12 -m pip install build

Then:

./packaging/build-rpm.sh

The artifacts are written to dist/: python3.12-promcsv-<version>-1.el8.noarch.rpm (install this) and the matching .src.rpm (for rebuilding on other EL releases).

Build the wheel

No prerequisites beyond Python 3.12 - no venv, no test dependencies:

python3.12 -m pip install build
python3.12 -m build --wheel

The artifact is written to dist/: promcsv-<version>-py3-none-any.whl - that single file is what you copy to the target host.

Development

Running the tests requires a one-time setup (a virtualenv with the package and the test tools):

python3.12 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"

Then:

pytest -q -m "not slow"      # default run (fast, skips the soak test)
pytest -q                    # everything, including the slow soak test
pytest -q -m "not slow" --cov=promcsv --cov-branch --cov-report=term-missing   # with coverage

To build the RPM or the wheel, see the Build section - neither needs the virtualenv or the test dependencies.

Releasing

Releases are fully automated by GitHub Actions on tag push:

# 1. bump the version in pyproject.toml, src/promcsv/__init__.py and the RPM
#    spec (new changelog entry), and update the two tag-pinned README links -
#    the test suite fails if any of these is missed
# 2. then:
git commit -am "promcsv vX.Y.Z: ..."
git tag vX.Y.Z
git push origin main vX.Y.Z

The release workflow verifies the tag (matches the version, commit is on main), runs the full test suite, builds the wheel/sdist and the RPM (in a Rocky Linux 8 container), publishes to PyPI via Trusted Publishing (after the required environment approval), and creates the GitHub release with the wheel and RPM attached. The manual Build instructions above remain as a fallback.