promcsv

What it does

promcsv turns Prometheus metrics into CSV files for systems that do not speak Prometheus. It scrapes a configured set of /metrics endpoints on a fixed interval and writes one CSV file per target per cycle - target outputs are never combined. Files land atomically in a ready/ directory, are optionally gzip-compressed and uploaded to S3 or SFTP, and are cleaned up by a retention sweep so disk usage stays bounded.

It is built to run unattended for months as a systemd service, but it runs just as well in the foreground or from cron.

Supported features

• One CSV per target per cycle with a fixed 5-column schema (timestamp,metric,type,labels,value) that never changes shape
• Wall-clock-aligned scraping (a 5m interval fires at :00, :05, :10, ...), targets scraped concurrently, one failing target never affects the others
• Optional gzip compression (.csv.gz), applied once at write time
• Upload to S3 (including S3-compatible stores like MinIO; SSE/KMS and storage class supported) or SFTP (key or password auth, strict host-key verification), or no upload - consumers pick files up from ready/
• Flat or per-target remote layout, one config switch
• Atomic file handling everywhere - no partial files, local or remote
• Automatic retention cleanup; upload outages self-recover by draining the backlog once the endpoint returns
• status.json health file for external monitoring and alerting
• systemd-native: Type=notify watchdog, hardened unit, journald logging
• Strict config validation with did-you-mean hints (--validate-config)
• Small footprint: one RPM or wheel; only requests and PyYAML required (boto3/paramiko only for the upload target you actually use)

Build

Build the RPM (Rocky/RHEL 8+)

One-time build-host prerequisites:

dnf install rpm-build python3.12-devel python3.12-pip \
            python3.12-setuptools python3.12-wheel systemd
python3.12 -m pip install build

Then:

./packaging/build-rpm.sh

The artifacts are written to dist/: python3.12-promcsv-<version>-1.el8.noarch.rpm (install this) and the matching .src.rpm (for rebuilding on other EL releases).

Build the wheel

No prerequisites beyond Python 3.12 - no venv, no test dependencies:

python3.12 -m pip install build
python3.12 -m build --wheel

The artifact is written to dist/: promcsv-<version>-py3-none-any.whl - that single file is what you copy to the target host.

Install

Two supported paths: the RPM (recommended on Rocky/RHEL 8+) or the wheel (any Linux). Do not mix them on one host.

Install from RPM (Rocky/RHEL 8+, recommended)

dnf install ./python3.12-promcsv-<version>-1.el8.noarch.rpm
vi /etc/promcsv/config.yaml
promcsv -c /etc/promcsv/config.yaml --validate-config
systemctl enable --now promcsv

For S3 or SFTP upload, also install the extra package (it has no python3.12 RPM): python3.12 -m pip install boto3 (S3) or paramiko (SFTP).

Install from wheel (any Linux)

Prerequisite: Python 3.12 or newer on the host. On Rocky/RHEL 8:

dnf install python3.12 python3.12-pip

Copy the wheel over and install it as root (system-wide):

sudo python3.12 -m pip install promcsv-*.whl
# When upload.target is s3, also install boto3:
sudo python3.12 -m pip install boto3
# When upload.target is sftp, also install paramiko:
sudo python3.12 -m pip install paramiko

On hosts with a hardened root umask (0027/0077), pip creates the package directories without world-read, and other users then cannot import the package (ModuleNotFoundError: No module named 'promcsv'). Install with a safe umask:

sudo sh -c 'umask 022 && python3.12 -m pip install promcsv-*.whl'

Create the configuration from the built-in example and validate it (no git checkout needed - the files ship inside the package):

install -d -m 0755 /etc/promcsv
promcsv --print-config > /etc/promcsv/config.yaml    # then edit
promcsv -c /etc/promcsv/config.yaml --validate-config

At this point the tool is fully usable from the command line (promcsv -c /etc/promcsv/config.yaml, or --once from cron). To run it as a systemd service, complete the one-time setup below - the RPM path does all of this automatically.

Run as a systemd service (wheel installs only)

One-time host setup:

# service user and directories
useradd -r -s /sbin/nologin promcsv
install -d -o promcsv -g promcsv /var/data/promcsv
chgrp promcsv /etc/promcsv /etc/promcsv/config.yaml
chmod 0750 /etc/promcsv; chmod 0640 /etc/promcsv/config.yaml

# verify the service user can load the package (catches permission problems)
sudo -u promcsv /usr/bin/python3.12 -c "import promcsv"   # must print nothing

# unit, log rotation and credentials template
promcsv --print-unit > /usr/lib/systemd/system/promcsv.service
promcsv --print-logrotate > /etc/logrotate.d/promcsv
promcsv --print-env > /etc/promcsv/env && chmod 0600 /etc/promcsv/env
systemctl daemon-reload
systemctl enable --now promcsv

The printed unit's ExecStart= assumes /usr/local/bin/promcsv; if command -v promcsv shows a different path (e.g. a venv), edit ExecStart= in the installed unit accordingly.

CLI

promcsv -c /etc/promcsv/config.yaml [--validate-config | --once]

• -c, --config FILE - path to the YAML config (required).
• --validate-config - load and validate the config, print a report, and exit 0 (valid) or 2 (invalid). Read-only; touches nothing.
• --once - run a single scrape cycle and exit: 0 if at least one target was scraped successfully, 1 otherwise. Cron fallback / smoke test.
• --print-config - print the example configuration and exit.
• --print-unit - print the systemd unit and exit. The printed ExecStart= assumes /usr/local/bin/promcsv; if command -v promcsv shows a different path (e.g. a venv), edit ExecStart= accordingly.
• --print-logrotate - print the logrotate snippet and exit.
• --print-env - print the /etc/promcsv/env template and exit.
• --version - print version and exit.

The --print-* flags need no configuration file and are mutually exclusive with each other and with --once/--validate-config.

Exit codes:

Code	Meaning
0	clean shutdown / successful --once cycle
1	runtime fatal (lock held, stuck scrape threads, unhandled error)
2	configuration error

The unit sets RestartPreventExitStatus=2: a broken config exits 2 and is not restarted, so systemd does not loop on an error no retry can fix. Runtime failures exit 1 and are restarted after RestartSec.

CSV format (for consumers)

Every file starts with the header:

timestamp,metric,type,labels,value

Example:

timestamp,metric,type,labels,value
2026-07-21T10:15:00Z,connections_active,gauge,,412
2026-07-21T10:15:00Z,messages_total,counter,direction=tx;interface=eth0,908311
2026-07-21T10:15:00Z,request_duration_seconds_bucket,histogram,le=0.5,17734
2026-07-21T10:15:00Z,request_duration_seconds_sum,histogram,,5401.25
2026-07-21T10:15:00Z,setup_seconds,summary,quantile=0.99,0.087

Rules:

• timestamp - the cycle's scrape time in UTC, identical for every row in a file. Format per csv.timestamp_format: iso8601 (2026-07-21T10:15:00Z) or epoch (integer Unix seconds).
• metric - the sample name exactly as exposed (e.g. foo_bucket, foo_sum).
• type - from the endpoint's # TYPE metadata: counter, gauge, histogram, summary or untyped. Child samples (_bucket, _sum, _count, _created) inherit their family's type. Samples without TYPE metadata are untyped.
• labels - key=value;key=value, sorted by key; empty when the sample has no labels. Within label values, backslash, ; and = are backslash-escaped (\\, \;, \=).
• CSV quoting - standard RFC 4180: fields containing the delimiter, " or newlines are quoted, embedded quotes doubled. Consume the files with a real CSV parser, or configure a delimiter that cannot appear in your data.
• value - floats with an integral value are rendered as bare integers (up to 2^53 in magnitude); +Inf, -Inf and NaN are passed through literally.

Filenames are <label>_<YYYYMMDDTHHMMSSZ>.csv (UTC cycle time) - .csv.gz with compression on. On the rare name collision a _1, _2, ... suffix is inserted before the full extension (app_..._1.csv, app_..._1.csv.gz).

Compression

csv.compress: gzip writes .csv.gz files instead of .csv. The gzip stream is deterministic (mtime=0 in the header), so identical content produces identical bytes. Consumers use any gzip-capable reader (zcat, Python gzip, ...). Flipping the flag between restarts is safe - ready/ may transiently hold both extensions, and upload and retention handle both. The remote object keeps the exact local name and bytes; nothing is re-compressed or renamed on upload.

Directory lifecycle

output_dir (default layout under /var/data/promcsv):

output_dir/
├── tmp/         in-progress files (*.part); purged at startup
├── ready/       completed CSVs awaiting upload or pickup
├── uploaded/    CSVs successfully uploaded
├── status.json  machine-readable health, rewritten atomically each cycle
└── .lock        single-instance flock (never deleted)

Flow: each CSV is written to tmp/, fsynced, then atomically renamed into ready/ - readers never see a partial file. When an upload succeeds, the file moves from ready/ to uploaded/. The retention sweep deletes files older than retention from both ready/ and uploaded/; deletion from ready/ is the disk-safety bound (it is logged as a WARNING when an uploader is configured, since it means undelivered data was lost). With upload.target: none, ready/ is the hand-off point: consumers collect files from there themselves before retention removes them.

S3 upload support

Requires boto3 (sudo python3.12 -m pip install boto3). Config:

upload:
  target: s3
  s3:
    bucket: my-metrics-bucket
    prefix: metrics/                   # optional; object keys become <prefix><filename>
    region: eu-west-1
    # endpoint_url: http://minio:9000  # optional: non-AWS S3-compatible storage (region then optional)
    # sse: aws:kms                     # optional server-side encryption: AES256 | aws:kms
    # kms_key_id: <arn-or-id>          # optional; requires sse: aws:kms
    # storage_class: STANDARD_IA       # optional; passed through verbatim

Credentials never go in the YAML - the S3 client uses the standard boto3 credential chain: environment variables, ~/.aws/credentials, or an instance profile / IAM role. For environment variables, uncomment the placeholders in /etc/promcsv/env (installed by the RPM; on wheel installs create it with promcsv --print-env; keep it mode 0600, root-owned):

AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...

The unit already wires it in via EnvironmentFile=-/etc/promcsv/env (the - makes the file optional). For non-AWS S3-compatible storage (MinIO, Ceph RGW, ...), set upload.s3.endpoint_url; region then becomes optional.

The uploaded object keeps the exact local name and bytes (ContentType is text/csv, or application/gzip for compressed files). Pruning the remote side is the consumer's responsibility - at 6 targets on a 5m interval the prefix grows by ~1,700 files/day.

Retry semantics: connection failures, HTTP 5xx and throttling abort the upload pass and retry next cycle; per-file permanent errors (AccessDenied, invalid KMS key) are skipped and surface as permanent_failures_pending in status.json.

SFTP upload support

Requires paramiko (sudo python3.12 -m pip install paramiko). Config:

upload:
  target: sftp
  sftp:
    host: sftp.example.com
    port: 22                                  # optional, default 22
    username: metrics
    key_file: /etc/promcsv/sftp_key           # private key auth (recommended)
    remote_dir: /upload/metrics               # files land in <remote_dir>
    known_hosts: /etc/promcsv/known_hosts     # strict host-key verification (required)

Key setup (then install sftp_key.pub on the server):

ssh-keygen -t ed25519 -f /etc/promcsv/sftp_key -N '' -C promcsv
chown root:promcsv /etc/promcsv/sftp_key && chmod 0640 /etc/promcsv/sftp_key

Host key verification is strict; capture the server's key into known_hosts:

ssh-keyscan -p 22 sftp.example.com > /etc/promcsv/known_hosts

Warning - trust on first use. ssh-keyscan records whatever key answered at that moment. Verify the fingerprint out-of-band with the server owner (ssh-keygen -lf /etc/promcsv/known_hosts) before trusting it - strict verification is only as strong as this first capture.

Password auth alternative: omit key_file and uncomment PROMCSV_SFTP_PASSWORD=... in /etc/promcsv/env (installed by the RPM; on wheel installs create it with promcsv --print-env; mode 0600, root-owned; systemd's EnvironmentFile delivers it) - never in the YAML.

The uploaded file keeps the exact local name and bytes. Each upload is staged as <filename>.part and atomically renamed on completion, so the server never exposes a partial file. remote_dir: / is supported for chrooted SFTP accounts. Pruning the remote side is the consumer's responsibility (see the S3 section for the growth math).

Retry semantics: connection failures (host unreachable, authentication, host-key mismatch) abort the upload pass and retry next cycle; per-file permanent errors (remote permissions, quota) are skipped and surface as permanent_failures_pending in status.json.

Monitoring and alerting

<output_dir>/status.json is rewritten atomically after every cycle:

Field	Meaning
version	promcsv version
state	running or stopped; written running at startup and every cycle, stopped by a clean shutdown. A crash/SIGKILL leaves running behind - that is the signal that the process was NOT stopped properly
started_at	UTC start time of the current (or last) process
stopped_at	UTC time of the clean shutdown, null while running
stop_reason	null while running; else e.g. signal SIGTERM, once complete, fatal: ...
last_cycle	UTC timestamp of the last completed cycle (null before the first cycle of this process)
last_cycle_duration_seconds	wall time of that cycle
targets.<label>.last_success	UTC time of the target's last successful scrape (null if never)
targets.<label>.consecutive_failures	failed cycles in a row for this target (0 = healthy)
upload	null when upload.target is none, else:
upload.last_success	UTC time of the last successful upload
upload.consecutive_retryable_failures	upload passes in a row aborted by a retryable error (endpoint down, throttling)
upload.permanent_failures_pending	files in ready/ whose upload fails permanently (e.g. AccessDenied)
upload.last_failure	{kind, detail, at} of the most recent failed upload, or null; set on every failed upload, cleared on the next success
ready_backlog	CSV files currently waiting in ready/
uploaded_this_cycle	files uploaded during the last cycle

Staleness rule: define staleness as now - max(last_cycle, started_at) exceeding your threshold (treat a null last_cycle as absent) - a freshly restarted daemon has last_cycle: null for the whole first alignment wait, and a naive last_cycle-only check would page on every restart.

Lifecycle decision table: stale + state: running -> the daemon died or hung -> page. state: stopped -> deliberate shutdown (or between --once cron runs) -> suppress the staleness page; apply your "service down" policy instead. The file is never deleted - the last-known state is kept for forensics.

Suggested alerts: staleness per the rule above (daemon stalled or down), ready_backlog growing over time (uploads or consumers falling behind), any consecutive_failures or permanent_failures_pending above 0 for sustained periods.

Upload alerting rule: if upload.consecutive_retryable_failures keeps climbing and upload.last_failure.kind is one of AuthenticationException, BadHostKeyException (SFTP) or AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch (S3), page a human - these are credential, host-key or bucket-policy problems and never self-recover. Other retryable kinds are self-recovering outages (endpoint down, throttling); the backlog drains on its own once the endpoint returns.

Logging: journald by default (journalctl -u promcsv). Errors are logged on state change - first failure of a target/upload, then a reminder every 10th consecutive failing cycle, then a recovery line - plus exactly one INFO summary line per cycle (cycle: targets 6/6, rows 5321, written 6, uploaded 6, backlog 0, 1.2s).

Watchdog: the unit sets WatchdogSec=900. Watchdog pings are time-driven (sent on a fixed cadence during scraping, uploading and idle waits), so the 900-second deadline works with any scrape_interval - no tuning needed.

Operational notes and best practices

• Validate before every restart. The config is read once at startup; changes require systemctl restart promcsv. Run promcsv -c ... --validate-config first - a broken config exits 2 and the service stays down until fixed (deliberately, to avoid a restart loop).
• Smoke-test with --once. After installing or changing the config, one --once run against a scratch output_dir shows exactly what will be produced, without touching the service.
• Single instance per output_dir. An exclusive lock on <output_dir>/.lock guards the directory. Running promcsv --once beside the daemon exits 1 with a clear message - by design; point it at a different output_dir if you need a parallel run.
• Target isolation. Targets are scraped concurrently; one target being down (or slow, or hung) never affects the others' CSVs. A scrape thread stuck beyond the timeout is abandoned; if stuck threads persist, the daemon exits 1 for a clean systemd restart.
• Upload outages self-recover. Files stay in ready/ and the backlog drains automatically once the endpoint returns - time-boxed to half the scrape interval per cycle, in filename order (per-target oldest first), so scraping cadence is never starved.
• Permanent upload errors (S3: AccessDenied, invalid KMS key; SFTP: remote permissions, quota) are skipped and retried each cycle until retention removes the file; they are visible as upload.permanent_failures_pending in status.json and in the logs.
• Upgrade the RPM, never reinstall it. dnf upgrade preserves your edited /etc/promcsv/config.yaml and env (they are %config(noreplace)), but dnf reinstall of the same version resets them to the packaged defaults and moves your files to .rpmsave - standard RPM behavior, easy to trip over.

Development

Running the tests requires a one-time setup (a virtualenv with the package and the test tools):

python3.12 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"

Then:

pytest -q -m "not slow"      # default run (fast, skips the soak test)
pytest -q                    # everything, including the slow soak test
pytest -q -m "not slow" --cov=promcsv --cov-branch --cov-report=term-missing   # with coverage

To build the RPM or the wheel, see the Build section - neither needs the virtualenv or the test dependencies.