provost

Turn command output into data. Run a command, and provost captures its output, auto-detects the format, and files it as a tidy, one-row-per-record dataset in a git-backed store — every capture a commit, every row traceable to its source.

Beta. provost is the simplified, single-tool successor to the three-binary Envoy suite — see docs/0001-rename-and-simplification.md.

Install

pip install provost

The PyPI package (provost) bundles the complete tool — the provost entry point plus its extract/analyze/transform engines — and a thin Python launcher execs it in place. Requires bash and git.

Quick start

provost doctor                   # check engines + store

free -m | provost                # capture stdin → auto-detect → tidy dataset 'free'
provost run df -h                # capture a command's output → dataset 'df'
provost ls                       # list datasets
provost show df                  # present a dataset (leading #N = row index)
provost source 5                 # the original input behind row #5
provost stats df                 # statistics on the dataset

What it does

run/inject  →  auto-detect (generic parsers + hint rules)  →  tidy melt  →  git-backed store

• Auto-capture format — ~110 generic format parsers (delimited and structured), a declarative hints table, and ~145 tool-specific strategies recognise JSON, tables, logs, and system-tool output (df, ps aux, free, mount, docker, …).
• Named datasets — provost run CMD files into a dataset named after the command (looking through sudo, env, timeout, …); piped input is named after the parser that recognised it; --name NAME overrides either.
• Present a dataset — a wide entry<N>.field extract is melted to one row per record; same-typed captures reconcile into one table.
• Git core — the store is a git repo; every capture is a commit tagged capture-<id>. provost log lists them, provost restore <id> brings one back up.
• Provenance — every dataset row has a globally-unique visible #N; provost source N shows the exact captured input that produced it.
• Global or per-directory — captures go to a global store (~/.provost) by default; provost control <dir> makes a directory its own instance.
• Portable — provost export bundles the store; provost import reopens it elsewhere. Datasets, history, and provenance all survive the move.
• Analyze — provost stats <dataset> runs the vendored analyze engine.

Command reference

Every dataset row carries # (its number in the store), capture (the capture it came from, e.g. c0007) and, for records, n (its place in that capture).

Layout

provost/
├── provost               # the single entry point
├── extract/            # extract engine, 268 strategies (~116 generic), hints.tsv
├── analyze/            # analyze engine
├── transform/          # tidy-melt transform
├── lib/                # shared suite libraries
├── completions/, man/  # bash completion, man page
└── tests/              # smoke + unit tests

Standalone repository

This tree is fully self-contained: provost resolves its engines from its own directory, so it runs with no external dependencies. The extract/, analyze/, transform/, and lib/ trees were originally imported from the Envoy suite but are now first-party provost code, maintained here directly — there is no live tie to upstream Envoy. Add extract strategies under extract/strategies/ and edit the engines in place; run each tree's own test suite to keep it green.

Integrating from another tool

Wrappers that pipe into provost (e.g. deframe) get two hooks:

• --plugin-dir <dir> (or PROVOST_PLUGIN_DIR): an extra strategy directory that competes in format detection for that capture — no symlink into the engine's ~/.config/provost/strategies user dir needed. The winning plugin dir is recorded in the capture's metadata.
• #% metadata prologue: a leading run of #% lines on injected input is capture context, not data. It is peeled off before detection (so it can never corrupt the generic parsers), stored with the capture as context.txt, and provost source still shows the original input verbatim. A #% line after the first data line is data and passes through untouched — so context no longer has to travel out-of-band in a run manifest. With --context-columns (or PROVOST_CONTEXT_COLUMNS=1) each key=value word in it is also put on every row the capture files, after # and capture: #% host=web03 rack=r2 gives each row host and rack columns. A data column of the same name keeps it and the context one becomes _host (the rule capture already follows), and the capture's meta.tsv records which columns came from the context (context_columns host=_host,rack=rack); a capture held in unsorted is kept as it came.
• provost batch FILE... [--from LIST]: many inputs, one commit. Each file is a capture of its own — its own id, its own provost source, its own capture-<id> tag — but they are parsed in parallel (-j N, default the CPUs, at most 16) and filed under one hold of the store lock, so capturing a 50-host fleet costs one commit rather than fifty serialised ones. Ids follow the order the files were named. --name NAME applies to the files after it; --from LIST reads PATH or NAME<TAB>PATH lines. A fleet wrapper writes one file per host opening with #% host=NAME and passes --context-columns, and every row says which host it came from.

The store itself is plain files (TSV datasets, per-capture meta.tsv + output.txt, provenance.tsv), so other tools can browse it read-only without provost installed. Meridian Commander ships a Provost data plug-in that does exactly that: datasets as scrollable tables, the capture log, and each row's original input via its #N provenance index — including stores on remote (SFTP/SSH) hosts.

Environment

Tests

bash tests/smoke.sh                  # end-to-end
bash transform/tests/test_melt.sh    # transform unit tests
bash extract/tests/run_tests.sh      # extract strategies + engine
bash analyze/tests/run_tests.sh      # analyze engine
bash tests/corpus/run.sh --golden    # tidy output of ~3,500 real tools vs goldens
python -m pytest                     # PyPI launcher (100% coverage enforced)

All six run in CI (.github/workflows/ci.yml).

License

Copyright (C) 2026 Martin J. Gallagher

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

Full text: the LICENSE file at the repository root, or https://www.gnu.org/licenses/gpl-3.0.html — SPDX identifier GPL-3.0-or-later. This repository follows the REUSE Specification: source files carry SPDX copyright and licence headers, and the licence text lives in LICENSES/.