provost

Turn command output into data. Run a command, and provost captures its output, auto-detects the format, and files it as a tidy, one-row-per-record dataset in a git-backed store — every capture a commit, every row traceable to its source.

Prototype (proposal 0001) — the simplified successor to the three-binary Envoy suite, as one tool. See docs/0001-rename-and-simplification.md.

Install

pip install provost

The PyPI package (provost) bundles the complete tool — the provost entry point plus its extract/analyze/transform engines — and a thin Python launcher execs it in place. Requires bash and git.

Quick start

provost doctor                   # check engines + store

free -m | provost                # capture stdin → auto-detect → tidy dataset
provost run df -h                # capture a command's output as a commit
provost ls                       # list datasets
provost show df                  # present a dataset (leading #N = row index)
provost source 5                 # the original input behind row #5
provost stats df                 # statistics on the dataset

What it does

run/inject  →  auto-detect (generic parsers + hint rules)  →  tidy melt  →  git-backed store

• Auto-capture format — ~29 generic format parsers (plus a declarative hints table and a tail of bespoke strategies) recognise JSON, tables, logs, and system-tool output (df, ps aux, free, mount, docker, …).
• Present a dataset — a wide entry<N>.field extract is melted to one row per record; same-typed captures reconcile into one table.
• Git core — the store is a git repo; every capture is a commit tagged capture-<id>. provost log lists them, provost restore <id> brings one back up.
• Provenance — every dataset row has a globally-unique visible #N; provost source N shows the exact captured input that produced it.
• Global or per-directory — captures go to a global store (~/.provost) by default; provost control <dir> makes a directory its own instance.
• Portable — provost export bundles the store; provost import reopens it elsewhere. Datasets, history, and provenance all survive the move.
• Analyze — provost stats <dataset> runs the vendored analyze engine.

Command reference

Layout

provost/
├── provost               # the single entry point
├── extract/            # extract engine, ~40 generic + ~130 bespoke strategies, hints.tsv
├── analyze/            # analyze engine
├── transform/          # tidy-melt transform
├── lib/                # shared suite libraries
├── completions/, man/  # bash completion, man page
└── tests/              # smoke + unit tests

Standalone repository

This tree is fully self-contained: provost resolves its engines from its own directory, so it runs with no external dependencies. The extract/, analyze/, transform/, and lib/ trees were originally imported from the Envoy suite but are now first-party provost code, maintained here directly — there is no live tie to upstream Envoy. Add extract strategies under extract/strategies/ and edit the engines in place; run each tree's own test suite to keep it green.

Integrating from another tool

Wrappers that pipe into provost (e.g. deframe) get two hooks:

• --plugin-dir <dir> (or PROVOST_PLUGIN_DIR): an extra strategy directory that competes in format detection for that capture — no symlink into the engine's ~/.mfe/strategies user dir needed. The winning plugin dir is recorded in the capture's metadata.
• #% metadata prologue: a leading run of #% lines on injected input is capture context, not data. It is peeled off before detection (so it can never corrupt the generic parsers), stored with the capture as context.txt, and provost source still shows the original input verbatim. A #% line after the first data line is data and passes through untouched — so context no longer has to travel out-of-band in a run manifest.

Environment

Tests

bash provost/tests/smoke.sh              # end-to-end (58 checks)
bash provost/transform/tests/test_melt.sh  # transform unit tests (10 checks)
python -m pytest                       # PyPI launcher (100% coverage enforced)