perfreport

Python performance benchmarking framework. Runs workers as subprocesses, collects metrics via a lightweight text protocol, and renders HTML/Markdown reports with interactive SVG charts.

Features

• Language-agnostic workers — a worker is any process that writes protocol lines to stdout. Python, C, Go, bash, whatever.
• Subprocess-isolated — each scenario runs as its own process; the runner collects stdout, parses the protocol, aggregates results.
• Hierarchical reports — nested tree of test scenarios with cascading defaults.
• Interactive SVG charts — pygal-based line/bar charts embedded as <object> in HTML.
• Matrix testing — run the same benchmark against multiple Python versions and library versions (uv-based provisioning).
• Configurable metrics — built-in checkpoints/summary + custom per-driver metrics.
• Multiple output modes — interactive (TUI with prompts), verbose (streaming stdout), silent (file-only).

Install

pip install perfreport

For matrix testing you also need uv in your PATH:

pip install uv
# or via pipx: pipx install uv

Quick start

A report is a Python module that exposes two functions: get_args(parser) and get_report(options) -> Report. Workers are Python modules invoked as subprocesses.

1. A minimal report

# my_bench/report.py
from perfreport.models import (
    Report, ReportDefaults, WorkerModel, WorkerArgs,
    OperationModel, OpArgs, ChartModel,
)


def get_args(parser):
    group = parser.add_argument_group('My benchmark')
    group.add_argument('-H', '--hosts', default='127.0.0.1')


def get_report(options):
    worker = 'my_bench.worker'
    return Report(
        name='My benchmark',
        description='Tests how fast my_lib does X',
        default=ReportDefaults(
            worker=worker,
            options={
                'hosts': options.hosts,
                'concurrency': options.concurrency,
                'num_queries': options.num_queries,
            },
        ),
        charts=[
            ChartModel(title='Requests per second', x_key='req', y_key='rps',
                       xlabel='Requests', ylabel='RPS'),
        ],
        items=[
            Report(
                name='Scenario A',
                worker=WorkerModel(
                    args=WorkerArgs(operations=[
                        OperationModel(key='run', name='Run', args=[
                            OpArgs(key='mode', value='fast', syntax='python'),
                        ]),
                    ]),
                ),
            ),
        ],
    )

Reports that want the conventional -n/--num-queries and -c/--concurrency flags can plug them in via a helper:

# my_bench/report.py — inside get_args:
from perfreport.args import add_common_workload_args

def get_args(parser):
    group = parser.add_argument_group('My benchmark')
    add_common_workload_args(group)  # adds -n and -c
    group.add_argument('-H', '--hosts', default='127.0.0.1')

2. A minimal Python worker

# my_bench/worker.py
import sys, time, asyncio

from perfreport import WorkerArgs
from perfreport.protocol import emit_checkpoint, emit_summary, emit_environment
from perfreport.metrics import sample_cpu, sample_ram_mb
from perfreport.utils import decode_base64_pickle


async def start_test(args: WorkerArgs):
    num_queries = args.options['num_queries']
    emit_environment('MyLib', '1.0.0')
    start = time.monotonic()
    cp_time = start
    cp_interval = max(1, num_queries // 10)
    for i in range(num_queries):
        # ... do work ...
        if (i + 1) % cp_interval == 0:
            now = time.monotonic()
            elapsed = now - cp_time
            emit_checkpoint(i + 1, sample_cpu(), sample_ram_mb(), elapsed,
                            cp_interval / elapsed if elapsed else 0)
            cp_time = now
    total = time.monotonic() - start
    emit_summary(num_queries, sample_cpu(), sample_ram_mb(), total,
                 num_queries / total if total else 0)


if __name__ == '__main__':
    args: WorkerArgs = decode_base64_pickle(sys.argv[1])
    asyncio.run(start_test(args))

No base class to inherit from. You own the loop, own the state, emit protocol lines when ready.

3. A worker in another language

Any process that prints protocol lines works. Example in C:

// my_bench_worker.c — compile with `cc -O2 my_bench_worker.c -o my_bench_worker`
#include <stdio.h>
#include <time.h>

int main(int argc, char** argv) {
    int n = 100000;
    printf("::environment:: Lang C\n");
    fflush(stdout);
    struct timespec t0, t1;
    clock_gettime(CLOCK_MONOTONIC, &t0);
    for (int i = 1; i <= n; ++i) {
        // ... do work ...
        if (i % (n / 10) == 0) {
            clock_gettime(CLOCK_MONOTONIC, &t1);
            double elapsed = (t1.tv_sec - t0.tv_sec) + (t1.tv_nsec - t0.tv_nsec) / 1e9;
            printf("::checkpoint:: %d 0 0 %.4f %.2f\n", i, elapsed, (n/10.0)/elapsed);
            fflush(stdout);
            t0 = t1;
        }
    }
    printf("::summary:: %d 0 0 1.0 %d\n", n, n);
    return 0;
}

Wire it up with worker_cmd in the report:

Report(
    name='C worker',
    worker=WorkerModel(
        cmd='./my_bench_worker',  # bypass Python module resolution
        args=WorkerArgs(),
    ),
)

Same for Go, Rust, bash — anything that writes to stdout works.

4. Run

perfreport --report my_bench.report -n 100000 -c 100 --report-path ./out --html

This produces:

• ./out/report.md — Markdown with tables, embedded SVG charts
• ./out/report.html — HTML with interactive charts
• ./out/charts/*.svg — per-chart SVG files

CLI reference

Base flags (always available):

perfreport exits with 1 when a worker failed (a non-zero exit code, or a matrix venv that could not be provisioned) or the run was stopped, the failed workers are listed on stderr and marked in the report.

The first SIGINT, SIGTERM or SIGQUIT stops the run: the running worker is terminated and the report of the workers that ran is written, marked as interrupted. A second one quits at once.

Reports contribute their own flags via get_args(parser). A common set (-n/--num-queries, -c/--concurrency) can be added via from perfreport.args import add_common_workload_args; reports with different semantics (duration-based, num-rows, etc.) define whatever they need.

Core concepts

Report tree

A Report is a node in a hierarchical tree. Leaves have a WorkerModel (spawned as subprocess); branches have items (child reports) and optional charts aggregating data from descendants.

ReportDefaults cascades options, custom_metrics, protocol, worker path, matrix and unit down the tree — children inherit, can override or extend.

unit names what the workers count when it is not requests: ReportDefaults(worker=..., unit='Binds') titles the checkpoints and the summary of the workers Binds and Binds/s and their charts Binds per second, in place of Requests, Req/s and Requests per second. It names the fields req and rps of the protocol, which stay.

Text protocol

Workers write to stdout. The runner parses these commands:

Any other stdout/stderr is captured and displayed in the Stdout/stderr section of the report, as is a protocol line the results do not take, with the reason.

The output is read as lines whatever chunks the pipes give, the last line may have no newline. A worker is done when it exited and its stdout and stderr are closed, its exit code is kept in results.returncode.

Charts

ChartModel declares a chart; data source is auto-collected from sibling/descendant worker results:

ChartModel(
    title='Summary RPS',
    x_key='req', y_key='rps',
    xlabel='Category', ylabel='RPS',
    data_source='summary',       # 'checkpoints' | 'summary' | 'custom'
    chart_type='bar',            # 'line' | 'bar'
    bar_sort='desc',             # 'desc' | 'asc' | 'none'
)

Bars auto-pivot by category when each category has one summary point.

Matrix testing

Run the same benchmark across multiple Python versions or library versions by declaring a matrix:

from perfreport.models import Matrix

default = ReportDefaults(
    worker='my_bench.worker',
    matrix=[
        Matrix(name='py311', python='3.11', deps=['my_lib==1.0.0']),
        Matrix(name='py313', python='3.13', deps=['my_lib==1.0.0']),
        Matrix(name='local-build', python='3.13',
               deps=['./dist/my_lib-1.1.0-py3-none-any.whl']),
    ],
)

Each matrix entry produces a parallel sub-tree. uv creates an isolated venv per entry, installs perfreport (its checkout editable when perfreport runs from one, else the same version from PyPI) and the deps of the entry, and runs the worker under that interpreter. Summary bar charts compare entries side-by-side.

The venv is cached by a hash of the perfreport version, python, deps and extras. A dep that is a local wheel, archive or directory counts by its size and modification time: a wheel built again at the same path gets a new venv. A worker given as a module is imported from the directory this perfreport imports it from, by PYTHONPATH: the report package need not be installed in the venvs.

Cache directory defaults to ./.perfreport-venvs/. Override via --matrix-cache-dir; force rebuild with --rebuild-envs.

Output modes

• Default (VerboseHandler) — streaming: header once, then each checkpoint appended as a single line. No screen redraws.
• -i, --interactive — full-report refresh with prompts before each worker.
• --silent — file output only.

Writing a driver / extending

1. Put your report and worker modules under a package (e.g. my_bench.report and my_bench.worker).
2. Worker must implement start_test(args: WorkerArgs) as the entry point when invoked as a script.
3. Worker args (queries, values, expected results) are serialized via encode_base64_pickle by default: one argument that keeps UUID, Decimal, bytes and datetime, read by decode_base64_pickle(sys.argv[1]). encode_typed_json suits workers in other languages. Args longer than 100 000 chars come as @<path> of a file, which the decoders of perfreport read.
4. Extend ResultsModel.custom_metrics to capture driver-specific values (connection stats, latency percentiles, etc.).

Development

git clone https://github.com/ssw-tech/perfreport
cd perfreport
pip install -e .[dev]
pytest

License

MIT — see LICENSE.