sr.py — structural reporting

A banded report formatter. Apply a template to a sequence of records and get a paginated document: bands, groups, page and column headers and footers, running aggregates, and page footers that know the final page count.

A Python library with a CLI over it, counterpart to the parallel Go project. Template plus JSON in, PDF out.

Documentation

How it works

Three artifacts and one language:

• A template is a KDL v2 document describing page geometry and a tree of bands. Bands are title, summary, header, footer, and detail; groups nest around the detail band.
• Data is JSON or NDJSON. Field types are declared in the template, which is what turns "19.99" into an exact decimal and "2005-05-24T22:53:30Z" into a time value.
• A printout is the engine's output: pages of absolutely-positioned marks with nothing evaluable left in them. Renderers do no layout.
• Expressions are Starlark — sandboxed, with no imports, filesystem, or network.

Layout is speculative: every band is measured before it is committed, then placed, split, or deferred to the next frame. Band splitting, keep-together, orphan and widow control, and correct deferred page counts all follow from that.

Command line

python sr.py build --template sakila.kdl --data payments.jsonl --out report.pdf

That is the launcher at the root of a checkout, which runs the program without installing the package. Installed with pip install s-reports, the same program is the s-reports command, and python -m sr where the interpreter's scripts directory is not on the path.

Other subcommands:

python sr.py validate sakila.kdl             # check a template without data
python sr.py inspect report.srp.jsonl        # dump a printout as readable text
python sr.py render report.srp.jsonl -o report.pdf

build writing a printout and render reading one back produce the same PDF as build writing PDF directly, which is what makes printouts worth archiving.

The document goes to standard output and everything about the run to standard error, so -o - pipes. Exit 0 is success, 1 a run that failed, 2 a mistake in the command line; warnings never change it, because they travel in the printout header where archiving keeps them. A one-letter flag takes one dash and a long one two, so -t and --template, but not --t or -template. A value is attached with = or separate, a flag may follow the positional arguments, and a switch also takes =true or =false. Full reference: doc/cli.md.

Library

from pathlib import Path

from sr.api import Options, build
from sr.printout.write import write_jsonl

result = build(
    Path("sakila.kdl"),
    Path("payments.jsonl"),
    Options(params={"period_start": "2005-06-01"}, strict_fonts=True),
)
out = Path("sakila.srp.jsonl")
with out.open("w", encoding="utf-8", newline="") as handle:
    write_jsonl(result.printout, handle, out.parent)

build returns the printout as an object, which is the primary artifact: serializing it is a separate step, and the directory it is written to is what its font and image paths are made relative to. Every flag on the command line is a field of Options, because the command line decides nothing.

build_records takes the records as Python values instead of JSON: mappings, named tuples, dataclasses, or other objects, such as the rows a SQLAlchemy query returns. A Decimal arrives as an exact decimal and a datetime as a time, rather than as text. A member the template declares is still coerced to its type, which for some types means writing the value as text and reading it back. doc/template.md says what each Python type becomes, and how a declared member coerces it.

from sqlalchemy import create_engine, text

from sr.api import build_records

engine = create_engine("sqlite:///sakila.db")
with engine.connect() as connection:
    rows = connection.execute(text("SELECT title FROM film ORDER BY title"))
    result = build_records(Path("minimal.kdl"), rows)

A template that names a typeface rather than a font file makes the first build enumerate the machine's fonts, which takes a few seconds. Later builds in the same process reuse that table while the font files are unchanged. For the changes that check cannot see, listed under host enumeration, sr.api.forget_host_catalog() drops the kept table.

Rendering is a separate step too, and takes the printout in memory or one read back from a file. Both give the same bytes:

from sr.pdf import render
from sr.printout.read import read

Path("sakila.pdf").write_bytes(render(result.printout))
Path("again.pdf").write_bytes(render(read(out)))

Running the examples

python sr.py build -t example/minimal/minimal.kdl -d example/minimal/films.jsonl -o minimal.srp.jsonl

minimal.kdl is the smallest report there is: one font, a header with a rule, and one field per record. Its printout is the example at the end of doc/printout.md, which the test suite builds and compares byte for byte, so that example is a document rather than an illustration of one.

python sr.py build -t example/sakila/sakila.kdl -d example/sakila/payments.jsonl -o sakila.pdf

Narrowed to one month, using the template's date parameters:

python sr.py build -t example/sakila/sakila.kdl -d example/sakila/payments.jsonl -o june.pdf --param period_start=2005-06-01 --param period_end=2005-07-01

The second example, which uses both kinds of subreport:

python sr.py build -t example/invoices/invoices.kdl -d example/invoices/invoices.jsonl -o invoices.pdf

sakila.kdl — a payment list grouped by customer in two columns: every band type, a group with its own title and summary, column header and footer, stretch fields, a floating element, six barcode types, an embedded image and a referenced one, both kinds of cross-reference, conditional outline entries, a deferred page count, justified text, and typed parameters.

invoices.kdl — invoices by region with line items: an inline embedded subreport with an arg and its own records, a template= subreport in region_sheet.kdl that paginates itself onto landscape pages of its own, a group using keeptogether with minrows and mintailrows, a summary with swapfooter, an image with embed=#false, a compressed data blob, iter="item" against iter="detail", and the calc modes sakila leaves out.

Between them the two templates use every node in the format but accept, and all twelve calc modes. They do not exhaust every property. What they leave out, so that reading them as a reference does not mislead:

Both reference the committed fonts by path, so they build identically on any machine and work under --strict-fonts. Swap file= for typeface= to use whatever the machine has instead.

Reproducible output

The same template over the same data produces byte-identical printouts when --build-time is fixed and --strict-fonts is set -- on any machine, not just across runs on one. Rendering adds no timestamp of its own. Without the first, the run timestamp differs; without the second, output depends on which fonts are installed.

Nothing machine-specific survives in a strict printout. Strict mode admits only fonts the template named by path, and every path the template named — fonts and embed=#false images alike — is written relative to the printout rather than absolute. So a printout and the files it points at move as one tree. The header records which font file each typeface resolved to and by which step, so a difference is diagnosable from the artifact.

Development

make install    # a venv, the dependencies, and the test, lint, and audit tools
make check      # ruff, zizmor, mypy, pytest
make audit      # known vulnerabilities in the packages and the actions
                # (the actions' half needs a GitHub token in GH_TOKEN)

Everything generated goes under build/: the pytest, ruff and mypy caches, setuptools' metadata, and compiled bytecode. The first three come from pyproject.toml and the fourth from PYTHONPYCACHEPREFIX, which the GNUmakefile exports — a pytest typed straight into a shell still writes __pycache__ beside the sources unless that variable is exported there too.

Tests are pytest. tests/unit/ covers a module at a time, tests/golden/ holds this engine to documents that are printed in the specification, and tests/differential/ compares this engine against the Go one, byte for byte, over the examples and over a corpus of probes. It compares the renderers too, by drawing each page of both PDFs with pdfium, from the oracles extra, and comparing the pixels; a PDF is also held to its structure by the tests' own reader, and by qpdf --check where qpdf is on the path. And it compares the command line: what validate reports on every template in the corpus, what inspect dumps of every printout, and the exit code, the output, and the summary line of a table of command lines, each asking one question of doc/cli.md.

The engine was built a milestone at a time, and every command and output format is here now. Every node the format has is built: it paginates reports of field, line, rectangle, image, barcode, and xref elements and outline entries, with groups, columns, band splitting, eject nodes, keep-together rules, balanced columns, deferred values, and subreports of both kinds, inline and with pages of their own, and writes them as NDJSON, CBOR, or PDF. Each printout is held to its own invariants before it is written, and s-reports inspect holds any printout it reads to them. Every barcode type is encoded here, from the standards, rather than by a library, and an image is interpreted only as far as the header that states its size. The PDF writer is written here too: the object table and the cross reference, the drawing, the outline and the links. fontTools subsets the faces and Pillow decodes the images the file does not carry as they stand. Each rendered page is drawn by pdfium and compared, pixel by pixel, with the same printout rendered by the reference. The command line and its two reports are held to the reference's too. The cases the differential corpus cannot yet compare are listed, with what each waits for, in tests/differential/pending.toml.

The Go implementation is the oracle: a reference binary whose outputs settle questions the specification leaves open. Its source is not read -- this is a second implementation from the specification, not a port. The harness builds it from a tree beside this repository, and skips with a reason where there is no Go toolchain to build it with. See tests/differential/README.md.

Where the two engines disagree and doc/ does not say who is right, the answer is a sentence in doc/, not a workaround in either engine.

License

MIT. See LICENSE.