Xilinx Test Runner

A test runner for FPGA verification work: build the bitstream, keep the old ones, run the tests on the board, compare against the baseline.

One tool for the whole loop, with a console, a terminal interface and a window over the same run.

_{The window follows the desktop theme. The same run on a dark one: window-dark.png.}

What it does

Verifying a design on an FPGA prototype is the same loop every day: get the current sources, build a bitstream, run tests in simulation and on the board, compare the behaviour against the documentation, fix something, then prove that nothing which used to work has broken.

Every step of that already has a tool. What is usually missing is the thing that holds them together and remembers what happened. This is that thing.

It is not a simulator, not a replacement for UVM and not a continuous integration system. It sits in front of those and keeps the results.

Installation

Python 3.10 or newer. Nothing else is required; the window needs Qt.

git clone https://github.com/shchuchkin-pkims/xilinx-test-runner
cd xilinx-test-runner
./bin/xtr --help              # runs straight from the checkout

pip install .                 # or install it properly
pip install ".[gui]"          # with PyQt6, for the window

Every release also carries a wheel and a source archive on its releases page.

python3 -m xtr works anywhere the xtr command does.

Quick start

A demo project is included. It builds a fake bitstream with a shell script, so it runs with no vendor tools installed at all:

cd examples/demo_project
xtr list                      # what is here
xtr run                       # run it in the console
xtr artifacts build           # "synthesise" and store a variant
xtr run --accept              # accept these results as the baseline
xtr --gui                     # or the window

For your own project:

cd /path/to/project
xtr init                      # writes xtr.toml and an example test
xtr doctor                    # is everything where the config says it is
xtr run

xtr --gui and xtr --tui also open with no project at all, and can point themselves at one from the File menu.

How a test is written

A test is any executable file. Its exit code is the verdict, and a few prefixed lines carry everything else. A script that has never heard of this runner works as it is.

#!/bin/bash
# TEST: nack_timing
# GROUP: Hardware
# DESC: NACK must start inside the window the standard allows
# NEEDS: board                     # one board: never two of these at once
# REQUIRES: bitstream up to date   # skip me if the build is stale
# RETRY: 1                         # flaky connection; the report will say so
# TARGET: lab_pc                   # run me on the machine with the board
# COVERS: ISO7816-3 8.3

command -v xsdb >/dev/null || { echo "SUMMARY: no debugger here"; exit 2; }

echo "PROGRESS: 30 programming the device"
etu=$(measure_etu)

echo "METRIC: nack_window $etu etu tol=0.5"
echo "CHECK: nack inside the window $( [ "$etu" -le 12 ] && echo pass || echo fail )"

[ "$etu" -le 12 ] || { echo "SUMMARY: NACK at $etu etu, limit is 11.5"; exit 1; }
echo "SUMMARY: NACK at $etu etu, inside the window"

Five verdicts, because two are not enough

A missing cable must not paint the run red, and an open defect must not hide under green. KNOWN has its own counter and does not fail the run.

Lines a test can print

The full contract is in docs/writing-tests.md.

What the runner adds

Bitstreams are objects, not a file that happens to be there. Every build is stored as a named variant together with a hash of the sources it was built from. Switching between variants is a file copy, so comparing a fix against the previous build takes seconds instead of another synthesis run. Because the provenance is a hash of the content, copying a variant around cannot make an old build look current. A check based on file dates can, and that mistake is how results from two different designs end up mixed in one report.

Regression means comparison, not a demand for perfection. Tests report measurements and named checks; the runner keeps the accepted values and reports what moved. A test that used to fail and still fails in exactly the same way is reported as a known defect rather than noise. A test that starts passing is pointed out rather than silently absorbed.

The board is usually on another machine. A test can declare where it runs, and the runner drives it over SSH. Stopping a run kills the whole remote process tree, which is what keeps a cancelled run from leaving a debugger attached and the cable locked for the next hour.

Random stimulus stays reproducible. Each run has a seed, and each test gets a stable seed derived from it. Both are recorded, so a failure found with random data can be replayed with --seed.

Reports carry their context. Tool versions, source revision, active bitstream, machine name, seed. A report without those cannot be interpreted a month later.

Also included: dependencies between tests, setup tests that stop a section cleanly, retries that are always visible in the report, exclusive resources that hold even between separate runs of the program, a configuration matrix with a comparison report, and reports in text, JSON, JUnit XML and HTML.

Configuring a project

One file in the project root. TOML is read by the standard library; YAML works too if PyYAML is installed.

[project]
name  = "uart_7816 on EBAZ4205"
tests = "tests"

[tools.vivado]
path = "/opt/Xilinx/2025.1/Vivado/bin/vivado"   # or omit it and let it be found

[targets.lab_pc]                                 # the board is on another machine
type     = "ssh"
host     = "lab@10.0.0.2"
identity = "~/.ssh/id_ed25519_lab"
root     = "/home/lab/project"                  # the same tree, over there

[artifacts.bitstream]
store       = "fpga/bitstreams/variants"
active      = "fpga/bitstreams"
files       = ["design_1_wrapper.bit"]
extra_files = ["design_1_wrapper.xsa"]
sources     = ["rtl/**/*.v"]                     # what the provenance hash covers
build       = "bitstream"

[flows.bitstream]
plugin  = "vivado"
project = "fpga/EBAZ4205/EBAZ4205.xpr"
tcl     = "fpga/scripts/build.tcl"
output  = "fpga/EBAZ4205/EBAZ4205.runs/impl_1/design_1_wrapper.bit"
tail    = ["fpga/EBAZ4205/EBAZ4205.runs/impl_1/runme.log"]

[vcs.rtl]
path = "rtl"

From that, the runner generates the steps a project would otherwise script by hand: check whether the bitstream matches the sources, build a new one, switch between variants, show the source revision, fetch and pull it, and verify that the tools are installed where the configuration says.

Machine specific paths belong in xtr.local.toml next to it, which is not committed. Every key is documented in docs/configuration.md.

The interfaces

All three drive the same run and share the same state.

The window has four tabs. Tests shows the tree, the live output and the measurements next to the accepted values. Artifacts shows the stored bitstreams, which one is active and whether it still matches the sources. Run log collects everything every test printed, verbose lines included, which is where a long synthesis run is watched. History lists previous runs and how a measurement moved between them.

Command line

xtr                                     # run everything, print a table
xtr run sim --jobs 4                    # only what matches "sim", four at a time
xtr run --matrix bitstream=fix,old hw   # the same tests on two builds, compared
xtr run --seed 1234                     # replay a random run exactly
xtr run --accept                        # record these results as expected
xtr artifacts list                      # what is stored, what is active
xtr artifacts activate old_build
xtr history trend etu                   # is that measurement drifting
xtr issue                               # a ticket draft from the last failure
xtr doctor                              # check tools, targets and paths
xtr selftest                            # the runner's own tests

A complete example on real hardware

examples/pynq_z1 is a full project for a Digilent PYNQ-Z1: a written specification with seven numbered requirements, the RTL, a simulation testbench, and tests that run on the board. It is programmable logic only, so no software has to be built for the processor and no SD card is involved; the host talks to the design over the same JTAG cable that programs it.

ok    program              31.3 s  device configured, VIO answering, counter running
ok    bitstream up to date  0.0 s  built from the current sources (9eda4dd157f06f4b)
ok    syntax                1.0 s  2 file(s) compile clean, 0 warning(s)
ok    step_counter_tb       5.1 s  16 checks, all as specified
ok    lfsr                 27.7 s  16 values match the model, the seed reload works
ok    steps                27.7 s  reset, edge stepping, up, down, hold and wrap behave as specified

Documentation

Russian translations of the main documents are in docs/ru/, and the overview is in README.ru.md.

Requirements

Nothing is downloaded at run time and no usage data is collected.

Design decisions

These are deliberate, and they are the reason the tool stays small.

• The core knows nothing about FPGAs. Everything vendor specific is either configuration or a test. If that stops being true, the tool is no longer portable to the next project, and a shell script would do the job instead.
• No external test manifest. Test metadata lives in the header of the test file, so adding a test means adding a file, and nothing can drift out of sync with a file that does not exist.
• Skipping is not failing. Absent hardware, an unlicensed tool and a dead connection are three different situations, and none of them is a defect in the design.
• A retry that hides instability is worse than no retry. The report always states that a test needed a second attempt.
• Accepting a baseline is a deliberate act. A baseline that updates itself compares a run against itself.
• No job queue, no agents, no scheduler. Those products exist. This one runs on the machine in front of the hardware.

Contributing

Bug reports, recipes for tools nobody has tried yet, and patches are all welcome. The suite that guards the behaviour above runs with no dependencies:

xtr selftest        # 94 tests

CONTRIBUTING.md describes the layering rule and what a change is expected to come with.

License and trademarks

MIT. See LICENSE.

This project is independent and is not affiliated with, endorsed by or sponsored by AMD, Cadence, Siemens or Intel. Product names are used only to describe which third party tools the runner can drive. No vendor artwork is included: the icon and the logo are original work. See TRADEMARKS.md.