fherma-runner

Takes measurement jobs from a FHERMA platform and reports what happened.

uv tool install fherma-runner        # or: pipx install fherma-runner
fherma-runner --api https://fherma.io --name bench-01 --token fhr_...

A command line tool belongs in an environment of its own, and on macOS it has no choice: the system Python has no pip on the path at all, and Homebrew's is marked externally managed so installing into it is refused. uv tool and pipx both make that environment and leave fherma-runner on the path, which is the part pip install --user does not do.

Neither installed? One line, and nothing else on the machine changes:

curl -LsSf https://astral.sh/uv/install.sh | sh

Check what you got:

$ fherma-runner --version
fherma-runner 0.5.0 · python 3.13.4

The token comes from the platform, is good for an hour and is spent once: the first start exchanges it for this runner's own key and keeps that in ~/.fherma/runners/<name>/key. Every start after the first needs no token.

--api is the API, not the website. They are usually different hosts, and a website answers an unknown path with a page and a 200 rather than an error — so the wrong one here fails as "answered with something that is not JSON".

Every start says what the machine is now, so a server that was re-provisioned is matched on what it is rather than on what it was.

Several on one machine

A machine can hold more than one: a card that is free while the CPU is busy, or two cards, or one runner kept for reference work and one for everything else. Each is its own runner on the platform, with its own invitation and its own key.

fherma-runner --name box-cpu --token fhr_...
fherma-runner --name box-gpu-0 --token fhr_...
fherma-runner --name box-gpu-1 --token fhr_...

--name is what keeps them apart on the machine as well as on the platform: the key and the jobs both live under it. On every start the runner prints who the platform says it is, which is the only way to notice that a name and a key have come apart:

$ fherma-runner --api https://api.fherma.io --name box-gpu-0
running as box-gpu-0 (6a87…f967)
polling https://api.fherma.io every 3.0s — ctrl-c to stop

~/.fherma/runners/<name>/
├── key       this runner's own, 0600
└── jobs/     what it is working on

Nothing is shared between them, which is the point — one key answering to two names would be one runner behaving very strangely, and a jobs directory two of them both tidy up is a directory that gets tidied up under one of them.

They do share the machine, so give each one limits it can actually have: --memory and --cpus are per runner and docker will not divide what you promise twice.

What it needs

An interpreter, docker and git. There are no Python dependencies: everything here ships with the interpreter, and both commands are called as commands.

Docker is not something a Python package can install — pip puts packages into an interpreter, and docker is a daemon that wants root and is a different thing to install on every system. So the runner does the next honest thing: it checks, and tells you the line to type.

$ fherma-runner --doctor
✓ python         3.13.4
✓ git            /usr/bin/git
✗ docker         not on the path
                 curl -fsSL https://get.docker.com | sh
· platform       https://api.fherma.io · answering
· key            none yet
                 pass --token to join

missing: docker

✗ is something to fix, · is something to know. The commands it suggests are for the system it is running on.

The same check runs before it joins, so a machine that cannot build never takes work: a job it accepts and fails counts against the job, and three of those drop the job for everyone.

What it declares

Hardware, and only hardware — cores, memory, architecture, CPU features, an accelerator and its driver. It does not declare libraries or images, because the environment a job runs in is not a property of a machine: it arrives with the job, pinned to a digest.

fherma-runner --describe

prints what it would report, and stops.

What it does with a job

~/.fherma/runners/<name>/jobs/<id>/
├── bundle/     the specification's bundle, unpacked
├── solution/   the repository at the commit, and its build
└── points/
    └── p000/   one point at a time, deleted once reported

Per point: the bundle makes the cases, the solution answers them, the bundle judges the answers. Three containers, and the solution is never given the directory holding the expected answers.

Options

--api           where the API is, not the website      FHERMA_API
--name          what to call this runner               FHERMA_RUNNER_NAME
--token         an invitation, spent once for a key
--interval      seconds between polls                  3
--once          take one assignment, then stop
--describe      print what this machine is, and stop
--doctor        check what it needs, say what is missing, and stop
--version       what is installed, and on what
--re-register   forget this name's key and join again
--memory        what a solution's container may take   FHERMA_MEMORY
--cpus          how many cores it may use              FHERMA_CPUS

The --speed, --fail-rate, --build-seconds and --point-seconds options belong to the half that is still pretended, and go away with it.

What the platform sees while it works

A heartbeat every ten seconds for the whole job, not one between points. It carries where the work is and whatever has been written since the last one, so a page can watch a build happen instead of waiting for it.

point 1 of 2 · N1190-L512 · generating 10 case(s)

00:34:34 STEP cloning the solution
00:34:34 OK   github.com/… at ed361af22049
00:34:34 STEP N1190-L512: generating 10 case(s)

Both halves matter. Beating only between points meant nothing was sent during the pull, the clone or the build — and a build longer than the lease had the assignment taken away underneath a machine that was working perfectly. And a transcript that only went out with the final report was never sent at all by a machine that died before making one, which is when somebody most wants to read it.

Stopping

The platform stops a runner by refusing its heartbeat. It then drops everything at once and sends nothing: finishing the work and reporting it would be reporting into a result somebody else now owns.

If a runner goes quiet instead, its lease lapses, the run is marked expired and the job goes to another machine.

Licence

Apache-2.0.