Overview

pfdo_run walks an input directory with pftree, selects files in each directory with the pfdo filters and --analyzeFileIndex, and runs the --exec command once per selected file. The command is a pftag template: %{inputWorkingDir}, %{outputWorkingDir} and %{inputWorkingFile} name the directory being processed, its counterpart in the output tree, and the file; functions derive names, %{inputWorkingFile|rmext}.png. The output tree mirrors the input tree, and each job leaves its command, stdout, stderr and return code beside its results. Directories run in parallel under --threads.

The template is split like a shell command line and each argument has its tags resolved on its own, so a path with a space stays one argument and nothing a file name contains can change the command. The command runs without a shell.

Installation

pip install pfdo_run

Python 3.11 or newer; depends on pfdo and pftag.

Command line

pfdo_run --inputDir /data --outputDir /png --fileFilter 'glob:*.jpg' --threads 4 \
    --exec "convert %{inputWorkingDir}/%{inputWorkingFile} %{outputWorkingDir}/%{inputWorkingFile|rmext}.png"

pfdo_run --inputDir /dicom --outputDir /nifti --fileFilter 'glob:*.dcm' --analyzeFileIndex f \
    --exec "mri_convert %{inputWorkingDir}/%{inputWorkingFile} %{outputWorkingDir}/%{inputWorkingDir|basename}.nii.gz"

pfdo_run --man

All of pftree’s and pfdo’s options apply. pfdo_run adds:

--exec <commandTemplate>

The command to run per selected file. Tags: %{inputWorkingDir}, %{outputWorkingDir}, %{inputWorkingFile}, every option by name (%{outputDir}), and pftag’s core tags. Functions: rmext, ext, basename, dirname, replace, re, upper, lower, strip, pad, slice, md5, sha256, strftime, mask. A tag that does not resolve fails that job before it runs.

--noJobLogging

Keep no record of the jobs. By default each job appends one line of JSON to jobs.jsonl in its output directory: the command, both streams complete, the return code, the host, the start, end and duration. pflog read out/**/jobs.jsonl prints them as a table.

--jobFiles

Also write each job as <file>-cmd, <file>-stdout, <file>-stderr and <file>-returncode beside its results, the stream files rendered as a terminal would show them, for opening in an editor.

A job whose command exits non-zero fails its directory; the run continues and the directory’s record says why. --json prints the whole result, each directory’s jobs included.

Library

from pathlib import Path
from pfdo_run import PfdoRun, PfdoRunOptions

options = PfdoRunOptions(
    inputDir=Path("/data"), outputDir=Path("/png"), fileFilter="glob:*.jpg", threads=4,
    exec="convert %{inputWorkingDir}/%{inputWorkingFile} %{outputWorkingDir}/%{inputWorkingFile|rmext}.png",
)
result = PfdoRun(options).run()
for key, stage in result.process.analyze.items():
    for job in stage.payload["jobs"]:
        print(key, job["file"], job["run"]["returncode"] if job["run"] else job["errors"])

pfdo_run.__main__ exposes its parser and manual in parts (parser_build(standalone=False), synopsis(command=...), CLI_DS, ARGS_SYNOPSIS_DS) so that a ChRIS plugin such as pl-shexec composes its own command around them.

Changes in 4.0

• Built on pfdo 6 and pftag 3; pfmisc, pudb and Faker are no longer dependencies.

• The template grammar is pftag’s: %{name} and %{name|fn(arg)}. The 3.x forms %name and %_fn_name are gone; a bare %name of a known tag is reported as a 1.x template and fails the job.

• The command is split before substitution and run without a shell; 3.x substituted into the string and split on whitespace, so a path with a space broke it.

• The command runs through hostlib: no shell, both streams read concurrently as they arrive, so nothing is dropped and nothing can deadlock; 3.x read stdout line by line, lost the last line, and could deadlock on stderr. At --verbosity 2 and above the output also streams to the console while the job runs.

• Per-job records are written by default and --noJobLogging exists; in 3.x the option was documented but undefined, so logs were never written. The stream files are rendered as a terminal would have shown them, so a redrawn progress bar is one line; the record in --json keeps every frame.

• Each job’s record is FileJob: the file, the command as built, status, errors, and run, hostlib’s envelope of the run (both streams, exit status, host facts, timing) when the command ran.

• A non-zero exit status fails the job and its directory; the run’s status is truthful (3.x reported False always).

• main(argv) honours argv; --version, --man and --synopsis exit 0; packaging is pyproject.toml only, published from CI on a v* tag.