pinexq CLI

A minimal command-line interface for creating and running pinexq workers.

Install (editable):

• Using uv or pip, from the project root:
  • uv pip install -e . or pip install -e .

Usage:

• Global options must precede the subcommand.
• Authentication: Provide --api-key or set the PINEXQ_API_KEY environment variable. The CLI prefers the flag over the env var.

Reading Resources (list and show)

The CLI can read every JMA Resource in your Context. Each Resource is a noun with two verbs — list (many) and show <ref> (one):

These commands run from any directory and do not need a Project.

Connection

Every read command resolves its endpoint and API key in this order:

1. the --endpoint / --api-key flags,
2. the PINEXQ_ENDPOINT / PINEXQ_API_KEY environment variables,
3. for the endpoint only, pinexq_endpoint in a pinexq.toml in the working directory (there is no file fallback for the API key).

Output

-o, --output selects table (default, human-readable), json, or csv. --wide adds the extra columns to the table; -q, --quiet prints only ids, one per line, for piping. JSON and CSV always carry the full column set and go to plain stdout so they can be redirected safely.

Pagination and sorting

--limit (page size, max 1000) and --offset page the result; --all fetches every page and overrides both. --sort FIELD picks the sort field (the choices differ per noun) and --desc / --asc the direction; each noun has a natural default (Jobs, WorkData, Templates, and Triggers newest-first; ProcessingSteps by title and Folders by path, ascending).

Filters

Filter flags are grouped under a Filters panel in each command's --help. The families a noun exposes follow its JMA query: tag filters (--tag, --tag-or, --tql), a folder scope (--folder, --recursive), and date bounds (--created-after, --created-before) where the Resource supports them, plus noun-specific flags — for example --state for Jobs, --kind and --media-type for WorkData, --function / --version / --deployment-state for ProcessingSteps, --enabled / --disabled for Triggers, and --path-prefix for Folders. Run pinexq <noun> list --help for the full set.

Reference forms for show

show accepts a Resource's UUID or its full URL for every noun. Two nouns accept a friendlier reference as well:

• pinexq ps show <name> or pinexq ps show <name>@<version> resolves a ProcessingStep by Function name (latest version, or the pinned one).
• pinexq folder show </path> resolves a Folder by its path; folder show also lists the Folder's direct child folders.

One example per noun

pinexq job list --state completed --sort created_at --limit 20
pinexq wd list --kind clientupload --media-type text/csv -o json
pinexq ps show resize@1.2.0
pinexq template list --tag nightly --wide
pinexq trigger list --enabled
pinexq folder show /projects/foo

Acting on Resources (actions, action, and the Bound verbs)

The JMA advertises every state change as an Action on the Resource (or on the Resource type's root, for creation), and lists an Action only while the Resource's state and your grants allow it. The CLI discovers and executes those Actions by name instead of hardcoding a fixed set. Every noun in the table above — job, workdata/wd, processing-step/ps, template, trigger, and folder — has the two generic verbs:

pinexq job actions <ref>                              # what can I do to this Job right now?
pinexq job action <ref> hide                          # execute a parameterless Action
pinexq wd action <ref> rename --set NewName=report    # execute one with parameters
pinexq folder actions --root                          # how do I create a Folder?
pinexq folder action --root create-folder --set Name=reports
pinexq template action <ref> execute-now              # prints the Job it created
pinexq job action <ref> delete --yes                  # Destructive, no prompt

show <ref> in table output ends with a footer naming the Actions the Resource offers right now (Actions: hide, rename, ...), so the next step is one screen away. The JSON and CSV output of show are unchanged.

Discovering Actions

<noun> actions <ref> lists each available Action with its name in kebab-case, its HTTP method, its markers (such as Destructive, or FileUploadAction), and its parameters as Name: type, required read from the schema the JMA serves for it. -o json adds the served name, the URL, the Siren class list, the current parameter values, the schema URL, and the schema itself; -q prints only the names.

Executing an Action

<noun> action <ref> <name> executes an Action and then prints the result with the same renderer and -o table|json|csv / -q options as show:

• an Action that changes the Resource prints the Resource, re-fetched;
• an Action that creates a Resource (a creation Action on --root, or a Template's execute-now) prints the created Resource, with the view of its own type — template action <ref> execute-now -q prints the new Job's id;
• an Action that answers with a report body (a Folder's clear) prints that report;
• an Action that removes the Resource (a delete) prints what is left of it: its id and url.

The name is matched case-insensitively and ignoring - and _, so StartProcessing, start-processing, and start_processing all work.

Parameters are built in three layers, each overriding the one before:

1. the current values the JMA sends with the Action (so you only mention what changes),
2. --json BODY, --json @file.json, or --json - (stdin), a JSON object,
3. --set KEY=VALUE, repeatable.

Each --set value is typed by the schema property: integer, number, boolean (true/false, yes/no, on/off, 1/0), string (kept verbatim, so --set NewName=1 is the string "1"), array (a repeated key appends, and a JSON list literal is accepted as is), and object (a JSON literal). null is accepted for a nullable property. The body is validated against the schema (Draft 2020-12) before anything is sent: a missing required property, an unknown property, or a type mismatch is a usage error (exit 2) naming the property.

When the JMA serves no schema for an Action, or the schema cannot be fetched, the CLI prints a notice on stderr, types each value heuristically (JSON if it parses, otherwise a string), and lets the JMA validate.

--dry-run resolves the target, finds the Action, types and validates the parameters, then prints the method, URL, markers, and the pretty-printed body without sending anything or asking for confirmation.

An Action that takes a file upload (marked FileUploadAction, such as the WorkData root's upload) cannot be sent by action; the CLI refuses it with a hint to the wd upload Bound verb (a separate ticket).

Destructive Actions and confirmation

The JMA marks an Action whose effect cannot be undone as Destructive (for example delete); the CLI never hardcodes that list. Before executing one:

• on a terminal, the CLI asks ... is Destructive and cannot be undone. Execute? and declining exits 1 without an error message;
• --yes / -y skips the prompt, for scripts;
• without a terminal and without --yes, the CLI refuses with exit 1 and a hint to pass --yes, so a pipeline never hangs on a prompt.

Non-Destructive Actions never prompt. The prompt goes to stderr, so -o json output stays clean. A Bound verb acting on several Resources asks once, naming the Action and the count.

Job Bound verbs

Jobs can be composed incrementally. job create --name NAME creates an empty Job; add --processing-step STEP for the one-shot RapidSetupJob behavior, or use job create --name NAME --parent JOB [--inherit-folder] for a sub-job. The step-by-step verbs are:

SLOT is either the 0-based input DataSlot index or its name (case-insensitive); job show and template show display the index. Use --param, --input, --tag, and --start only with the RapidSetupJob form. DataSlot Actions are available through these Bound verbs rather than the generic action command; see ADR-0006.

A Bound verb executes exactly one Action with typed flags. It runs through the same path as action (same typing, validation, confirmation, and rendering) and adds nothing the JMA does not advertise. pinexq job --help lists them in their own panel:

Every Bound verb also accepts --dry-run, -o, and -q.

WorkData Bound verbs

The same housekeeping verbs exist under workdata and its alias wd, with comment in place of Job's start and set-error:

A WorkData that a Job produced does not allow deletion until you say so: the JMA offers AllowDeletion instead of Delete. Plain wd delete fails on such a WorkData with the usual not-available message, listing allow-deletion among the Actions it does offer, and continues with the rest of the batch. wd delete --force executes AllowDeletion first, re-reads the WorkData, and then executes Delete; a WorkData that already allows deletion is deleted directly. The one confirmation is asked up front, before anything is sent, because the end of the two-step is the Destructive Delete. --dry-run prints both steps. Should Delete fail after AllowDeletion succeeded, the error is followed by a note that the WorkData now allows deletion.

pinexq wd list -q --folder /runs/2024 --unused | pinexq wd delete - --force -y

Upload and download

pinexq wd upload ./input.csv --tag raw --folder /data/2024
pinexq wd download <ref> -o - | head

ProcessingStep Bound verbs

Under processing-step and its alias ps; references accept a UUID, URL, name, or name@version as ps show does:

Template Bound verbs

The DataSlot verbs act on a sub-entity of the Template, which the generic action verb cannot reach.

Trigger Bound verbs

Folder Bound verbs

Folder references accept a /path, UUID, or URL as folder show does.

job create

pinexq job create --name run-1 --processing-step resize@1.2.0 \
  --param quality=90 --input 0=<wd-ref> --input 1=<wd-ref>,<wd-ref> \
  --tag nightly --folder /runs --start

Creates a fully configured Job in one root Action (the JMA's RapidSetupJob) and prints it (-q prints its id). The ProcessingStep accepts a UUID, URL, name, or name@version; WorkData references accept a UUID or URL, and several on one 0-based DataSlot index assign a collection. --param values are typed and validated against the ProcessingStep's own parameter schema when it carries one (a type mismatch or unknown parameter exits 2), heuristically otherwise, and are sent as the Parameters JSON text the Action takes (a repeated key builds an array, as with --set). --start starts the Job in the same request. --folder is not a RapidSetupJob field, so the created Job is moved afterwards as a follow-up Action; if that fails the Job exists, unmoved, and the exit code is 1. --dry-run prints the assembled body. WorkData references in --input and Template references elsewhere must be a UUID or URL; anything else is a usage error before a request is made.

Creating verbs and follow-ups

wd upload, template create, trigger create, job create, and folder create create a Resource through a root Action and then chain zero or more follow-up Actions on it (tags, a Folder, parameters, linked Templates). The creation is planned and validated before anything is sent; --dry-run prints it and names the follow-ups. If a follow-up fails, the failure is reported with a note naming the created Resource's id and the failed step, the created Resource is still printed (-q its id), and the exit code is 1.

Several Resources at once

The multi-ref verbs (start, delete, hide, unhide, tag, move, and their equivalents on the other nouns: deprecate, restore, enable, disable) accept several references and print one table (or JSON array) with one row per Resource; -q prints the ids. A reference that fails — not found, not in the right state, a JMA error — is reported on stderr and the rest continue; the exit code is 1 if any failed. A <ref> of - reads references from stdin, one per line (blank lines ignored), which closes the loop with list -q:

pinexq job list -q --state failed | pinexq job delete - -y
pinexq job list -q --tag nightly | pinexq job tag - --add archived
pinexq wd list -q --tag scratch | pinexq wd hide -

Reading references from stdin leaves no terminal for a prompt, so a Destructive batch fed from a pipe needs --yes.

Exit codes

Connection resolution and the <ref> forms are the same as for show.

Job logs (pinexq job logs <ref>)

pinexq job logs <ref> prints the Log entries of a Job, oldest first, one per line. <ref> is the Job's UUID or URL, and the connection resolves exactly as for list and show. The CLI follows the Job's Logs link to the monitoring service and pages through the whole log.

Formats

-o, --output selects the line format:

• text (default): [timestamp] LEVEL message. On a TTY the level is coloured (yellow for WARN, red for ERROR and above); piped output has no escape codes. --no-color or the NO_COLOR environment variable forces plain text.
• raw: the stored message only, verbatim.
• json: NDJSON, one flat object per line with timestamp, level, message, and metadata (not an array, so jq -c .level works line by line).

raw and json go to plain stdout and are never coloured. No line is ever wrapped, whatever the terminal width.

Filters and --tail

The monitoring service has no server-side filters, so these run in the CLI:

• --level L keeps entries at that Log level and above. Levels in ascending order: TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL (FATAL and CRITICAL rank equal). The value is case-insensitive and tab-completes; an entry with an unknown level always passes.
• --search TERM keeps entries whose message contains the term (case-insensitive substring, no regex).
• --level and --search combine with AND.
• --tail N prints only the last N matching entries, still oldest first. The CLI reads the log newest-first and stops as soon as N entries matched, so a tail of a long log is cheap.

pinexq job logs 11111111-1111-1111-1111-111111111111 --level warn
pinexq job logs <ref> --search timeout --tail 20
pinexq job logs <ref> -o json | jq -c 'select(.level == "ERROR") | .message'

Exit codes

• 0: the log was read, including when no entry matched (a notice goes to stderr and stdout stays empty).
• 1: the Job has no Logs link (job <ref> has no logs), the Job was not found, authentication failed, or the endpoint could not be reached. The reason is printed on stderr with the same wording as list and show.
• 2: a usage error, such as an unknown --level.

Job metrics (pinexq job metrics <ref>)

pinexq job metrics <ref> fetches the Metrics of the Worker that ran a Job: CPU usage in cores, memory usage in bytes, and both as a percentage of the Worker's requested resources. The window is the Job's own lifetime as the JMA built it (a running Job is read up to now); there are no time flags.

• Default: a header with the Worker name, status, termination reason, duration, queried window, and the peak CPU and memory with their utilization percentages, followed by two terminal plots (drawn with plotille, braille glyphs): CPU usage in cores over time, then memory usage in MiB over time. The x axis spans the queried window. Plots take the terminal width on a TTY and 100 columns when piped; colour follows the job logs rule (--no-color / NO_COLOR turn it off). Use -o json or -o csv on a terminal that cannot show braille.
• -o json: one flat object with worker, status, terminated, termination_reason, duration_seconds, queried_start, queried_end, and the four series cpu_usage, memory_usage, cpu_utilization, memory_utilization as lists of {timestamp, value}.
• -o csv: the columns timestamp,cpu_usage,memory_usage,cpu_utilization,memory_utilization, one row per distinct timestamp across the four series, blank where a series has no sample, ascending.

Exit codes follow job logs: 0 with a stderr notice when the series are empty (-o csv then prints the header row only), 1 when the Job has no Metrics link (job <ref> has no metrics), was not found, or the connection or authentication failed.

Deployment resources (pinexq.toml)

pinexq deploy configures the resources each job pod of a function reserves, from the [deployment] table (for every function) or [function.<name>.deployment] (for one), as Kubernetes quantities:

[deployment]
cpu = "1"                   # cores: "500m", "1.5", "2"
memory = "4Gi"              # "512Mi", "16Gi", or bytes
disk = "10Gi"
max_replicas = 2

[function.train.deployment]
cpu = "6"
memory = "16Gi"
disk = "100Gi"
max_replicas = 1

Explicit resources need all three keys. The platform takes whole millicores, MiB and GiB, so a value that does not convert exactly (memory = "1G", disk = "500Mi") is a configuration error; exponent notation (1e3) is not supported. Without resources, a function gets Medium's values (1000 millicores, 4096 MiB, 10 GiB). JMA rejects a value that exceeds the platform's resource ceiling; pinexq ps show displays the resources a deployment got.

resource_preset = "Large" (XSmall, Small, Medium, Large, XLarge, XXLarge) still works in place of the three keys, but is deprecated and will be removed in a future release. pinexq deploy warns for each function that uses one and prints the cpu, memory and disk that replace it.

Deploying on Docker Desktop (containerd image store)

Fresh Docker Desktop installs since version 4.34.0 default to the containerd image store. This changes what a local image looks like, and both differences are expected and supported by pinexq deploy:

• An image ID is the manifest (or index) digest rather than the config digest.
• A docker build produces an OCI image index that also carries provenance and SBOM attestation manifests. This is normal; the deploy pushes it and registers the digest the registry reports.

Check which store is active:

docker info --format '{{ .DriverStatus }}'   # containerd store shows "io.containerd..."

Note that docker system prune without -a does not remove tagged images — so if a bad deploy is fixed by "prune and retry", it is the retry's fresh push that fixes it, not the prune.

Troubleshooting: ImagePullBackOff right after a deploy

If a function pod fails to pull the image you just deployed, work through this checklist. The full analysis is in docs/research/2026-08-28-imagepullbackoff-after-successful-push.md.

1. Re-run the deploy with --verbose. The push stream is printed chunk by chunk, so a swallowed registry error (denied, unauthorized, blob upload unknown) becomes visible. Since the fix in this repo, a failed push aborts the deploy instead of registering an unpullable digest.

2. Read the pod events and map the error text to a cause:

   kubectl describe pod <pod>
   

   • failed to resolve reference ... not found → the registered digest is not a manifest the registry holds for this repository (the classic bug).
   • failed to authorize ... 401 Unauthorized → registry token / pull-secret problem, not a missing image.
   • could not fetch content descriptor ... not found / httpReadSeeker: failed open → the manifest exists but a referenced blob is missing.
   • A localhost:6443 / mirror address in the message is the k3s Spegel mirror being tried first; the upstream registry error follows it.

3. Inspect the local image digests (compare against what was registered):

   docker image inspect --format '{{json .RepoDigests}}' <name:tag>
   docker image inspect --format '{{ .Id }}' <name:tag>
   docker buildx imagetools inspect <registry>/<context>/<function>:<version>
   

4. Ask the registry directly whether the digest is present:

   # with a token from the registry realm
   curl -sI -H "Authorization: Bearer $TOKEN" \
     https://<registry>/v2/<context>/<function>/manifests/<digest>
   # or, if available:
   skopeo inspect --raw docker://<registry>/<context>/<function>@<digest>
   crane manifest <registry>/<context>/<function>@<digest>
   

5. Check the registry pod for restarts or manifest unknown log lines around the time of the deploy:

   kubectl -n <ns> logs deploy/registry --since=1h | grep -iE 'manifest|blob|unknown'
   kubectl -n <ns> get pod -l app.kubernetes.io/name=registry