Roejobs

Roejobs is a local job scheduling and management tool designed for running, monitoring, and controlling long-running tasks on a single machine. It provides:

• A background job server managing multiple tasks simultaneously.
• A CLI for submitting jobs from files or the command line.
• A web dashboard to monitor job status, view logs, cancel or restart jobs.
• Automatic management of working directories, output logs, and job queuing.

Features

• Queue jobs from files or directly from the CLI.
• Optionally wait for all queued jobs to finish and report their outcome (--wait).
• Support for CWD=... and ENV=KEY=VALUE prefixes per job, or a global CWD override.
• View running, queued, succeeded, failed, or cancelled jobs in a web browser.
• Stream job stdout/stderr to log files and view them via the dashboard.
• Cancel, restart, or permanently remove jobs from the web UI, TUI, or API.
• Live-adjust the number of concurrent jobs without restarting the server.
• Per-job CPU% and RAM usage for running jobs, aggregated across any child processes it spawns.
• Multi-stage pipelines defined in YAML (e.g. preprocess -> train -> evaluate), with arbitrary nesting of parallel/sequential steps, via roejobs-pipeline.
• Shutdown the server gracefully from the web UI.
• CLI automatically launches the server in the background if it is not running.
• The dashboard, TUI, and CLI all talk to the same JSON API — anything one can do, the others can too.

Installation

pip install roejobs

This will install the CLI commands and the package itself. Or, if you cloned this package locally:

pip install -e .

Starting the server

Start the job server on the default port (5678):

roejobs-server

Optional arguments:

roejobs-server --port 1234           # Specify a custom port
roejobs-server --n-processes 20      # Set maximum concurrent jobs

• The server runs locally (127.0.0.1) and serves the web dashboard.
• Default maximum concurrent jobs: 10.
• The pool size isn't fixed at startup — it can be changed live at any time, without restarting the server or losing running/queued jobs (see "Live pool resizing" below).

Access the dashboard

Open your browser at: (http://127.0.0.1:5678/)

From here you can:

• Monitor all jobs (auto-refreshes in place — no page reload, so your scroll position is preserved)
• View logs
• Cancel, restart, or remove jobs
• Live-adjust the concurrent job limit
• Shut down the server

Submitting jobs via CLI

1. From a task file

Each line of a task file is a job:

# Comment lines start with #
CWD=./exp python train.py --lr 0.001 --batch-size 64
python eval.py --checkpoint ckpt.pt

Submit jobs:

roejobs-cli tasks.txt

Optional flags:

• --override-cwd — ignore CWD=... in job specs and use the CLI's current working directory.
• --server — point to a server on a custom port.
• --wait — after queuing, block until all submitted jobs conclude, then print their final states and exit.

2. Inline submission

You can also submit jobs directly from the CLI:

roejobs-cli --jobs "python quick_test.py" "CWD=./exp ./run.sh --config cfg.yaml"

• Syntax matches task files.
• Works with or without --override-cwd.

3. Lazy server start

If the server is not running, the CLI automatically launches it in the background on the correct port.

4. Waiting for jobs to finish

By default the CLI returns as soon as the jobs are queued. Pass --wait to block until every job it submitted reaches a terminal state (Succeeded, Failed, or Cancelled):

roejobs-cli --wait --jobs "python train.py --lr 0.01" "python eval.py"

When all jobs conclude, the CLI prints each job's final state and exits. The exit code is 0 only if every submitted job succeeded; if any job failed or was cancelled it exits 1, which makes --wait convenient in scripts and pipelines.

There is no timeout — job durations are hard to predict — but you can stop waiting at any time with Ctrl+C. Interrupting only stops the CLI from waiting; the jobs keep running on the server and can still be monitored from the dashboard.

Job spec syntax

• Lines starting with # are ignored.
• Optional working directory per job:

CWD=some/path command args...

• Optional per-job environment variables — repeat ENV= for each variable:

ENV=CUDA_VISIBLE_DEVICES=0 ENV=WANDB_MODE=offline python train.py

• CWD= and ENV= can appear in any order, any number of times (repeat ENV= per variable), before the command:

ENV=CUDA_VISIBLE_DEVICES=1 CWD=./exp python train.py --lr 0.001

• Env values can't contain spaces (no quoting support yet) — ENV=MSG=hello world will parse world as part of the command, not the value.
• If no CWD is specified, the CLI/server uses the current directory.
• Two jobs are only treated as duplicates (same UID) if they match on command, working directory, and env — so the same script with a different CUDA_VISIBLE_DEVICES is tracked as a distinct job.
• Commands can be very long; they are truncated in the dashboard for readability, with full output available in the log view.

Live pool resizing

The number of concurrent jobs can be changed at any time without restarting the server:

roejobs-cli --set-pool-size 4 --server http://127.0.0.1:5678

This only affects new job intake — it never kills already-running jobs. Setting it to 0 pauses the queue (nothing new starts) without touching what's currently running. The same control is available as a field on the web dashboard, and as +/- keybindings in the TUI.

Equivalent API call (this is what the CLI/dashboard/TUI all use under the hood):

curl -X POST http://127.0.0.1:5678/pool-size -H 'Content-Type: application/json' -d '{"n_processes": 4}'

Resource monitoring

Running jobs report live CPU% and RAM (RSS, in MB) in the dashboard table, job detail page, TUI, and /jobs//jobs/<uid> API responses. Usage is aggregated across the job's process and any children it spawns — e.g. if your command is a launcher script (torchrun, a shell wrapper, etc.), the actual worker processes' usage is included, not just the launcher's.

Sampled once per scheduler tick (every 0.5s by default), not per request, so checking the dashboard doesn't add polling overhead. Values are null/empty once a job isn't running (queued, or finished) — there's nothing to measure. CPU% can briefly read 0 for a child process on the tick it first appears; it's accurate from the next tick onward.

Web dashboard features

• Job table showing status: Queued, Running, Succeeded, Failed, Cancelled.
• Full command, working directory, and env overrides available per job.
• Cancel, restart, or remove individual jobs.
• Live pool size control.
• Shut down server with a single button (gracefully kills running jobs).
• Commands truncated visually for long parameters, with full command in a tooltip.

Cancel vs. Remove

• Cancel stops a queued or running job but keeps it in the job list/history as CANCELLED — useful for a record of what happened.
• Remove permanently deletes a job from the list and deletes its logs. Only allowed on queued or already-finished jobs; a running job must be cancelled first.

API

The dashboard, CLI, and TUI are all just clients of the same local JSON API (default http://127.0.0.1:5678) — anything they can do, a script or agent can do too by hitting the same endpoints:

Method	Path	Description
GET	/jobs	List all jobs (JSON), including live cpu_percent/memory_mb for running ones.
GET	/jobs/<uid>	Single job status. Returns JSON if Accept: application/json is set, otherwise an HTML detail page.
GET	/jobs/<uid>/logs	{"stdout": "...", "stderr": "..."} for a job.
POST	/jobs	Submit a job: {"cmd": [...], "cwd": "...", "env": {...}} (cwd/env optional).
POST	/jobs/<uid>/cancel	Cancel a queued or running job.
POST	/jobs/<uid>/restart	Requeue a finished/cancelled job.
POST	/jobs/<uid>/remove	Permanently delete a queued/finished job and its logs. 409 if still running.
GET/POST	/pool-size	Get/set the live concurrent-job limit: {"n_processes": N}.
POST	/shutdown	Gracefully shut down the server.

Set Accept: application/json on GET requests if you want JSON back from routes (like /jobs/<uid>) that can serve either HTML or JSON.

Example workflow

# Start server (background auto-start optional)
roejobs-server --port 5678

# Submit jobs from a file
roejobs-cli tasks.txt

# Submit inline jobs
roejobs-cli --jobs "python train.py --lr 0.01" "CWD=./exp ./run.sh"

# Open browser to monitor jobs
firefox http://127.0.0.1:5678/

Pipelines

Pipelines let you compose jobs into ordered/parallel structures, and those structures nest arbitrarily. Every step defaults to being a single job. A step can also be:

• parallel: [<step>, ...] — run all of these together, wait for all to finish
• steps: [<step>, ...] (optionally with its own name/on_stage_failure) — a nested sequence, itself a full sub-pipeline

Since steps can contain other parallel/steps blocks, you can build things like "N independent preprocess -> train -> eval sub-pipelines running in parallel, followed by one aggregation step that only starts once all N are done":

# pipeline.yaml
name: my-experiment
on_stage_failure: halt   # halt (default) | continue

steps:
  - name: all-samples
    parallel:
      - name: sample-0
        steps:
          - CWD=./data/0 python preprocess.py
          - CWD=./exp/0  python train.py
          - CWD=./exp/0  python eval.py
      - name: sample-1
        steps:
          - CWD=./data/1 python preprocess.py
          - CWD=./exp/1  python train.py
          - CWD=./exp/1  python eval.py
      # ... one block per sample
  - CWD=./exp python aggregate.py

A flat preprocess -> train -> eval pipeline (no nesting) is just this with parallel used once per stage — see the simple 3-line-per-stage examples further down for that shape. Each job line uses the same CWD=/ENV= syntax as task files. A job can also be written in structured form ({cmd: [...], cwd: ..., env: {...}}) if you need something the string syntax can't express, like env values containing spaces.

on_stage_failure (settable per-sequence, defaults to halt):

• halt — a failed/cancelled step stops that sequence immediately; later steps in it are never queued.
• continue — later steps in that sequence still run. The sequence's own final status is still FAILED if anything in it ever failed — continue only affects whether execution proceeds, not whether the outcome is reported honestly. This composes: if an inner sequence used continue and still ends up FAILED overall, an outer sequence with halt will correctly stop rather than proceeding past it.

parallel failure handling: a parallel block waits for every child to reach a terminal state before deciding pass/fail — it does not cancel still-running siblings the moment one fails, so a failure in one of 20 parallel branches doesn't touch the other 19. Whether the pipeline proceeds past that parallel block afterward is then decided by the enclosing sequence's on_stage_failure, same as any other step.

A sharp edge worth knowing: a job's identity elsewhere in roejobs is a hash of cmd+cwd+env (this predates pipelines - see below), and submitting a second job with an identical hash while the first is still queued/running is treated as a no-op, not a second execution. If two parallel branches happen to submit byte-identical commands, they'll share one execution rather than run twice. This won't come up in the common case (branches differing by data path/index), but it's worth having in mind if a branch's command doesn't otherwise vary.

Running pipelines

roejobs-pipeline run pipeline.yaml              # submit and return immediately
roejobs-pipeline run pipeline.yaml --wait        # block, print live job-count progress, exit non-zero on failure
roejobs-pipeline run pipeline.yaml --override-cwd  # ignore CWD=... in the file, use the CLI's cwd for every job

roejobs-pipeline list                            # all pipelines, with status and jobs-done count
roejobs-pipeline status <id>                     # full step tree with status, indented to match nesting
roejobs-pipeline cancel <id>                     # cancel every currently in-flight step; steps not yet started never run

Every job spawned by a pipeline is a completely ordinary job as far as the rest of roejobs is concerned — it shows up in the dashboard/TUI//jobs with its normal uid, and cancel/restart/remove/CPU+RAM monitoring all work on it unchanged. It's additionally tagged with pipeline_id/pipeline_stage (visible in /jobs and /jobs/<uid>) so you can tell which pipeline run it came from.

Pipeline API

Method	Path	Description
POST	/pipelines	Submit a pipeline: {"name": ..., "root": <recursive step, see below>}.
GET	/pipelines	List all pipelines.
GET	/pipelines/<id>	Full step tree with each step's status (and job uid, for job steps).
POST	/pipelines/<id>/cancel	Cancel every currently in-flight step; steps not yet started never run.

A step in the JSON body is one of:

{"type": "job", "cmd": [...], "cwd": "...", "env": {...}, "name": "..."}
{"type": "parallel", "children": [<step>, ...], "name": "..."}
{"type": "sequence", "children": [<step>, ...], "on_stage_failure": "halt", "name": "..."}

roejobs-pipeline compiles the YAML above into exactly this shape client-side before posting — the same way roejobs-cli already resolves task-file lines before POSTing to /jobs. The server never parses YAML.

A caveat worth knowing: pipelines have no persistence, same as everything else in roejobs right now (see Notes below). A multi-hour pipeline is exactly the case where that costs the most — a server restart mid-pipeline would leave any already-running job running headless, with nothing left to queue what comes after it.

Notes

• Roejobs is local-only and designed for single-machine usage.
• The web interface is unprotected, so only bind to 127.0.0.1.
• Logs for each job are stored in temporary directories and accessible from the web UI.