ox_proc

Launch, monitor, and clean up detached external processes with simple on-disk status tracking — no message broker, no persistent worker process.

ox_proc is aimed at the common case where a short-lived caller (e.g., a Flask/gunicorn request handler) needs to kick off a long-running command, return immediately, and let any later process check progress, read recent output, fetch the final result, or kill the run.

How it works

Each launch gets a run ID like myslug-20260611T143022-a1b2 and a run directory:

<base_dir>/<slug>/<run_id>/
    status.json     written by the launcher; finalized by observers
    messages.jsonl  appended by the child via post_message()
    result.json     written by the child via write_result()
    stdout.log      child's stdout
    stderr.log      child's stderr

The child is launched in its own session (setsid), so it survives the launcher's exit. Liveness checks verify both the PID and the process creation time recorded at launch, so a reused PID is never mistaken for the original process. No file locking is needed: status.json and result.json use atomic write-and-rename, and messages.jsonl is append-only.

Quick start

Launcher side (e.g., in a web request handler):

import ox_proc

run_id = ox_proc.launch(
    ["python", "-m", "myproject.analysis", "--full"],
    slug="analysis",
    description="Nightly full analysis",
    env_updates={"ANALYSIS_MODE": "full"},
    max_live=1,            # raise TooManyLiveError if one is running
)

Child side (inside the launched command):

import ox_proc

ox_proc.post_message("loading data", progress=0.1)
...
ox_proc.write_result({"rows": 12345, "ok": True})   # always on success

Status side (any process, any time):

info = ox_proc.get_info(run_id)
info["state"]          # "running", "finished", "died", or "killed"
info["messages"]       # recent post_message() records
info["stdout_tail"]    # last lines of stdout
info["result"]         # contents of result.json, or None

ox_proc.count_live()   # {"analysis": 1, ...}
ox_proc.kill_run(run_id)
ox_proc.cleanup()      # call periodically: kills over-runtime runs,
                       # deletes expired finished runs

Conventions and caveats

• Always call write_result() on success. Because the child is detached, its true exit code is lost; a dead process with no result.json is reported as state "died".
• End times are observed: recorded when an observer first notices the process is gone. TTL-based deletion counts from that time, with a backstop (default 24 h from launch) for runs whose end was never observed.
• Live runs are killed by cleanup() once they exceed their per-run max_runtime_seconds (default 8 h; pass None for unlimited).
• kill_run() sends SIGTERM to the whole process group; there is no SIGKILL escalation.
• The default base directory is tempfile.gettempdir()/ox_proc-<username>, which the OS may purge (reboots, tmpfiles cleaning) — run history is not durable. Pass base_dir= everywhere for a durable location.
• POSIX only (relies on sessions/process groups and atomic appends).
• The max_live limit is best-effort: two simultaneous launches can race past it.

Installation

pip install ox_proc

Requires Python 3.9+ and psutil.

Development

pip install -e ".[dev]"
pytest

To build for pypi, first build via

pip install --upgrade build twine
python -m build

and then push to test pypi via:

twine upload --repository testpypi dist/*

and finally if that looks good you can upload to the production pypi via

twine upload dist/*