nirs4all-cluster

Status: beta · trusted-LAN. A small distributed job queue for nirs4all.run(), built for a trusted local network — not the open internet or untrusted multi-tenant use. The documented non-goals and security posture still hold (a single static token, no sandbox, a single SQLite server). See SECURITY.md, PROTOTYPE_DESIGN.md (the design source of truth) and PROTOTYPE_TO_PRODUCTION.md. The broader question of a native cluster vs. an opt-in Dask backend in nirs4all remains open.

📖 Docs: https://nirs4all-cluster.readthedocs.io · 🖥️ Dashboard: http://HOST:8765/ui

Distributed execution ofnirs4allpipelines (client / server / workers): a coordinator receives jobs and dispatches the work to workers that poll the server. The prototype does not modify any other library in the ecosystem:nirs4allis only imported by the runner subprocess, and the server/client works without it.

What it does

• Submission of anirs4all.run()job via Python SDK or CLI. - FastAPI server + SQLite file + local object store addressed by SHA-256. - Workers polling (long-polling HTTP + heartbeat), with a task sandbox per folder. - Atomic job (Level 0) andpipelines × datasetsdecomposition (Level 1) with aggregation/ranking. - Download artifacts: JSON summary, logs, best.n4amodel. - Recovery after worker crash (lease + retry), cooperative cancellation, idempotence. - Routing by capabilities: labels, memory, package versions (PEP 440), GPU/CUDA (auto-detected,requirements.min_gpu_countorcuda=truelabel). Anirs4all.runjob requiresnirs4allby default. - Live web dashboard at /ui (jobs, workers, events; cancel) over a global WebSocket stream. - Version compatibility tracking: client/server/worker advertise their nirs4all-cluster + protocol version; incompatible protocol majors are rejected, compatible drift is logged/traced. Pipeline content fingerprints recorded end-to-end. - Richer job listing/filtering (n4cluster jobs --status … --name …, GET /v1/stats) and structured logging (--log-level, --log-file).

Installation

# Worker environment = an existing nirs4all environment + this package:
uv pip install -e .            # serveur + client + transport worker
# (workers provide nirs4all themselves; it is not a hard dependency)

Python ≥ 3.11. The server and client only need FastAPI/uvicorn/httpx/pydantic; only the worker needs a provisionednirs4allenvironment.

Quickstart (LAN de confiance)

# 1) serveur
n4cluster server --host 0.0.0.0 --port 8765 --state ./cluster-state

# 2) one or more workers (on machines that can see nirs4all and the dataset)
#    The worker auto-detects GPUs (nvidia-smi) and declares the cuda + gpu_count labels;
#    force with --gpus N (0 to hide GPUs).
n4cluster worker --server http://HOST:8765 --labels site=lab --slots 1

# 3) submit a job and wait for the result
n4cluster submit examples/job.shared-path.yaml --wait --out ./results
n4cluster status   <job_id>
n4cluster jobs     --status running        # filter the job list
n4cluster logs     <job_id>
n4cluster cancel   <job_id>
n4cluster artifacts <job_id> --out ./results

# 4) watch everything live in the browser
#    open http://HOST:8765/ui

SDK Python :

from nirs4all_cluster import ClusterClient

client = ClusterClient("http://host:8765", token=None)
job = client.submit_run(
    pipeline="/shared/pipelines/pls.yaml",                 # kind=path
    dataset="/shared/datasets/corn",                       # kind=shared_path
    params={"random_state": 42, "refit": True},
)
job = client.wait(job.id)
print(job.aggregate.best_metric, job.aggregate.ranking)
client.download_best_model(job.id, "best_model.n4a")

Architecture

submitter (SDK/CLI/Studio) ──REST + WS──► serveur (FastAPI + SQLite + object store + scheduler + events)
                                              ▲
                          long-polling HTTP + heartbeat
                                              │
                                          workers ──► subprocess runner ──► nirs4all.run(workspace=task_ws)

• nirs4all_cluster/server/ —app.py(API),db.py(SQLite file, atomic leasing, reaper),scheduler.py(state machines + matching),artifacts.py(SHA-256 store),events.py(broker). - nirs4all_cluster/worker/ —agent.py(polling loop),materialize.py(resolution of references → local paths),executor.py(subprocess + capture + undo). - nirs4all_cluster/runners/nirs4all_run.py — only module that importsnirs4all. - client.py (SDK), cli.py (n4cluster), schemas.py (Pydantic contract).

Tests and validation

pytest -q                                   # unit/API tests without nirs4all + skipped integration if unavailable
pytest -q tests/test_release_smoke.py       # wheel + installed CLI/server/worker smoke, no nirs4all
python scripts/validation.py                # end-to-end harness on nirs4all-data (8/8)

Results measured onnirs4all-data(see WORKLOG.md): atomic job →.n4a, 2 workers in parallel, kill worker → retry, cancellation not restarted,pipeline × datasetaggregation, and exact metric parity vs localnirs4all.run()(diff = 0.0) — beyond the criterion go/no-go ≤ 1e-10.

Go/no-go criteria to upgrade to product

The go remains conditional on all of these conditions:

1. ≥ 2 labs/partners explicitly request distributed execution. (not measurable here)
2. Speedup ≥ 3× on a real workload (grid search AOM / HPO on ≥ 32 datasets). (to be measured)
3. Metric-identical results (≤ 1e-10) with single-machine. → reached: diff = 0.0 on the atomic job. 4. Data + security + recovery model written before the code. → done inPROTOTYPE_DESIGN.md. 5. Framing topics covered from the start (mTLS, secrets, third-party sandboxing, IP/GDPR datasets, heavy TF/Torch/JAX environments, transfer costs, idempotence/resumption, quotas/fairness, heterogeneous scheduling). → listed inPROTOTYPE_TO_PRODUCTION.md.

These criteria gate a full production commitment (mTLS, sandboxing, Postgres, network object storage — see PROTOTYPE_TO_PRODUCTION.md). Until they are met, the ecosystem's default option remains an opt-in Dask backend in nirs4all; this repository is published as a usable trusted-LAN beta and an auditable reference for the design, not as a production roadmap commitment.

Non-objectifs (rappel)

No modification of other libs, no open multi-tenant, no sandbox for Python code arbitrary, no K8s/Ray/Dask type scheduler, no concurrent writing in a workspacenirs4allshared, no distribution of folds. SeePROTOTYPE_DESIGN.md§ Non-objectives.

References

PROTOTYPE_DESIGN.md,PROTOTYPE_TO_PRODUCTION.md,WORKLOG.md, andnirs4all-ecosystem/NIRS4ALL-ECOSYSTEM_VISION.md(annex Perspective: distributed execution, risk R13).

License

nirs4all-cluster is dual-licensed open-source — CeCILL-2.1 OR AGPL-3.0-or-later (your choice) — with an optional commercial license for closed-source / SaaS use. For any commercial use, contact nirs4all-admin@cirad.fr. See LICENSING.md, the texts under LICENSES/, and THIRD_PARTY_NOTICES.md.