sflow

Install

uv venv --python python3 && source .venv/bin/activate
uv pip install "nv-sflow @ git+https://github.com/NVIDIA/nv-sflow.git@main"
sflow --version

Already on sflow v0.3.0+? Upgrade in place — no incantation to remember:

sflow upgrade                 # latest main; --branch <ref> / --dry-run also supported

No uv? pip install uv (or curl -LsSf https://astral.sh/uv/install.sh | sh). Optional extras: sflow[s3] (S3 uploads), sflow[monitor] (PNG hardware charts). Contributing? Skip to Development Setup for the editable install.

What is sflow

A declarative workflow descriptor for massive GPU clusters that separates what to deploy from where to deploy it.

One semantic across every platform — backend agnostic by design. A deployment's logic never changes: start etcd and NATS, launch a frontend, spin up workers, run the benchmark. Only the infrastructure glue does — and today that glue is rewritten from scratch for every platform. sflow consolidates it into a single portable YAML: tasks, dependencies, resources, and launch methods. The same sflow.yaml runs on Docker, Slurm, and Kubernetes — swap the backend fragment, rewrite nothing. Backends delegate to each platform's native ecosystem (srun/MPI, docker run, pods and MPI jobs) rather than reimplementing it.

Cluster-level orchestration at scale. Topology-aware node and GPU placement, multi-node replicas and sweeps, readiness/failure probes, and batch submission — so one descriptor drives hundreds of GPUs instead of a pile of hand-written bash.

All four backends ship today: local, docker, slurm, and kubernetes (k8s / k8s_mpi). It is light enough to write and debug a recipe on your laptop and submit that same file to the cluster.

Define what to run in a sflow.yaml — tasks, dependencies, how to launch each task, and required resources. sflow executes the DAG in order, collects logs, and organizes outputs into a consistent directory structure. Example of a dynamo PD disaggregation LLM inference service workflow:

What it's for

Any workload that runs in steps that depend on each other: build → train → evaluate, start a service → wait for it → drive load against it → collect results, stage data → process shards in parallel → merge. If you would otherwise write a shell script that launches things in order, polls until something is ready, and tidies up afterwards, that script is a workflow — and sflow already does the tedious parts.

Distributed LLM inference is the hardest case it is designed around, not the only one: a dozen processes across several nodes that must come up in the right order, find each other, share GPUs correctly, and be torn down cleanly when any one of them dies. Whatever survives that tends to handle simpler pipelines comfortably.

The parts you don't write again

Each of these is a YAML field, not a page of bash:

Instead of hand-rolling…	You declare
until curl -sf …; do sleep 5; done	probes.readiness (HTTP / TCP / log-watch) with a timeout
retry loops and backoff arithmetic	retries: {count, interval, backoff}
stray-process cleanup, trap handlers	teardown runs on success, failure and Ctrl-C
mkdir -p logs/$(date +%s), tee, scattered output	one run directory per workflow, a log per task, a run summary
grep/awk over logs into a CSV	result: parsing (regex, named patterns, or JSON) → result.json
aws s3 cp sprinkled through the script	uploads: per task on COMPLETED, credentials from boto3's chain
copying config/data files onto nodes by hand	artifacts: staged and mounted at the same path on every backend
for cfg in …; do sbatch …; done	replicas:, or batch over a CSV of parameter rows
hard-coding hosts, ports and device ids	${{ }} expressions over variables, backend nodes and task services

Plus a live TUI, --dry-run for the whole plan, and per-task GPU/CPU/network sampling — none of which you have to build.

One recipe, laptop to cluster

At cluster scale:

• Multi-node placement — resources.nodes (indices / count / exclude) and resources.gpus (count, or indices to pin exact device ids) pack tasks onto specific nodes and devices, with conflict detection at plan time rather than a failure ten minutes into the run. On Slurm the plan is re-applied inside the step by GPU UUID, so slurmstepd and container renumbering cannot silently land every rank on device 0.
• Real ordering — depends_on waits for a process to start; readiness probes (HTTP, TCP, log-watch) make dependents wait for a service to actually be up. required_by expresses the reverse edge so a workload fragment can attach itself to a benchmark.
• Distributed launchers — srun with MPI/pmix on Slurm, k8s and k8s_mpi on Kubernetes, where a multi-node task becomes one pod per reserved node (GPUs via the device plugin or DRA) and co-located GPU tasks can be merged into a single pod.
• Fleet-scale runs — replicas for fan-out, release_after so a later task can reuse a still-running server's GPUs, plus batch / bulk-input CSV sweeps over a parameter grid.
• Results you can compare — result: parsing, uploads: to S3, and a passive hardware monitor that samples GPU/CPU/network per task, with monitor.window log markers to clip a report to the phase you actually measured.
• Run output that explains itself — sflow_summary.log records planned vs. observed GPU assignment, node topology, and external command health (kubectl/srun/docker latency and failures); a stalled event loop dumps every thread's stack to loop_stalls.txt.

On one machine:

• local backend, no scheduler — the Quickstart below runs a DAG in a few seconds.
• --dry-run renders the full plan — placement, GPU slices, and the exact commands — without touching a cluster, so most recipe mistakes surface before you queue anything.
• docker backend runs the same containerized tasks on one box, reserving GPUs per task so several sflow run sessions can share a workstation without colliding.
• Composable fragments — a workload YAML pairs with a backend fragment, so the recipe you proved locally is the one you submit to the cluster. See examples/modular/backend_agnostic/.

The same workload, three platforms. examples/modular/backend_agnostic/ composes a backend fragment + a workload + a benchmark. Moving platforms means swapping one -f:

# Kubernetes
sflow run -f backends/k8s.yaml    -f workloads/dynamo_common.yaml -f workloads/dynamo_trtllm.yaml -f benchmark.yaml
# Slurm — same workload, same benchmark
sflow run -f backends/slurm.yaml  -f workloads/dynamo_common.yaml -f workloads/dynamo_trtllm.yaml -f benchmark.yaml
# One local box, containers
sflow run -f backends/docker.yaml -f workloads/sglang_serve.yaml -f benchmark.yaml

CLI at a Glance

Command	Purpose	Key Flags
sflow run	Execute a workflow	--dry-run --tui --set -f (multi-file)
sflow batch	Generate / submit sbatch scripts	--submit --bulk-input --row
sflow compose	Merge multiple YAMLs	--resolve --missable-tasks -o
sflow visualize	Render DAG graph	--format png/svg/mermaid
sflow sample	List / copy examples	--list -o
sflow skill	Export AI agent skills	--list -o
sflow upgrade	Reinstall sflow in place (alias: sflow update)	--branch --dry-run --force

Introduction

Start here:

• User docs: docs/user/intro.md
• Release notes: docs/release_notes/ — start with v0.3.1 for the current breaking changes and upgrade checklist
• PLC docs:
  • SPP: docs/plc/sflow_spp.md
  • SRD: docs/plc/sflow_srd.md
  • SADD: docs/plc/sflow_sadd.md

Quickstart

If you just want to validate the workflow engine locally (no Slurm required):

uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"

sflow run --file examples/self_contained/local/hello_world.yaml --tui

That example is the whole file — no backend block, no operator, no boilerplate:

version: "0.1"

variables:
  WHO:
    description: "who to greet"
    value: Nvidia

workflow:
  name: local_hello_world
  tasks:
    - name: hello
      script:
        - echo "Hello ${WHO}"

With nothing declared, sflow runs on the local backend with its default operator. Add a backends: block later and the same tasks run under Docker, Slurm or Kubernetes.

Development Setup

This guide will help you set up the development environment for contributing to sflow.

Prerequisites

• Python 3.10 or higher

  python --version  # Check your Python version
  

• uv (Python package installer and resolver)

  If you don't have uv installed, you can install it using:

  curl -LsSf https://astral.sh/uv/install.sh | sh
  

  Or via pip:

  pip install uv
  

Getting Started

1. Clone the repository

   git clone <repository-url>
   cd sflow
   

2. Create a virtual environment

   uv venv
   

3. Activate the virtual environment

   source .venv/bin/activate
   

4. Install the project with development dependencies

   uv pip install -e ".[dev]"
   

   This will install:

   • The sflow package in editable mode
   • All runtime dependencies (typer, pydantic, pyyaml, etc.)
   • Development tools (pytest, pytest-cov, ipython, etc.)

5. Run unit tests to validate your setup

   pytest
   

Contributing to the project

Please follow the Git Flow process defined here: https://coda.io/d/_d0pJwz5IXhP/Manage-Git-branches-and-workflows_suLlBLYx