FlowX

Python-native, restart-first workflow engine for scientific HPC computing

---

Why FlowX?

FlowX is built for the reality of HPC scientific computing:

• Air-gapped clusters with strict schedulers
• Preemption and walltime limits that kill long jobs
• Mixed environments (Nix, modules, conda)
• Filesystem as truth (no database required)

The FlowX Advantage

Feature	FlowX	AiiDA	Jobflow
Setup time	5 min	90-180 min	45 min
Restart granularity	Phase-level	Job-level	Job-level
Database required	No	Yes (PostgreSQL)	Optional
Air-gap ready	Yes	Limited	Limited
Walltime handling	Cooperative requeue	Manual	Manual

---

Quickstart

pip install git+https://gitlab.com/siestastudio/flowx.git
flowx configure siesta.command /path/to/siesta   # point FlowX at your engine
flowx pseudo download -e Si                      # fetch pseudopotentials

A single calculation:

from flowx import run
from flowx.engines import siesta

job = siesta.scf("Si.cif", tier="standard")
result = run(job)

print(result.parsed.total_energy_eV)
print(result.parsed.scf_converged)

A workflow -- plain Python, one decorator:

from flowx import run, task, workflow
from flowx.engines import siesta

@task
def relax_structure(structure, tier="standard"):
    job = siesta.relax(structure, tier=tier)
    return run(job)

@workflow
def study(structure):
    return relax_structure(structure)

result = study("Si.cif")

Then watch it from the shell:

flowx status                 # every run, with phase and state
flowx logs <run_id> -f       # follow output live
flowx workflow list          # workflows and their nodes

---

Core Philosophy

1. Filesystem is Truth

Every run is a self-contained folder. No database required for correctness.

runs/relax-Si-20250130-143052/
├── FlowX-runspec.json       # Immutable job intent
├── FlowX-state.json         # Phase tracking (atomic, updated per phase)
├── FlowX-provenance.md      # Human-readable audit trail
├── FlowX-events.jsonl       # Append-only event log
├── FlowX-results.json       # Parsed results (typed schema)
├── siesta.fdf         # Generated engine input
├── siesta.out         # Raw program output
└── ...                # Other engine-native files (.XV, .DM, .bands, …)

2. Restart-First

Every phase checkpoints. Resume is trivial.

# Job gets killed after 3 hours? No problem.
flowx restart runs/relax-Si-20250130-143052
# Continues from last completed phase

Time savings: 5-50x on failures compared to restarting from scratch.

3. Decoupled Architecture

Submitter ≠ Runner ≠ Database. Air-gap friendly.

Laptop → JobBundle → Cluster Inbox → Runner → Results

4. Pure Python DSL

No YAML. No schema ceremony. Just Python.

@task
def phonon_workflow(structure):
    relaxed = relax(structure)
    displacements = generate_displacements(relaxed["structure"])
    forces = [calculate_forces(d) for d in displacements["structures"]]
    return compute_phonons(relaxed["structure"], forces)

---

Supported Engines

• SIESTA (native, primary engine)
• Quantum ESPRESSO (native)
• VASP (native)
• exciting (native) -- all-electron full-potential LAPW+lo, so its numbers share none of the pseudopotential approximations the other three rest on
• GPAW (transpiler skeleton -- no executor)
• CP2K, CASTEP, ORCA (planned)

All engines use a Canonical Input Representation (CIR) for portability:

from flowx.cir import CanonicalInput, Structure, Kpoints, XC

cir = CanonicalInput(
    structure=Structure.from_file("Si.cif"),
    kpoints=Kpoints.gamma_centered(6, 6, 6),
    xc=XC.gga("PBE"),
    task="relax",
)

# Transpile to any engine
to_engine(cir, "siesta")  # → *.fdf files
to_engine(cir, "vasp")    # → INCAR, POSCAR, KPOINTS
to_engine(cir, "qe")      # → *.in files
# exciting takes a CIR without pseudopotentials; use exciting.scf() instead

---

Phase Machine

Every job goes through deterministic phases with automatic checkpointing:

prepare → stage → run → harvest → validate → publish

• prepare: Generate engine inputs, resolve pseudopotentials
• stage: Ensure runtime environment, preflight checks
• run: Execute code with heartbeats and cooperative requeue
• harvest: Parse logs, collect outputs (idempotent)
• validate: Check success criteria
• publish: Push artifacts to external stores

FlowX-state.json is atomically updated after each phase. On crash, flowx restart re-enters the first incomplete phase.

---

CLI

# Project
flowx init myproject                 # Initialize a new project
flowx configure --list               # Show resolved configuration

# Execution
flowx run <target>                   # Run a run_id, workflow_id, or .py script
flowx submit <wf_id> --cluster mn5   # Submit to an HPC scheduler
flowx restart <run_id>               # Restart from the last completed phase
flowx stop <run_id>                  # Graceful stop

# Monitoring
flowx status [<run_id>]              # Table of runs, or one run in detail
flowx logs <run_id> -f               # Follow output
flowx history <run_id>               # Execution timeline from the event log
flowx watch <wf_id>                  # Live workflow dashboard

# Analysis and housekeeping
flowx compare <run1> <run2>          # Diff two runs
flowx template list                  # The 20 workflow templates
flowx sweep                          # Classify the run store for retention
flowx archive <run_id>               # Archive to .tar.gz

flowx --help lists every command; most have short aliases (ls, wf, cmp).

---

HPC & Air-Gap Operation

FlowX is designed for offline, pull-based execution:

1. Prepare locally → run directories with generated engine inputs
2. Transfer via rsync/SSH (flowx submit <wf_id> --cluster <name> --zero-install)
3. The cluster-side coordinator executes them (no inbound network required)
4. Results written to the filesystem, pulled back with flowx download

Nothing is required on the cluster but the engine itself: zero-install submit ships a stdlib-only retry wrapper, and --remote-python moves the coordinator onto the cluster so the campaign survives your laptop disconnecting.

---

Installation

Not on PyPI yet. The distribution name is reserved as flowx-hpc -- the name flowx belongs to an unrelated package -- and the import name stays flowx.

# From the repository
pip install git+https://gitlab.com/siestastudio/flowx.git

# With all optional dependencies
pip install "flowx-hpc[all] @ git+https://gitlab.com/siestastudio/flowx.git"

# Development installation
git clone https://gitlab.com/siestastudio/flowx.git
cd flowx
pip install -e '.[dev]'

---

Documentation

flowx.readthedocs.io

• Getting Started
• Workflows
• CLI Reference
• Configuration
• Engine APIs -- SIESTA, QE, VASP, exciting
• Workflow Templates
• API Reference
• Runnable examples -- in this repository

Sources are under docs/source/; build them locally with make -C docs html.

---

Development Status

Current Phase: v0.1.0 — Phases 1–3 complete

• Phase machine with phase-level restart + human-readable provenance
• Local runner, Slurm/PBS submitters, SSH transport, zero-install cluster submit
• Engines: SIESTA, Quantum ESPRESSO, VASP, exciting (GPAW transpiler skeleton)
• Workflow DAG: @task/@workflow, parallel(), all_pairs(), portfolio(), loop_until(), flowx replay
• 20 analysis templates (convergence, EOS, bands, phonons, NEB, defects, surfaces, …)
• Phase 4 (scale, AI, dashboard) and Phase 5 (CP2K, full GPAW, docs site) planned

---

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

FlowX follows the Contributor Covenant Code of Conduct.

---

License

GNU General Public License v3.0 - see LICENSE for details.

DFT engines are not included

FlowX drives external DFT codes; it does not contain, bundle or redistribute any of them. Each engine must be obtained, installed and — where its own terms require it — licensed by the user.

The engine adapters are interface code: they generate each code's native input files and parse its output. No engine's source, binaries or pseudopotential data are present in this repository. In particular, the VASP adapter assembles a POTCAR by concatenating files from a pseudopotential library that the user supplies and is licensed to hold; no VASP pseudopotentials are distributed here.

Engine	Obtained from	Terms
SIESTA	siesta-project.org	GPL / open source
Quantum ESPRESSO	quantum-espresso.org	GPL / open source
exciting	exciting-code.org	GPL / open source
GPAW	gpaw.readthedocs.io	GPL / open source
VASP	vasp.at	proprietary; user must hold a valid licence

Pseudopotential libraries (PseudoDojo, SSSP, VASP PAW) are likewise downloaded or supplied by the user under their own terms and are not redistributed.

---

Citation

If you use FlowX in your research, please cite:

@software{flowx,
  title = {FlowX: Python-native workflow engine for scientific HPC},
  author = {Akhtar, Arsalan},
  year = {2026},
  version = {0.1.0},
  url = {https://gitlab.com/siestastudio/flowx}
}

---

Support

• Issues: GitLab Issues
• Chat: Discord