symfluence 0.8.5

SYnergistic Modelling Framework for Linking and Unifying Earth-system Nexii for Computational Exploration

SYMFLUENCE

SYnergistic Modelling Framework for Linking and Unifying Earth-system Nexii for Computational Exploration

---

Overview

SYMFLUENCE is a computational environmental modeling platform that streamlines the hydrological modeling workflow—from domain setup to evaluation. It provides an integrated framework for multi-model comparison, parameter optimization, and automated workflow management across spatial scales.

---

Quick Links

• Install: pip install symfluence or uv pip install symfluence
• Documentation: symfluence.readthedocs.io
• Website: symfluence.org
• Discussions: GitHub Discussions
• Issues: GitHub Issues

---

Installation

Quick Start (Recommended)

Option 1: pip

pip install symfluence

Option 2: uv (Fast Python installer)

# Into current environment
uv pip install symfluence

# As an isolated CLI tool
uv tool install symfluence

Option 3: pipx (Isolated CLI)

pipx install symfluence

After installation, install external model binaries:

symfluence binary install

Development Installation

# Clone repository
git clone https://github.com/symfluence-org/SYMFLUENCE.git
cd SYMFLUENCE

# Use built-in installer
./scripts/symfluence-bootstrap --install
source venv/bin/activate

# Build model binaries (bootstrap installs Python deps but not binaries)
symfluence binary install

This creates a clean Python 3.11 virtual environment and installs dependencies. The separate binary install step compiles model binaries (SUMMA, FUSE, NGEN, etc.) from source. For detailed instructions (ARC, FIR, Anvil, custom builds), see the installation guide.

npm (Optional — Experimental)

The npm package bundles pre-built binaries (SUMMA, mizuRoute, FUSE, NGEN, TauDEM) for supported platforms:

npm install -g symfluence

# Verify bundled binaries
symfluence binary info

# Check system compatibility
symfluence binary doctor

Supported platforms:

• Linux: Ubuntu 22.04+, RHEL 9+, or Debian 12+ (x86_64)
• macOS: macOS 12+ (Apple Silicon M1/M2/M3)

Note: The npm package bundles pre-built model binaries and auto-installs the Python CLI via its postinstall hook (requires Python 3.11+ and pip/uv on the system). The Python package (pip/uv) is the primary installation method.

System Requirements

• Build dependencies: See the installation guide at https://symfluence.readthedocs.io/en/latest/installation.html
• npm installation: See tools/npm/README.md for platform-specific requirements

System Dependencies (Important)

SYMFLUENCE requires several system-level libraries that must be installed before pip installation:

GDAL (Required)

GDAL is a complex geospatial library that requires system-level installation. The Python bindings (gdal package) will fail to install if the system library is not present.

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y gdal-bin libgdal-dev
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal

# macOS (Homebrew)
brew install gdal

# Windows (conda recommended)
conda install -c conda-forge gdal

# Verify installation
gdalinfo --version

Other System Libraries

# Ubuntu/Debian
sudo apt-get install -y libnetcdf-dev libhdf5-dev libproj-dev libgeos-dev

# macOS (Homebrew)
brew install netcdf hdf5 proj geos

# Windows
# Use conda-forge channel for these dependencies
conda install -c conda-forge netcdf4 hdf5 proj geos

R (Required for rpy2)

Some hydrological models require R integration via rpy2:

# Ubuntu/Debian
sudo apt-get install -y r-base r-base-dev

# macOS
brew install r

# Windows
# Download and install from https://cran.r-project.org/

Troubleshooting

If you encounter GDAL installation issues:

1. Ensure GDAL system library version matches the Python package version
2. On Windows, prefer conda installation over pip for geospatial packages
3. Run symfluence binary doctor to diagnose system dependencies

macOS Apple Silicon (M1/M2/M3):

# Recommended: use Homebrew
brew install gdal
pip install gdal==$(gdal-config --version)

# Alternative: use conda-forge
conda install -c conda-forge gdal geopandas rasterio

Windows:

# Use conda-forge for all geospatial dependencies
conda create -n symfluence python=3.11
conda activate symfluence
conda install -c conda-forge gdal geopandas rasterio netcdf4 hdf5
pip install symfluence

For detailed troubleshooting, see the installation guide

---

Containerised installs

The docker/ directory contains a Dockerfile per documented install method. Each method has two files:

• Dockerfile — the install method as described in the docs, verbatim. No workarounds. Useful for reproducing what a user following the README/installation.html would actually get.
• Dockerfile.fixed — a copy with the minimum set of workarounds needed for a fully-working image. Each workaround is annotated with the upstream bug it papers over so it can be removed once fixed upstream.

Method matrix

All eight install paths build successfully on both linux/amd64 and linux/arm64. Numbers count model binaries produced by symfluence binary install (out of 12, except npm which bundles a different prebuilt set).

Method	Base image	Dockerfile (doc-faithful)	Dockerfile.fixed
pip	python:3.11-slim-bookworm	builds; 11/12¹	builds; 11/12¹
uv	python:3.11-slim-bookworm	builds; 11/12¹	builds; 11/12¹
uv-tool	python:3.11-slim-bookworm	builds; 11/12¹	builds; 11/12¹
pipx	python:3.11-slim-bookworm	builds; 11/12¹	builds; 11/12¹
npm	node:20-bookworm-slim	builds; runtime-only (prebuilt binaries)	21/23 binaries (ngiab + troute not bundled by npm)
conda	condaforge/miniforge3:24.11.3-2	builds; 11/12¹	builds; 11/12¹ (single-source conda toolchain)
source	python:3.11-slim-bookworm	builds; 11/12¹ (bootstrap stage; clones upstream main)	builds; 12/12 (manual stage; installs from local checkout, picks up in-development ngen fixes)

¹ The 1-of-12 gap is ngen, which the wrapper marks as failed despite the binary being built. This is a known false-negative in published symfluence's _build.sh — fixed on a development branch and landing in the next release, at which point every row above reports 12/12.

Build & run

docker-compose.yaml defines one service per install method, each wired to its Dockerfile.fixed. Run from the repo root:

# pip
docker compose build pip
docker compose run --rm pip --help

# uv
docker compose build uv
docker compose run --rm uv --help

# uv tool (isolated CLI)
docker compose build uv-tool
docker compose run --rm uv-tool --help

# pipx (isolated CLI)
docker compose build pipx
docker compose run --rm pipx --help

# npm (pre-built binaries, no compilation; service pins platform: linux/amd64)
docker compose build npm
docker compose run --rm npm --help

# conda (Windows install path / macOS ARM64 GDAL workaround)
docker compose build conda
docker compose run --rm conda --help

# source — bootstrap from upstream clone
docker compose build source
docker compose run --rm source --help

# source — manual editable install of local checkout (Dockerfile target: manual)
docker compose build source-manual
docker compose run --rm source-manual --help

Building for linux/amd64 on Apple Silicon

A quick terminology note, because the names are confusing:

• arm64 (aka aarch64) — ARM 64-bit. Apple Silicon (M1/M2/M3/M4), AWS Graviton, Raspberry Pi.
• amd64 (aka x86_64) — Intel/AMD 64-bit. Intel Macs, most Linux servers and HPC nodes, most cloud VMs, GitHub Actions runners. The "AMD" in the name is historical; Intel chips use the same instruction set.

Your Apple Silicon Mac is arm64. Docker images are architecture-specific, so by default docker compose build produces an arm64 image on Apple Silicon — fast, native, no emulation. That's the right choice for day-to-day local work on pip / uv / uv-tool / pipx / conda / source.

You'd want to override to linux/amd64 (which Docker Desktop runs under QEMU emulation, so noticeably slower) in these cases:

• Reproducing what CI sees — GitHub Actions runners are linux/amd64.
• Matching what end users actually deploy — most production Linux (cloud VMs, HPC, university workstations) is amd64.
• Running the npm install path — the published prebuilt tarballs only exist for linux/amd64 + darwin/arm64. There's no linux/arm64 tarball, so the npm container has to be amd64 (this is why the npm service already pins amd64 in the base file).
• Cross-arch validation before a release — your Mac is arm64; emulated amd64 fills in the other half of the test matrix.

To force amd64, layer docker-compose.amd64.yaml on top of the base file. It pins every service to platform: linux/amd64:

# One-off per command:
docker compose -f docker-compose.yaml -f docker-compose.amd64.yaml build pip
docker compose -f docker-compose.yaml -f docker-compose.amd64.yaml run --rm pip --help

# Or set COMPOSE_FILE once for the shell session:
export COMPOSE_FILE=docker-compose.yaml:docker-compose.amd64.yaml
docker compose build pip
docker compose run --rm pip --help

Source-compile methods (source, source-manual) take 30+ minutes natively and considerably longer under QEMU — plan accordingly.

Which one should I use?

• Most users: pip-fixed or uv-fixed. Both produce 12/12 working binaries on Linux. uv is faster to install.
• Need pre-compiled binaries (no host build toolchain): npm-fixed. Linux x86_64 and macOS ARM64 only.
• Developing on the project: source --target manual so the venv contains an editable install of your local checkout.
• Avoid for now: the unmodified Dockerfile files. They are kept as a record of what the docs literally say; they're not meant to be used directly.

---

Quick Start

Basic CLI Usage

# Show options
symfluence --help

# Run full workflow
symfluence workflow run --config my_config.yaml

# Run specific steps
symfluence workflow steps setup_project calibrate_model

# Define domain from pour point
symfluence project pour-point 51.1722/-115.5717 --domain-name MyDomain --definition semidistributed

# Check workflow status
symfluence workflow status

# Validate configuration
symfluence config validate --config my_config.yaml

First Project

# Initialize project from template
symfluence project init

# Or copy template manually
cp src/symfluence/resources/config_templates/config_template.yaml my_project.yaml

# Run setup
symfluence workflow step setup_project --config my_project.yaml

# Run full workflow
symfluence workflow run --config my_project.yaml

---

Python API

For programmatic control or integration:

from pathlib import Path
from symfluence import SYMFLUENCE

cfg = Path('my_config.yaml')
symfluence = SYMFLUENCE(cfg)
symfluence.run_individual_steps(['setup_project', 'calibrate_model'])

---

Configuration

YAML configuration files define:

• Domain boundaries and discretization
• Model selection and parameters
• Optimization targets
• Output and visualization options

See src/symfluence/resources/config_templates/config_template.yaml for a full example.

---

Project Structure

SYMFLUENCE/
├── src/symfluence/           # Main Python package
│   ├── core/                 # Core system, configuration, mixins
│   ├── cli/                  # Command-line interface
│   ├── project/              # Project and workflow management
│   ├── data/                 # Data acquisition and preprocessing
│   ├── geospatial/           # Domain discretization and geofabric
│   ├── models/               # Model integrations (SUMMA, FUSE, GR4J, etc.)
│   ├── optimization/         # Calibration algorithms (DDS, DE, PSO, NSGA-II)
│   ├── evaluation/           # Performance metrics and evaluation
│   ├── reporting/            # Visualization and plotting
│   └── resources/            # Configuration templates and base settings
├── examples/                 # Progressive tutorial examples
├── docs/                     # Sphinx documentation source
├── scripts/                  # Build and release scripts
├── tools/                    # NPM packaging and utilities
└── tests/                    # Unit, integration, and E2E tests

---

Branching Strategy

• main: Stable releases only — every commit is a published version.
• develop: Ongoing integration — merges from feature branches and then tested before release.
• Feature branches: feature/<description>, PR to develop.

---

Contributing

See CONTRIBUTING.md for:

• Code standards and testing
• Branching and pull request process
• Issue reporting

---

License

Licensed under the GPL-3.0 License. See LICENSE for details.

Commercial Licensing

SYMFLUENCE is free and open-source software under GPL-3.0-or-later. For organizations that require alternative licensing terms — including proprietary integration, redistribution without copyleft obligations, or operational deployment support — commercial licenses are available.

For commercial licensing, derivative-platform inquiries, and the Foundation's dual-licensing policy, see LICENSING.md.

Contact: [e-mail] (licensing) · [e-mail] (general)

---

Happy modelling! The SYMFLUENCE Team