lpjguess-runner 1.0.4

Python wrapper for LPJ-Guess experiment runner (via pythonnet + .NET 9)

lpjguess-runner

Python wrapper for LPJ-Guess experiment runner (via pythonnet + .NET 9)

Build

Prerequisites:

• .NET 9 SDK
• Python 3.11 (if building wheel)

# Clone the repository with submodules
git clone --recurse-submodules [e-mail]:hie-dave/lpjg-gui.git
make wheel

Install

• Prerequisites: .NET 9 runtime

From PyPI

pip install lpjguess-runner

From Source

make install                 # install globally
make install-venv VENV=.venv # install into a local virtual environment (default: .venv)
make install-user            # install for the current user

Concepts and Workflow

This package is a thin Python wrapper around a .NET job runner for LPJ-Guess. A typical workflow configures a run, defines which simulations to execute, and optionally customises how progress and model output are handled.

• Run settings: A RunSettings object specifies the LPJ-Guess executable path, output directory, input module, CPU count, and job name.

• Simulations: A list of Simulation objects describes variations to apply to base instruction files (for example changing top-level parameters or PFT block parameters). The helper simulation(name, factors) creates these objects.

  factors should be a list of IFactor objects, which each represent a change to an instruction file parameter. This may be either a top-level parameter (e.g. npatch, wateruptake, etc) or a block parameter (e.g. sla within a PFT block, etc). These factors may be constructed via TopLevelParameter(name, value) and BlockParameter(block_type, block_name, parameter_name, parameter_value) respectively. See the examples below for details.

  Parameter typing note: All parameter values passed to TopLevelParameter(...) and BlockParameter(...) must be strings. For example, use TopLevelParameter("nindiv_max", "1") rather than TopLevelParameter("nindiv_max", 1). When constructing simulations programmatically, convert values explicitly with str(value).

• Instruction files and PFTs: A list of .ins files and PFT names to run. Each instruction file will be run once for each defined simulation.

• Progress and output handling (optional): Objects can be supplied to receive progress updates and model stdout/stderr. For convenience, this package exports Python base classes that already implement the required .NET interfaces, so only the required methods need to be overridden:

  • CustomProgressReporter.ReportProgress(percent, elapsed, ncomplete, njob)
  • CustomOutputHelper.ReportOutput(jobName, output) / ReportError(...)

  See the "Customising Progress and Output" section below for more details.

• Outputs: Files are written under the output directory specified in RunSettings. The structure depends on the LPJ-Guess configuration and runner settings.

• Result: run_simulations(...) returns an ExperimentResult with summary counts (TotalJobs, SuccessfulJobs, FailedJobs) and Error if present.

Notes for Python users:

• When using custom progress/output handlers, the overriden hooks may be called from background threads; handlers should therefore be fast and thread-safe.

• If only console output is required, use ConsoleProgressReporter() and ConsoleOutputHelper() and skip writing custom classes.

• To customise behavior, subclass the provided bases rather than implementing .NET interfaces directly.

• Parameter values for TopLevelParameter and BlockParameter are string typed. Convert numeric or boolean values with str(...) when building simulation lists programmatically.

Example Usage

from lpjguess_runner import *

run_settings = RunSettings.Local(
    "/path/to/guess/executable",
    "/path/to/output/directory",
    "nc",   # input module
    4,      # cpu count
    "job_name",
    True)   # Whether to allow context switching of LPJ-Guess processes between CPUs (recommended: True)

simulations = [
    simulation("nindiv_max_0_sla_26", [               # Run all .ins files with:
        TopLevelParameter("wateruptake", "wcont"),    # - wateruptake = wcont
        BlockParameter("pft", "MRS", "sla", "26")     # - SLA of "MRS" pft set to 26
    ]),
    simulation("nindiv_max_1_sla_39", [               # Run all .ins files with:
        TopLevelParameter("wateruptake", "rootdist"), # - wateruptake = rootdist
        BlockParameter("pft", "MRS", "sla", "39")     # - SLA of "MRS" pft set to 39
    ]),
]

ins = ["/path/to/file1.ins", "/path/to/file2.ins"]
pfts = ["MRS"]

result = run_simulations(run_settings,
                         simulations,
                         ins,
                         pfts,
                         ConsoleProgressReporter(),   # Write progress messages to stdout
                         ConsoleOutputHelper(),        # Propagate subprocess output to stdout
                         ExistingOutputPolicy.CleanManaged)

print(f"Total jobs: {result.TotalJobs}")
print(f"Successful jobs: {result.SuccessfulJobs}")
print(f"Failed jobs: {result.FailedJobs}")
print(f"Error: {result.Error}")

Existing Output Files

LPJ-Guess outputs are written into per-simulation directories under the run output directory. When an experiment is rerun, those directories may already contain files from an earlier run. This can make results ambiguous: a file left behind by an old configuration can look like it was produced by the latest run, and outputs from simulations that have since been removed from the experiment can remain beside current results.

The existing-output policy controls how the runner handles these cases before starting new jobs. run_simulations(...) accepts the policy as its final argument. The default is ExistingOutputPolicy.CleanManaged, which is usually the right choice for rerunning experiments because it deletes old files for simulations that are about to be regenerated/rerun.

Policies are flags and may be combined with |:

• ExistingOutputPolicy.Preserve: leave existing outputs untouched.
• ExistingOutputPolicy.CleanManaged: remove files for simulations that are about to be rerun.
• ExistingOutputPolicy.PruneStale: remove files from previous simulations in this output directory that are not part of the current run.
• ExistingOutputPolicy.Fail: abort if existing output directories are found.

For example, to clean rerun simulations and remove stale managed outputs:

policy = ExistingOutputPolicy.CleanManaged | ExistingOutputPolicy.PruneStale
result = run_simulations(run_settings, simulations, ins, pfts,
                         ConsoleProgressReporter(),
                         ConsoleOutputHelper(),
                         policy)

Managed outputs are outputs tracked by the runner's result catalog.

Customising Progress and Output

The default helpers ConsoleProgressReporter() and ConsoleOutputHelper() propagate progress and model stdout/stderr to stdout. For custom behavior, subclass the Python base classes exported by this package:

• CustomProgressReporter (implements .NET IProgressReporter).
• CustomOutputHelper (implements .NET IOutputHelper).

These hide pythonnet interop details so only the necessary methods must be overridden in subclasses.

Example: capture stdout/stderr and print compact progress

from lpjguess_runner import *
import threading

class MyOutput(CustomOutputHelper):
    def __init__(self):
        super().__init__()
        self.stdout = {}
        self.stderr = {}
        self._lock = threading.Lock()

    def ReportOutput(self, jobName, output):
        with self._lock:
            self.stdout.setdefault(jobName, []).append(output)

    def ReportError(self, jobName, output):
        with self._lock:
            self.stderr.setdefault(jobName, []).append(output)

class MyProgress(CustomProgressReporter):
    def ReportProgress(self, percent, elapsed, ncomplete, njob):
        print(f"[{elapsed}] {ncomplete}/{njob} ({percent:.1f}%)")

out = MyOutput()
pr = MyProgress()

result = run_simulations(run_settings, simulations, ins, pfts, pr, out)

Notes

• Methods may be called from background threads; keep handlers fast and thread-safe.
• Output can be frequent; consider buffering or filtering.
• Advanced: if implementing the .NET interfaces directly instead of subclassing these bases, the class must inherit from System.Object, call Object.__init__, and set a valid __namespace__ (for example "LpjGuess.Runner.Python"). Using the provided base classes is recommended.

Programmatic construction of simulations

Many workflows define a small grid of parameter values and generate one simulation per combination. The following example shows a minimal pattern using itertools.product. Note the explicit conversion of parameter values to strings.

from itertools import product
from lpjguess_runner import simulation, TopLevelParameter, BlockParameter

param_grid = {
    "wateruptake": ["wcont", "rootdist"],
    "npatch": range(1, 50, 10)
}

sims = []
for (wu, npatch) in product(param_grid["wateruptake"], param_grid["npatch"]):
    factors = [
        TopLevelParameter("wateruptake", str(wu)),
        TopLevelParameter("npatch", str(npatch))
    ]
    name = f"wu_{wu}_npatch_{npatch}"
    sims.append(simulation(name, factors))
# sims now contains one Simulation per combination.

for sim in sims:
    print(f"# {sim.Name}")
    print("\n".join([f"- {c.GetName()}" for c in sim.Changes]))
    print()

Results and Outputs

run_simulations(...) returns an ExperimentResult with at least:

• TotalJobs
• SuccessfulJobs
• FailedJobs
• Error

Files are written under the output directory provided to RunSettings.Local. If jobs produce per-run subdirectories, those will appear under that directory.