eta-stress 1.0.7

Library for conducting global sensitivity analyses and stress tests. Interoperable and model agnostic API. HPC suitability approved.

ETA Stress

eta-stress is the standalone stress-test orchestration package for ETA workflows. It integrates with eta-incerto through a thin adapter boundary and provides a clean Tier 1 + Tier 2 + GSA pipeline with explicit contracts and artifact persistence.

Key Features

• Tier 1 local sensitivity analysis on a solved baseline
• Tier 2 scenario generation, perturbation, and scorecard aggregation
• Tier 3 global sensitivity analysis (PAWN / Sobol) with HDF5 artifacts
• Thin adapter boundary to eta-incerto with explicit run contracts
• CLI-driven orchestration and optional plotting summaries

Overview

flowchart TD
  bootstrap[bootstrap] --> preflight[preflight]
  preflight --> variant_registry[variant_registry]
  variant_registry --> baseline[baseline solve]
  baseline --> tier1[tier1 analyze]
  tier1 --> tier2[tier2 scenarios]
  tier2 --> aggregation[tier2 aggregation]
  aggregation --> gsa[tier3 gsa]
  gsa --> provenance[provenance]

What This Package Runs

The orchestrator executes the full stress-test pipeline:

1. bootstrap: create namespace folders under run_dir.
2. preflight: validate config, mode, scenario files, series files, and variant layout.
3. variant_registry: import variant runtime module and validate registry availability.
4. baseline resolution: solve baseline through EtaBackend or import an existing baseline.
5. baseline artifacts: persist baseline.json and incumbent.npz.
6. tier1 analyze: local sensitivity analysis over solved baseline.
7. tier2 source: resolve scenarios from generate or existing.
8. tier2 runner: run per-scenario context -> perturb -> incumbent_eval -> metrics -> persist.
9. tier2 aggregation: aggregate scenario results into scorecard.
10. tier3 gsa: build factor/response data (X/Z), run PAWN/Sobol, persist gsa.h5.
11. provenance: persist final run provenance and artifact links.

Installation

poetry install

Compatibility:

• Python: >=3.11,<3.13
• eta-incerto: >=1.3.5,<2.0.0 (current package constraint: ^1.3.5)

See Solver and third-party requirements for MILP solver notes (via eta-incerto / eta-components).

CLI Usage

The CLI takes required runtime arguments plus an optional stress-config overrides file.

poetry run python -m eta_stress.cli.main \
  --run-dir /path/to/run_dir \
  --config-name config \
  --mode stochastic \
  --variant variant_one \
  --variant-import-root examples.conventional \
  --scenario-source generate \
  --stress-config /path/to/stress_test_config.json \
  --plot \
  --plot-heatmap

Required CLI arguments:

• --run-dir
• --config-name
• --mode
• --variant
• --variant-import-root
• --scenario-source (generate|existing)

Optional:

• --stress-config (policy/tolerance overrides)
• --stress-evaluation-mode (fixed_first_stage|full_reopt) overrides evaluation mode from config
• --strict-antifragile / --no-strict-antifragile toggle strict antifragile execution (no synthetic fallback when strict)
• --antifragile-pareto-selection (weighted_sum|min_first_objective) controls antifragile Pareto point selection for stress baseline
• --antifragile-objective-weights comma-separated objective weights for weighted_sum (default 1/3,1/3,1/3)
• --chapter8-price-only-uncertainty / --no-chapter8-price-only-uncertainty enforce Chapter-8 uncertainty scope
• --plot (render plotting summaries after pipeline completion)
• --plot-preset (style preset for plotting)
• --plot-heatmap (enable heatmap panel in per-scenario plotting)

Supported config file formats:

• .json
• .yml
• .yaml
• .ymal

Stress-Test Config Contract

Main model: eta_stress.orchestrator.stress_test_config.StressTestConfig.

Required fields:

• none (all fields are optional overrides now)

Important behavior:

• sensitivity_namespace is auto-generated by orchestrator using timestamp + incremental index.
• If generation_policy.scenario_probability_candidates is provided, scenario_probability_deltas is ignored.
• If neither scenario_probability_candidates nor scenario_probability_deltas is provided, probability deltas are auto-generated.
• generation_policy.max_scenarios is applied during scenario generation (early cap), so only up to that many candidates are built.
• Solver option path for MIP gap is normalized to pyomo.solver_settings.options.mipgap.
• stress_evaluation_mode controls scenario solve semantics:
  • fixed_first_stage: evaluate baseline first-stage decisions under stress. This is the default production path.
  • full_reopt: re-optimize first- and second-stage decisions under stress. This is kept for compatibility/debugging and explicit re-optimization studies.
• strict_antifragile controls antifragile fallback behavior:
  • true: fail fast if antifragile runtime/h5/incumbent requirements are not met.
  • false: allow synthetic placeholder fallback behavior for compatibility/debugging.
• antifragile_pareto_selection controls antifragile Pareto scalarization for stress baseline:
  • weighted_sum (default) uses antifragile_objective_weights (defaults to equal weights 1/3,1/3,1/3).
  • min_first_objective reproduces legacy behavior.
• chapter8_price_only_uncertainty defaults to true and enforces uncertainty channels to gas/electricity yearly unit_price only.
• rulebook_preset.allowed_carriers / rulebook_preset.allowed_fields default to ["gas","electricity"] and ["unit_price"].

Complete example config:

• eta_stress/stress_test_config.example.json

Variant Registry Import Rule

variant_registry imports:

<variant_import_root>.variants.<variant>.system

Then it validates runtime registration via eta_incerto.registry.get_system(variant).

Example:

• variant_import_root = "examples.conventional"
• variant = "variant_one"
• imported module = examples.conventional.variants.variant_one.system

Output Layout

For each run, a namespace directory is created under run_dir:

run_dir/
  sensitivity_YYYY-MM-DD_HH-MM-SSZ_XXXX/
    baseline/
      baseline.json
      incumbent.npz
      tier1_analysis.json
    scenarios/
      scenarios.json
    results/
      <scenario_id>.h5
    aggregate/
      scorecard.h5
    gsa/
      gsa.h5
    plots/
      conventional_stress_scorecard_summary.png
      conventional_stress_scorecard_summary.pdf
      conventional_stress_scorecard_evaluation.md
      conventional_stress_per_scenario_summary.png
      conventional_stress_per_scenario_summary.pdf
      conventional_stress_per_scenario_evaluation.md
    provenance.json

gsa/gsa.h5 payload contract (payload_json) is:

• gsa.factor_values
  • scenario_ids[]
  • factor_names[]
  • matrix[][]
• gsa.validation_z
  • metric_key
  • values[]
• gsa.pawn (flatten metrics map, keys gsa.pawn.*)
• gsa.sobol (flatten metrics map, keys gsa.sobol.*)

Runtime Model

eta-stress runs one sensitivity namespace per execution:

1. bootstrap
2. preflight
3. variant registry
4. baseline resolution + baseline artifacts
5. Tier 1 analysis when a solved baseline model is available
6. Tier 2 source (generate/existing)
7. Tier 2 scenario runner
8. Tier 2 aggregation
9. GSA (X/Z + PAWN/Sobol payload)
10. optional plotting (--plot)
11. provenance persistence

Chapter 8 Evidence Protocol

• Pilot stress run: generation_policy.max_scenarios = 200 for ranking/interval stability check.
• Final stress run: generation_policy.max_scenarios = 500 once pilot is stable.
• Keep Chapter-8 uncertainty scope price-only (chapter8_price_only_uncertainty=true and price-only rulebook preset).

Package Structure (High Level)

• eta_stress/orchestrator: top-level pipeline and run config loading.
• eta_stress/services/tier1: baseline local sensitivity analysis.
• eta_stress/services/tier2: scenario source, runner, and aggregation runners.
• eta_stress/services/gsa: Tier 3 GSA encoding/reports/runner.
• eta_stress/services/artifacts.py: artifact persistence (idempotent writes).
• eta_stress/adapters/eta_backend/: boundary to eta-incerto runtime operations.
  • backend.py: public EtaBackend facade.
  • baseline.py: baseline solve dispatch for deterministic, stochastic, robust, regret, and antifragile modes.
  • runtime.py: incumbent evaluation and optional full re-optimization runtime calls.
  • perturbations.py: local stress perturbation injection.
  • antifragile.py: antifragile Pareto H5 extraction and selection.
  • status.py: solver/status normalization.
• eta_stress/io: JSON/YAML/H5/path utilities.
• eta_stress/core: shared domain types, statuses, provenance, and numeric helpers.

Testing and Validation

See CONTRIBUTING.md for setup, tests, and merge request workflow.

Run tests (environment with dev dependencies):

poetry run pytest

Quick syntax check:

find eta_stress -name '*.py' -print0 | xargs -0 python3 -m py_compile

Notes

• tools/extract_stress_from_incerto.py is a migration utility and not part of normal runtime execution.
• The current implementation prioritizes strict contracts and simple stage boundaries over legacy report shape parity.
• A successful end-to-end run typically persists: baseline.json, tier1_analysis.json, scenarios/scenarios.json, aggregate/scorecard.h5, gsa/gsa.h5, and provenance.json (plus plots/* if --plot is enabled).
• HDF5 writer format tags are eta_stress.* (eta_stress.scenario_result, eta_stress.scorecard, eta_stress.gsa).
• HDF5 readers currently accept payloads from both old and new format tags as long as payload_json is valid.
• E2E artifact checks are available via eta_stress.e2e.contract.assert_e2e_artifacts.
• Current status: Tier 1–2 (including GSA) is integrated in eta-stress; remaining epic closure items are mainly cross-repo cutover and real-config E2E/harness alignment.
• Detailed Slurm/venv split patterns (optimization stage vs stress stage, array indexing, worker/thread tuning) are intentionally tracked in a separate follow-up issue and are not finalized in this README yet.

Recent Backend and Tier 2 Changes

Eta Backend Adapter

eta_stress.adapters.eta_backend is now a package instead of a single adapter file. The public import remains stable:

from eta_stress.adapters.eta_backend import BackendRequest, EtaBackend, RuntimeVersions

The public EtaBackend facade intentionally stays small:

• solve_baseline(request)
• evaluate_incumbent(request)
• solve_scenario(request)
• get_runtime_versions()

The implementation is split by responsibility:

• baseline.py handles baseline solves for deterministic, stochastic, robust, regret, and antifragile.
• runtime.py handles per-scenario runtime evaluation.
• perturbations.py applies local model, solver, scenario probability, and scenario CSV perturbations.
• antifragile.py reads antifragile Pareto H5 outputs and selects a point via weighted_sum or min_first_objective.
• incumbent.py extracts first-stage incumbent maps from solved frameworks, bundles, or antifragile Pareto H5 files.
• status.py normalizes solver status and feasibility flags.

The new adapter no longer imports eta_incerto.stress_test.*. It may still import normal eta_incerto runtime modules, evaluators, config builders, and model utilities.

Baseline Resolution

There are two baseline paths:

• baseline_resolution=solve: eta-stress solves the baseline, persists baseline.json and incumbent.npz, and keeps the live solved system object for Tier 1.
• baseline_resolution=import: eta-stress imports the baseline objective and incumbent from an existing optimization result H5, persists baseline.json and incumbent.npz, and skips Tier 1 because no live solved Linopy model is available.

Tier 1 currently requires the solved model object, specifically access to solved_system._backend.model, model constraints with lhs.solution, rhs, sign, and variables with solution, lower, upper. Existing H5 outputs are not a uniform replacement for that object across all modes.

Observed H5 shape by mode:

• stochastic, robust, regret: expose conventional RODMO-style expected groups such as __expected__/investment, __expected__/unit_dispatch, and __expected__/carriers.
• deterministic: writes scenario-level result groups and dispatch data rather than the same expected-result contract.
• antifragile: exposes Pareto data such as pareto/X, pareto/F, pareto/variable_names, and pareto/objective_names.

Because of that, import mode is currently Tier-2-oriented: it imports the incumbent design and evaluates that fixed design under stress.

Incumbent Evaluation Flow

evaluate_incumbent evaluates the existing first-stage design under one stress scenario:

1. Read incumbent.npz from the namespace baseline artifacts.
2. Load the eta-incerto experiment and the scenario/system collection for the requested variant.
3. Apply the scenario perturbations to model parameters, solver options, scenario probabilities, and/or scenario CSV values.
4. Convert the incumbent vector into first-stage investment decisions.
5. Fix those first-stage investment variables on each perturbed scenario system.
6. Run eta-incerto's DeterministicEvaluator on the perturbed scenario system with the first-stage variables already fixed. At this point the scenario is concrete and the investment design is fixed, so the remaining problem is a deterministic operational evaluation rather than a stochastic/robust/regret investment optimization.
7. Return objective_value, solver status, feasibility, fixed-design metadata, and diagnostics for scenario metrics.

The important distinction is that incumbent_eval does not choose a new investment design. It fixes the baseline/imported first-stage design first, then evaluates how that design performs under the perturbed scenario. This matches the production Tier 2 goal: post-hoc stress evaluation of an existing design.

full_reopt remains available only when explicitly requested:

• fixed_first_stage: scenario_runner calls evaluate_incumbent and persists resolve as skipped for compatibility.
• full_reopt: scenario_runner calls both evaluate_incumbent and solve_scenario.

The decision to run or skip resolve lives in scenario_runner, not in the backend facade.

Scenario Metrics

Scenario-level metrics always include incumbent metrics:

• objective.incumbent_eval.cost
• loss.incumbent_vs_baseline.abs
• loss.incumbent_vs_baseline.rel
• flag.incumbent_feasible
• decision.hamming_rate
• fragility.F_s

fragility.F_s is based on incumbent stress loss and incumbent feasibility. It does not depend on resolve metrics in the default production path.

When stress_evaluation_mode=full_reopt produces a real resolve result, scenario metrics may also include resolve extras:

• objective.resolve.cost
• loss.resolve_vs_baseline.abs
• loss.resolve_vs_baseline.rel
• loss.resolve_vs_incumbent.abs
• loss.resolve_vs_incumbent.rel
• flag.resolve_feasible

Aggregation Metrics

Aggregation always computes incumbent-based summaries:

• core objective spread uses objective.incumbent_eval.*;
• threshold and tail metrics use objective.incumbent_eval.cost;
• pair/asymmetry metrics use loss.incumbent_vs_baseline.abs;
• fragility CVaR uses loss.incumbent_vs_baseline.rel.

When resolve values exist, aggregation also emits resolve summaries:

• objective.resolve.*
• resolve-vs-baseline loss summaries;
• resolve tail/CVaR summaries;
• resolve-enabled flatness/count diagnostics.

Those resolve summaries are optional and are not required for fixed-first-stage production runs.

Plotting

Plotting now prefers incumbent metrics:

• scorecard summaries use objective.incumbent_eval.* as the primary objective family;
• per-scenario summaries use incumbent stress cost and incumbent-vs-baseline loss;
• all-method sensitivity plots use incumbent objective and feasibility signals.

When full_reopt is used and resolve metrics exist, plotting can include resolve information as an additional comparison layer. Fixed-first-stage plots do not require resolve metrics.

Provenance and Summaries

Run summaries now report incumbent objective flatness/count diagnostics independently of resolve:

• incumbent_objective_count
• incumbent_objective_unique_count
• incumbent_objective_missing_count
• incumbent_objective_flat_detected

Resolve diagnostics are only emitted when resolve is enabled:

• resolve_enabled
• resolve_skipped_count
• resolve_objective_count
• resolve_objective_unique_count
• resolve_objective_missing_count
• resolve_objective_flat_detected

In fixed_first_stage, scenario payloads keep a compatibility block:

{
  "resolve": {
    "status": "skipped",
    "skipped": true,
    "reason": "fixed_first_stage"
  }
}

This keeps older readers stable while making the production semantics explicit.

Documentation

Project overview and usage: this README on GitLab (README.md).

Related software

• eta-incerto — required dependency; provides optimization runtime and variant registry

Solver and third-party requirements

Stress tests solve MILP models through eta-incerto and eta-components (Linopy). Baseline and scenario runs inherit solver requirements from those packages:

• Gurobi (gurobipy) — required for many default eta-incerto paths; commercial license required
• CPLEX — optional; see eta-incerto scripts/README_CPLEX.md
• HiGHS — open-source alternative where supported

This package is licensed under BSD-2-Clause. Solver products are third-party software with their own licenses and terms.

Citation

For academic use, cite this repository using CITATION.cff. See AUTHORS.rst for further contributors.

License

BSD-2-Clause — see LICENSE. See also CHANGELOG.md and SECURITY.md.