FORECAST

FORECAST is a debugging approach that generates diagnoses to boost repair agents.

Given a program with failing and passing tests, FORECAST traces their executions, finds the execution features that separate the failing runs from the passing ones, and links them into a cause chain: a short, ordered explanation of how the failure comes about. Every step carries a discriminability, an MCC-based score of how sharply it separates failing from passing runs. A repair agent gets this diagnosis next to the issue it has to fix.

1. Variable "y" defined at middle.py:1 is used at middle.py:6 AND In middle.py:1, y < x (discriminability: 1.00)

FORECAST works in five steps:

Contents:

• Installation
• Quick start
• SWE-bench instances
• Repair agents
• Models
• Trace collection
• Test selection
• Plugin system
• Evaluation: reproducing the paper's results
• License

Installation

FORECAST needs Python 3.12 or newer, and Docker for SWE-bench instances and repair agents.

pip install forecast-repair                       # diagnoses only
pip install "forecast-repair[agents,evaluation]"  # plus repair agents and the SWE-bench harness

The optional extra zstd adds zstd as a codec for traces (gzip is the default).

SWE-agent and Agentless run from checkouts at pinned commits, which the repository's agents/install_upstream.py installs. To use them, to run the quick start below, or to reproduce the evaluation, install from a clone instead:

git clone https://github.com/smythi93/forecast.git
cd forecast
pip install -e ".[agents,evaluation]"
python agents/install_upstream.py      # SWE-agent and Agentless at their pinned commits

Quick start

resources/middle holds a small program, middle(x, y, z), which should return the middle of its three arguments, and six tests, one of which fails. The commands below run from the repository root.

T=tests.py::TestMiddle
OUT=/tmp/middle

# 1. Instrument the program (into $OUT/instrumented) and trace the failing and passing tests
forecast collect -s resources/middle -o $OUT \
    -f $T::test_middle_213 \
    -p $T::test_middle_335,$T::test_middle_123,$T::test_middle_321,$T::test_middle_555,$T::test_middle_534

# 2. Build the traced relevance tree ($OUT/tree.pkl)
forecast tree -s resources/middle -d $OUT -o $OUT

# 3. Derive the diagnosis ($OUT/diagnosis.json, .txt and .md)
forecast diagnose -s resources/middle -d $OUT -o $OUT
cat $OUT/diagnosis.txt

# 4. Build an agent's prompt with the diagnosis ($OUT/prompt.txt)
forecast prompt -s resources/middle --agent mini-swe-agent --mode forecast \
    --diagnosis $OUT/diagnosis.json --problem-statement "middle(2, 1, 3) returns 1, not 2." \
    -f $T::test_middle_213 -o $OUT -v

The prompt is the issue followed by the diagnosis:

middle(2, 1, 3) returns 1, not 2.

## Failure Diagnosis

The following diagnosis shows the cause-effect chain leading to the test failure:

1. Variable "y" defined at middle.py:1 is used at middle.py:6 AND In middle.py:1, y < x (discriminability: 1.00)

--mode baseline gives the issue alone. run then runs the agent with a model (--model, see Models) and saves its trajectory and patch, and evaluate --patch <file> checks a patch against the tests. Every command has --help.

SWE-bench instances

For SWE-bench Lite, the repository's evaluation package runs the steps per instance. It collects the traces in the instance's SWE-bench Docker image, with the developer's test patch applied, from the instance's FAIL_TO_PASS and PASS_TO_PASS tests:

# Diagnosis: collect, tree, diagnose, and SBFL (which ACR and RepairAgent use)
python -m evaluation.diagnosis_batch --subset lite --stage all \
    --instance-ids astropy__astropy-12907 --output-dir results

# Repair in both modes; each is evaluated by the SWE-bench harness
python -m evaluation.repair_batch --agent mini-swe-agent --mode baseline --stage all \
    --model claude-haiku-4-5-20251001 --instance-ids astropy__astropy-12907 --output-dir results
python -m evaluation.repair_batch --agent mini-swe-agent --mode forecast --stage all \
    --model claude-haiku-4-5-20251001 --instance-ids astropy__astropy-12907 --output-dir results

Results go to results/<instance>/:

• diagnosis/: the diagnosis and SBFL;
• prompt/, run/ and evaluate/<agent>/<mode>/: the repair, its trajectory and the harness verdict;
• report.json: every step, with its time.

evaluation.parallel_repair runs one agent and mode over many instances in parallel. The Evaluation uses it.

Repair agents

FORECAST evaluates a diagnosis by what it does for repair agents. Every agent:

• runs inside the instance's SWE-bench image;
• sends every LLM call through FORECAST's model interface (ModelClient.chat, recorded per run in calls.jsonl);
• receives the diagnosis through one injection point, the issue text (RepairAgent.issue_text).

Patches are judged by the official SWE-bench harness.

AutoCodeRover. AutoCodeRover's implementation is published under the SONAR Source-Available License v1.0. That is not an open-source license, and it does not permit our use. FORECAST therefore contains none of its code. The acr agent is a complete re-implementation of the approach as the ISSTA'24 paper (Zhang et al., "AutoCodeRover: Autonomous Program Improvement") describes it, with the paper's parameters:

• stratified context retrieval with the paper's seven search APIs;
• retrieval augmented by spectrum-based fault localization;
• validated patch generation.

The original code was consulted only to verify the re-implementation; nothing was copied from it. Results for acr are results of this re-implementation, not of the original tool.

Models

--model picks the provider by the model's name:

FORECAST computes each call's cost from its tokens at the provider's list prices (forecast.agents.models.utils.PRICES). Any price a provider or proxy reports is ignored. FORECAST_PRICE_AS=<model> prices a model without a price, such as a local one, as another model.

Trace collection

Tracing records every observation a run makes, so a traced run can produce orders of magnitude more data than a plain one and outlive its timeout.

Warm images

The single biggest cost is usually not the test. Each test runs in its own fresh container, and the first test to run in a container pays to build whatever caches the project uses — matplotlib, for example, scans and parses every system font. That work is small on its own, but it is paid once per test, and under instrumentation it is traced.

Measured on matplotlib__matplotlib-23314, one test in a fresh container:

setup() therefore runs one test before committing the image, so the caches are built once, land in the image, and are never traced. Disable with warmup=False on the subject if a subject's tests must not run during setup.

Budgets

The tracer runs inside the subject's environment, so the limits below only apply if that environment has sflkitlib >= 0.2.3. When sflkitlib is installed from a working copy on the host, FORECAST copies that working copy into the container automatically; set FORECAST_SFLKITLIB to a checkout, wheel, or sdist to override.

Compression is the effective lever here, shrinking traces about 4x. The value and loop caps matter only for subjects that genuinely move large buffers or run very hot loops; on matplotlib they change trace size by about 4%.

Limits are tuned with host environment variables:

Test selection

A diagnosis compares passing against failing runs only at functions both reach, so a passing test sharing no executed code with the failure cannot change the result — it only costs a traced run. ExperimentRunner therefore picks which passing tests to trace from ordinary coverage, which is far cheaper to collect than a trace:

• coverage-overlap (default) — keep tests that execute code the failing tests execute, most overlapping first; ties favour the test that executes least besides it.
• reference-patch — additionally require the test to reach code the developer patch changes. Oracle-based: it consults the fix, so results must be reported separately. The selection is recorded in <instance>/diagnosis/test_selection.json with uses_oracle: true.
• name-heuristic — the earlier per-repository name filter; needs no coverage run.
• none — trace every listed test.

Plugin system

FORECAST loads plugins via Python entry points in the forecast.plugins group.

Plugins can register:

• repair agents
• model providers
• localization formulas
• localization techniques
• subject classes

Example plugin entry point:

[project.entry-points."forecast.plugins"]
MyForecastPlugin = "my_pkg.forecast_plugin"

Example plugin module:

from forecast.plugin_loader import PluginContext


def register(context: PluginContext):
    @context.register_formula("MyFormula")
    def my_formula(ep, np, ef, nf):
        return (ef + 1) / (ep + np + ef + nf + 1)

    @context.register_localization_technique("my-localizer")
    def my_localizer(*, subject, location_type, formula="Tarantula", **kwargs):
        # Can call any subject API; technique is fully plugin-defined.
        return subject.sfl(location_type=location_type, formula=formula, technique="sfl")

Evaluation

The paper evaluates FORECAST on all 300 instances of SWE-bench Lite (test split). Five repair agents ran with two models, each agent in two modes:

• baseline: the agent gets the issue;
• forecast: the agent gets the issue plus FORECAST's diagnosis.

The diagnoses were computed once per instance, before any repair, and both models and all agents got the same one. Each instance had a limit of 6 hours per repair.

Everything needed to reproduce the results is in this repository:

The runs themselves, about 54 GiB, are on Zenodo as one file, forecast-results.tar.zst (DOI: to be added).

Analyzing the results

analyze.py reads only evaluation/results/results.json and prints Markdown tables, one section per research question:

python3 evaluation/scripts/analyze.py              # all sections
python3 evaluation/scripts/analyze.py rq1 rq3      # only these

Every number is computed from the per-instance entries in results.json:

• rates and sums count instances;
• distributions run over instances, given as median (first to third quartile) and mean;
• the McNemar tests pair each instance's baseline and FORECAST verdicts;
• each FORECAST run is charged the diagnosis time of its own instance.

A run without a harness verdict counts as unresolved.

Reproducing the runs

Run the scripts from the repository root, after the installation. <runs> is the folder that will hold the results, runs/<model>/<instance>/.

1. Diagnoses. This collects traces for every instance, keeping them in <work>, which needs much disk. The paper's diagnoses took about 330 hours in total (the rq4 section lists the time per step). Building a tree takes much memory for the large (sympy) instances, so keep the number of parallel instances low:

   bash evaluation/scripts/diagnose.sh <work> <runs> 4
   

2. Models.

   • For Haiku, run a LiteLLM proxy that serves anthropic/claude-haiku-4-5-20251001, and set LITELLM_PROXY_API_BASE and LITELLM_PROXY_API_KEY.
   • For Qwen, start one Ollama server per GPU serving qwen3.5:27b, and set OLLAMA_ENDPOINTS to their comma-separated URLs.

3. Agentless mirrors. Optional, but needed with many workers:

   bash evaluation/scripts/mirrors.sh <mirrors>
   export MIRRORS=<mirrors>
   

4. Repairs. These are all 20 combinations of model, agent and mode, or one at a time; finished instances are skipped:

   bash evaluation/scripts/run_all.sh <runs>                                  # everything
   bash evaluation/scripts/run_agent.sh <runs> haiku swe-agent forecast 10    # one model, agent and mode
   

   Each instance runs in its own process, so the number of workers does not affect the results.

5. Results and archive. This computes results.json from the per-instance files and packs everything into one archive:

   python3 evaluation/scripts/pack.py <runs> <out> --add README.md
   python3 evaluation/scripts/analyze.py --results <out>/results.json
   

Results

results.json has one entry per instance and repair run. The comments below only explain the fields; the file itself is plain JSON.

{
  "diagnoses": {
    "<instance>": {
      "discriminability": 1.0,          // of the chain's sharpest step (MCC-based)
      "mean_discriminability": 1.0,     // over the chain
      "causes": 10,                     // length of the cause chain (0: empty diagnosis)
      "cause_types": {"atom": 8, "conjunction": 1, "disjunction": 1},  // and "divergence"
      "names_fixed_file": true,         // a cause lies in a file the developer patch changes
      "names_fixed_line": false,        // a cause names a line within one of its hunks
      "failing_tests": 3,               // traced tests
      "passing_tests": 19,
      "listed_passing_tests": 19,       // PASS_TO_PASS tests SWE-bench lists
      "selection": "all",               // how the passing tests were selected
      "selection_reason": "below selection threshold",  // why, where recorded; else null
      "selection_uses_patch": false,    // the selection consulted the developer patch
      "tree_workers": 1,                // processes that built the tree
      "instrument_seconds": 168.1,      // part of collect_seconds
      "collect_seconds": 315.3,
      "tree_seconds": 841.4,
      "diagnose_seconds": 47.3
    }
  },
  "repairs": {
    "<model>": {
      "<agent>": {
        "<mode>": {
          "<instance>": {
            "resolved": true,           // harness verdict; null: no verdict (counts as unresolved)
            "error": false,             // the agent ended with an error
            "patch": true,              // a non-empty patch was submitted
            "applied": true,            // the harness applied it cleanly; null: no verdict
            "lines_added": 2,
            "lines_removed": 2,
            "files": 1,
            "new_files": 0,             // files the patch adds (the agent's own scripts, notes)
            "existing_lines": 4,        // added + removed lines in files that existed before
            "existing_files": 1,
            "exit_status": "submitted", // how the run ended (SWE-agent, mini-SWE-agent); else null
            "calls": 4,                 // LLM calls, and their tokens and cost (USD)
            "prompt_tokens": 8633,
            "completion_tokens": 1924,
            "cost": 0.018253,           // null for Qwen, which runs locally (see Costs)
            "max_tokens": {"1024->1024": [4, 0]},  // per answer length requested->given:
                                        // answers, answers cut off at that length
            "seconds": 18.08            // the agent's runtime; null if not recorded
          }
        }
      }
    }
  }
}

Costs

All costs use Anthropic's list prices for Claude Haiku 4.5. Prices reported by a provider or proxy are never used, since they may include institutional discounts.

The overview section also shows the cost without prompt caching: every prompt token priced as input.

Qwen runs locally and costs nothing per token, so results.json has no cost for its runs and analyze.py compares it by tokens. Its calls are nevertheless counted at Haiku's prices (FORECAST_PRICE_AS), without caching, because SWE-agent's and mini-SWE-agent's budgets ($3 per run) are given in dollars; for Qwen this amounts to about three million tokens per run.

The archive

forecast-results.tar.zst holds this README, results.json, the diagnoses and every run. It is about 320 MB; extracted it takes about 54 GiB.

sha256sum -c SHA256SUMS                        # SHA256SUMS from evaluation/results/
tar --zstd -xf forecast-results.tar.zst        # or: zstd -dc forecast-results.tar.zst | tar -xf -

tar --zstd needs GNU tar and zstd. The archive was compressed with --long=27, which zstd decompresses without extra flags.

forecast-results/
  README.md                        this file
  results.json                     the per-instance results
  diagnoses/
      instances.txt                the 300 instances
      <instance>/
          diagnosis.json           the diagnosis: cause chain with discriminability
          diagnosis.txt            the text the forecast mode appends to the issue
          diagnosis.md             the same text as Markdown
          sbfl.json                spectrum-based fault localization (used by ACR and RepairAgent)
          pipeline.json            FORECAST's report of the steps that built the diagnosis:
                                   collect (instrumentation, traced tests), tree, diagnose, with times
  <model>/<mode>/
      <agent>.txt                  the instances of this run
      <instance>/run/<agent>/<mode>/
          result.json              the agent's result: patch, elapsed_seconds, error, ...
          patch.diff               the submitted patch
          logs/run_<instance>.log  the agent's log
          trajs/<instance>/
              calls.jsonl          every LLM call: messages, response, tokens, cost
              outcome.json         per-run totals (calls, tokens, cost)
              run_info.json        model, prices, diagnosis and fault localization given
              <trajectory>         the upstream agent's own trajectory, see below
      <instance>/evaluate/<agent>/<mode>/result.json
                                   SWE-bench harness verdict: resolved, applied, per-test status

The files are archived byte for byte as the evaluation wrote them, and the archive stores no owner and no time stamps. pack.py checks every archived file:

• neither the API key nor the proxy's address appears (--forbid-env);
• no file holds a date written at run time. This uses the rules forecast.run_dates applied after each run: dates from 2025 on, epoch time stamps under time-stamp keys, and setuptools-scm date suffixes. Patches are exempt, since they are kept as the agents submitted them; dates in the repositories' own code are older and do not count.

The archive leaves out:

• the SWE-bench harness's test logs (the verdict file keeps the per-test outcome);
• FORECAST's execution traces and relevance trees;
• the schedulers' logs.

results.json also compares the diagnoses with the developer patches, which pack.py reads from SWE-bench Lite (--dataset, through the datasets package); FORECAST never used them. To rebuild results.json from the extracted archive and SWE-bench Lite:

python3 evaluation/scripts/analyze.py --results rebuilt.json --from forecast-results

The rebuilt file is byte-identical to results.json.

License

FORECAST is licensed under the Apache License 2.0; see LICENSE.

The repair agents FORECAST integrates are licensed as follows: