failroute

Static detection of failure-routing anti-patterns in Python: the practice of converting an underlying failure into a success-like outcome at the wrong layer.

Quickstart

$ pip install failroute
$ failroute judge.py
judge.py:4: silent-suppress: contextlib.suppress(Exception) silently discards every failure inside the block; callers can never learn the operation failed
1 finding(s)

The flagged code — the exact pattern ruff's SIM105 rule recommends as a fix:

import contextlib

def score_answer(prompt: str, answer: str) -> float:
    with contextlib.suppress(Exception):   # judge outage == silence
        return judge(prompt, answer)
    return 0.0

How it differs from the linters you already run

failroute is not a replacement: it runs alongside your linter as a separate, higher-scrutiny pass over eval, judge, and agent code paths.

Why this exists

While fixing correctness bugs across mainstream AI/ML open source, one defect family kept recurring: an LLM judge outage becoming a legitimate-looking 0.0 score, a network error becoming "no results", a red-team metric reporting success from a judge that never ran. Existing linters reason about the shape of an exception handler; the defect lives in what the handler returns. failroute is an attempt to detect it at the level of the return value; its own evaluation below shows how far syntax alone gets. It is built around three principles:

• Precision over volume — every rule has positive and negative fixtures in tests/corpus/, whose expected results (manifest.json) are specified from each fixture's intended semantics; that corpus is a regression gate, and the real-code labels are a separate set (see "Test corpora" below).
• Honest triage — findings without a production consequence chain are benchmark material, not issues (see docs/process.md for a worked example).
• Everything reproducible — every number in this README can be re-run from this checkout with one command.

Failure-routing is the root cause behind some of the most insidious correctness bugs in real LLM/eval/agent codebases:

# Before — a judge API outage becomes a perfect "0.0 score" with no way to tell
try:
    score = await llm_judge(prompt)
except Exception:
    return 0.0  # ← silent fallback: failure looks like a legitimate low score

# After — the failure propagates; callers can route it to the right outcome
return await llm_judge(prompt)

What it detects

Findings are emitted as file:line: mode: message, or as JSON for CI.

Severity, and why logging no longer exempts

Every finding carries a severity (high / medium / low / info), an isomorphism verdict, a covered_by list, and a plain-language verdict string explaining the call.

Up to v0.8.0 a handler that logged the failure was exempt — it was treated as informational and not reported at all. That was wrong, and the rewrite in 0.9.0 removes it: logger.warning("judge failed"); return 0.0 still hands the caller a score it cannot distinguish from a real one. A log record makes a failure auditable after the fact; it does not make the returned value distinguishable. Logging is now a severity modifier (one level down), not an exemption. What does clear a finding is the exception reaching the caller — return Score(0.0, error=str(e)), self.errors.append(e) — because then the consumer really can tell the two apart.

Logger objects are recognised by a name heuristic (logger, LOG, audit_logger, err_log, self._log, …) rather than a fixed whitelist; exact-name matching misjudged every unconventional logger name.

The predicate: value-domain membership, then isomorphism

0.9.0 replaces "the handler returned a constant from a fixed set" with two questions asked in order:

1. Is the value inside the function's success domain? -> Optional[Doc] returning None is a declared outcome — the caller can tell. -> float returning 0.0 is not. Return annotations, other return statements in the same function, and whether the handler substitutes a value the try body computed all feed this.
2. Is the caught exception the answer to the question the function asks? _is_serializable() wrapping one json.dumps — the exception is the answer, so returning False is a contract. is_vulnerable() wrapping a loop over results — an arbitrary exception is not "not vulnerable", so returning False claims a scan completed that never did.

Handler bodies are walked path-sensitively, so except: ...; if cond: return 0.0 else: return 1.0 is reachable — v0.8.0 only inspected top-level statements and missed it.

Usage

$ failroute path/to/file.py
$ failroute path/to/dir      # recursive
$ failroute --repo .         # skip .git/.venv/build/...
$ failroute --repo . --exclude tests/corpus   # repeatable path exclusions
$ failroute --json --repo .  | jq 'select(.mode=="silent-fallback")'
$ failroute --format sarif --output results.sarif --repo .   # code scanning
$ failroute --threshold 5    # exit 1 when more than 5 findings
$ python -m failroute .      # module form (no console script needed)

Exit codes: 0 clean, 1 findings above threshold, 2 usage error.

Project configuration

Repositories can commit their policy instead of repeating CLI flags; the nearest pyproject.toml at or above the scan path is consulted:

[tool.failroute]
exclude = ["tests/corpus", "vendor"]
threshold = 0
ignore = ["name-shadowing"]            # disable rules by id
fallback_values = [-1, "N/A"]           # project-specific fallback sentinels

[tool.failroute.rules.implicit-fallback]
enabled = false                          # same as adding to `ignore`
severity = "warning"                     # override the SARIF level

Sentinel canonicalisation: a TOML int/float is its decimal token (-1), a TOML string its Python repr form ("N/A" -> "'N/A'"). The tool matches what you write against the same canonical rendering the detector produces — and without configuration it refuses to guess project-specific sentinels.

CLI flags always override config (--ignore RULE, --exclude PATH). Malformed config is ignored, never fatal. Large trees: --jobs 0 fans the scan across cores and --cache keeps an mtime-keyed result cache in the system temp dir; both produce findings identical to the serial scan.

pre-commit

repos:
  - repo: https://github.com/feiiiiii5/failroute
    rev: v0.9.2
    hooks:
      - id: failroute

Output formats

$ failroute --format text   path/       # default: file:line: mode: message
$ failroute --format json   path/       # one JSON object per finding
$ failroute --format sarif  --output scan.sarif path/   # SARIF 2.1.0

SARIF output plugs straight into GitHub code scanning via the upload-sarif action, so findings appear inline on pull requests:

- run: failroute --format sarif --output results.sarif --repo .
- uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: results.sarif

Or use the bundled composite action, which installs failroute, scans, and uploads SARIF in one step:

- uses: feiiiiii5/failroute/action@main
  with:
    path: src
    exclude: tests/corpus fixtures
    threshold: "0"

Severity mapping: silent-fallback → error, no-action and masked-exception → warning.

Suppressing findings

Reviewed-and-accepted handlers can be opted out with a line marker (the scanner honors both):

try:
    return best_effort()
except Exception:  # failroute: ignore - documented fallback semantics
    return None

# pragma: no cover markers are honored as well (explicitly defensive code).

Examples that trip it

def classify(text):                       # no-action
    try:
        return model.predict(text)
    except Exception:
        pass                                # 💥 swallowed

def score(prompt):                          # silent-fallback
    try:
        return judge(prompt)
    except Exception:
        return 0.0                          # 💥 outage == "0.0 score"

def fetch(url):                             # silent-fallback (assign)
    data = None
    try:
        data = download(url)
    except Exception:
        data = {}                           # 💥 error looks like an empty result
    return data

def evaluate(prompt):                       # silent-suppress
    with contextlib.suppress(Exception):    # 💥 outage == silence, no trace at all
        score = judge(prompt)
    return score

What it does not flag (by design)

• except KeyboardInterrupt / except SystemExit — normally intentional.
• Non-empty "error-shaped" containers (return {"items": [], "error": True}) — an explicit error object is the remediation the tool itself recommends, so it refuses to second-guess one. Loop skip-and-continue / retry-break handlers (continue / break) are deliberate control flow, not fall-throughs.
• The same control-flow exception types under contextlib.suppress (KeyboardInterrupt, SystemExit, StopIteration, CancelledError, GeneratorExit) — absorbing cancellation or iterator termination is idiomatic, not failure routing. One real error type in the same call (e.g. suppress(CancelledError, OSError)) still flags.
• Handlers that re-raise unconditionally without a fallback.
• except bodies that log at warning/error and re-raise — the failure still propagates; we only flag the success-looking path.

Run failroute on its own checkout as a smoke test:

$ pip install -e .
$ failroute --repo .     # expected: zero findings (self-hosting)

Benchmarks & validation

All numbers below are reproducible from this checkout; nothing here is copy-pasted from a run that cannot be re-executed.

Test corpora

Two, with different jobs:

• tests/corpus/ — 68 explicit fixtures (34 positive / 34 negative, corpus v7), ground truth in manifest.json, written from each fixture's semantics rather than from tool output. It is a regression gate, not evidence of real-world precision: it was developed within the same project as the detector, sharing its blind spots, and it never caught the top-level-only bug that a labelling pass over real code found immediately.
• bench/realworld/ — 87 cases anchored to real coordinates in the pinned corpus, including the 80 findings behind the paper's labelling sample and seven discriminating probes. Those labels were written by two rounds of LLM agents, not by humans (the paper's Limitation 1); no human labelling pass has been run. This adds real-code regression coverage; it does not establish independently validated defect labels or accuracy on unseen packages.

The full suite is 221 tests across a 3 OS × Python 3.9–3.13 matrix, plus mypy --strict.

What syntactic linters miss

Measured against eight pinned PyPI releases (garak, inspect_ai, pydantic-ai, uqlm, trl, smolagents, deepteam, fickling). The pinned corpus is 2,354 .py files and 563,270 lines, locked by URL, SHA-256 and tree hash in paper/corpus-lock.json so it is byte-reproducible. The scan root the detector actually opens is 2,124 .py files and 524,229 lines; the 230-file, 39,041-line difference is tests, examples and vendored code that failroute never reads. Every finding count below is over that scan root. failroute reports 476 findings (v0.9.1). The v0.8.0 detector reported 621; the drop is the predicate rewrite described above, not a change of corpus. The paper freezes the v0.8.0 frame at 621 deliberately, so its numbers and this README's will differ. The v0.7.0 detector reported 28 more; all 28 were precision fixes (this CHANGELOG's 0.8.0 entry), each one individually attributed in docs/f-batch-report.md.

Compared against four standard linters at their default configurations (ruff, bandit, pylint, flake8 + bugbear), matching on ±1 line:

Per rule, and which tool (if any) reaches it:

Of the 208 high findings, 25 are reported by no shipped linter.

covered_by was wrong until 0.9.0. It inferred lint coverage from the handler's body shape and never looked at the caught type, so every narrow except X: pass was credited to S110/B110 — rules that only fire on broad handlers. 146 findings were over-attributed; correcting it moved the novel share from 18.3% to 47.1%. The mapping is now verified by running all four linters over a 68-cell probe matrix (tools/lint_mapping_probe.py) rather than asserted statically.

A note on baselines. Earlier versions of this README compared only against ruff's S110/S112. That is not a fair baseline: pylint is much stronger (much stronger than ruff alone), and a ruff-only comparison overstates the gap relative to the four-tool union. The table above uses that union rather than a ruff-only comparison. A project-authored semgrep ruleset targeting these patterns covers most of the silent-suppress findings — so "no shipped linter reaches this" is a statement about default configurations, not about what is expressible.

And a blunter one. All 12 sites labelled DEFECT in the original sample sit on bare except:. flake8 --select E722 flags 134 sites in this corpus and contains all 12. On this corpus E722 is a 4.6× tighter search-space reducer than failroute's v0.8.0 frame at equal recall on those labels. The paper is about why.

Re-run: see paper/ARTIFACT.md. Results are checked into bench/.

How often is a finding a bug?

Mostly, it is not — and that is the most useful thing this project measured.

On a stratified random sample of 80 findings drawn from the v0.7.0 finding set (by rule × package, fixed seed), labelled in two LLM-agent rounds. Their agreement establishes neither independence nor label accuracy. The v0.8 precision fixes changed the sampling frame, so these labels describe the v0.7.0 finding set only; see paper/ARTIFACT.md before citing them.

All 12 DEFECT labels fall inside a single, young package; the other seven packages have 0 DEFECT labels out of 66 sampled findings. Five of those 12 concern missing assignments whose failure-conversion mechanism is unestablished. These are label counts, not confirmed failure-routing defect rates; the paper reports the corresponding family sensitivity analysis.

The conclusion is not flattering to this tool, and it is the point: a syntactic detector cannot distinguish an intentional fallback from a bug. failroute narrows where to look; a person still has to argue the failure chain. Treat its output as a review queue, not a defect list.

The internal tests/corpus/ fixtures (68 samples, precision = recall = 1.0 in CI) are a regression gate on the rules themselves — construct validity. They say nothing about how the tool behaves on code it has never seen; the table above does.

The current manuscript reports the recall audit and its scope limits; LaTeX source and reproduction instructions accompany it. The manuscript has not yet been posted to arXiv.

Throughput

Measured 2026-08-29 on the vLLM core tree (vllm/vllm, 2,032 files, ~758 kLOC) with the six-rule v0.6 engine: ~74 kLOC/s serial (10.3 s, 388 findings), ~377 kLOC/s with --jobs 0 (2.0 s, 10 workers, identical findings), and ~0.1 s warm with --cache. The v0.6 serial number is lower than v0.5.1's single-rule figure because findings now come from six independent rules re-walking each handler; the parallel path is the intended operating point for large trees. Re-run: time failroute <path> --repo --jobs 0 --quiet.

Development

$ pip install -e ".[test]"
$ pytest
$ ruff check .

How this project is built

The project idea originated with Yufeiyang Chen, who directs the work and makes the key decisions, including those made across successive manuscript revisions. Claude and other AI tools assist with implementation, tests, refactoring, analysis, and drafting. This is an author-led, AI-assisted project; the author is responsible for its claims and final manuscript. Pushes to main and pull requests run the deterministic gates in .github/workflows/ci.yml: the pytest suite on Linux, macOS and Windows with Python 3.9–3.13, ruff, mypy in strict mode (pyproject.toml), a coverage floor, a self-scan of the repository with the tool itself, and tools/benchmark.py, which fails unless precision and recall on the tests/corpus/ fixtures stay at 1.0. These gates check behaviour against written expectations; they do not establish that a finding is a defect. The fixture expectations are a regression gate, and the real-code labels in bench/realworld/ come from two rounds of LLM agents with no human labelling pass (see "Test corpora" above). The triage process, with a worked example that includes a case where nothing was filed, is in docs/process.md.

License

MIT — see LICENSE.