silent-killers 0.1.0

Audit pipeline that detects unsafe exception handling in LLM‑generated Python code

Silent Killers

An Exploratory Audit of Exception‑Handling in LLM‑Generated Python

tl;dr We show that large‑language models often add try/except blocks that silently swallow errors. Our AST‑based metric pipeline lets anyone quantify that risk across thousands of generated scripts in seconds.

---

1  Scope of this study

Modern LLMs can write Python that “runs”, but how it fails matters. A bare except: or a blanket except Exception: with no re‑raise can mask fatal bugs, leading to silent data corruption or debugging nightmares—these are the silent killers.

We collected 5 seeds × 8 models × 3 prompts (easy → hard rewrite tasks) and asked:

• How often do models inject try/except at all?
• Of those, how many are “bad” under a strict re‑raise rule?
• Does difficulty exacerbate the problem?

The full paper is in docs/ (LaTeX source) and the main plots live in data/figures/.

---

2  Repository layout

repo-root/
├─ src/
│   └─ llm_exception_audit/        ← **reusable package**
│        ├─ __init__.py
│        ├─ metrics.py             (AST visitors & regex metrics)
│        └─ cli/
│             ├─ process_files.py
│             └─ post_processing.py
│
├─ data/                           ← study‑specific artefacts
│   ├─ propagation_prompt/
│   ├─ calibration_prompt/
│   ├─ optimization_prompt{,2}/
│   └─ figures/
├─ tests/
│   └─ test_exception_labels.py
├─ pyproject.toml
└─ README.md

Everything under src/llm_exception_audit/ is published to PyPI; data/ stays in the repo (or Git LFS) but is not shipped inside the wheel.

---

3  Installation

git clone https://github.com/your‑org/llm-exception-audit.git
cd llm-exception-audit
python -m pip install --upgrade pip
pip install -e .[dev]          # runtime + pytest + ruff

Requires Python ≥ 3.9
Runtime deps: pandas, numpy, matplotlib

---

4  Quick start

4.1  Generate metrics CSVs

process_files --base-dir data/propagation_prompt
process_files --base-dir data/calibration_prompt
process_files --base-dir data/optimization_prompt
process_files --base-dir data/optimization_prompt2

Each run creates

data/<prompt_dir>/
    llm_code_metrics.csv
    llm_response_metrics.csv

4.2  Plots & summary tables

post_processing --root data

Creates:

plots_grid_refactored/
    grid_status_3color.png
    grid_loc_continuous.png
    grid_bad_exception_rate.png
    grid_bad_exception_count.png
    bar_parsed_ok_by_difficulty.png
    summary_by_model.csv
    summary_by_difficulty.csv

Example output

code‑status	bad‑rate heatmap

4.3  Library usage

from llm_exception_audit import code_metrics

python_code = "try:\n    1/0\nexcept Exception:\n    pass"
for metric in code_metrics(python_code):
    print(metric.name, metric.value)

---

5  Metrics at a glance

metric	description
exception_handling_blocks	count of except clauses
bad_exception_blocks	bare except: or except Exception: without raise
bad_exception_rate	bad / total, 2 dp
uses_traceback	calls traceback.print_exc() / .format_exc()
…	see src/llm_exception_audit/metrics.py

---

6  Key pilot finding

When a model adds any error handling, 50–100 % of those handlers are unsafe.
Inclusive bad‑rates look tame (0 – 0.6) but conditional bad‑rates (only_with_try) spike to 1.0 for several models on simple prompts.

---

7  Development

ruff check .          # lint
pytest                # run unit tests
coverage run -m pytest && coverage html

CI runs on GitHub Actions across Python 3.9‑3.11 (see .github/workflows/ci.yml).

---

8  Roadmap

• 🚧 dynamic execution traces (runtime errors, coverage)
• 🚧 extend to other unsafe patterns (weak crypto, insecure I/O)
• 🚧 publish TestPyPI wheel

PRs & issues welcome!

---

9  License & citation

MIT License.
If you use the metrics or figures, please cite:

@misc{Quick2025SilentKillers,
  title  = {Silent Killers: An Exploratory Audit of Exception‑Handling in LLM‑Generated Python},
  author = {Julian Quick},
  year   = {2025},
  url    = {https://github.com/your‑org/llm-exception-audit}
}

Happy auditing – don’t let silent errors slip through!