pyfix-agent 1.0.0

An autonomous, multi-turn AI debugging agent built from scratch using AST surgery.

🚀 PyFix Agent: Autonomous ReAct Debugging Loop

An autonomous, multi-turn AI debugging agent built entirely from scratch in Python.

Unlike standard wrappers that simply ask an LLM to "fix this code," PyFix Agent implements a custom ReAct (Reasoning and Acting) state machine and utilizes Abstract Syntax Tree (AST) manipulation to surgically patch Python files in real-time. It evaluates its own fixes by executing the code inside a sandboxed subprocess, iterating dynamically until the script passes or it reaches the maximum iteration limit.

---

🧠 Core Architecture

This project deliberately avoids high-level agentic abstractions (such as LangChain or LlamaIndex) to build the core agentic loop from first principles.

graph TD
    A[Start] --> B[Execute target script via Subprocess]
    B --> C{Execution successful?}
    C -- Yes --> D[Stop: Bug Fixed 🎉]
    C -- No --> E[Extract Stack Trace & Error Message]
    E --> F[Parse Stack Trace for last function name]
    F --> G[Construct LLM Prompt with Context Memory]
    G --> H[Query LLM for patch]
    H --> I[Clean LLM markdown and parse AST]
    I --> J{Function-level error?}
    J -- Yes --> K[Use AST surgery to replace target function node]
    J -- No --> L[Fallback: Replace entire file]
    K --> M[Write patched script to disk]
    L --> M
    M --> N{Max iterations reached?}
    N -- Yes --> O[Stop: Max iterations reached ❌]
    N -- No --> B

Key Architectural Pillars

1. Execution Engine: Runs the target script via Python subprocesses, capturing standard outputs, standard errors, and stack traces with safety timeout thresholds.
2. Context Memory: Maintains a chronological conversation history array, allowing the LLM to learn from its previously failed patching attempts without losing the original code context.
3. AST Surgery: Parses the LLM's response and uses Python's native ast.NodeTransformer to swap out broken function nodes with the corrected logic, leaving the rest of the file entirely untouched.

---

⚖️ Design Choices & Trade-offs

Building an autonomous agent requires balancing safety, context window limits, and real-world unpredictability.

1. AST Function Surgery vs. Full File Overwrites

• The Problem: Asking an LLM to rewrite an entire 1,000-line script to fix a single typo is slow, expensive, and risks the model "truncating" or getting lazy with existing, working code.
• The Solution: The agent extracts the specific function_name from the traceback. It prompts the LLM only for the corrected function. The PythonSurgery class (inheriting from ast.NodeTransformer) then traverses the syntax tree, finds the broken ast.FunctionDef, and seamlessly swaps it with the new node.
• The Trade-off: While this guarantees perfect preservation of unrelated code, it requires specialized routing logic for errors that occur at the top-level <module> scope, which bypass the AST function surgery and require full-file patching.

2. Execution-Based Evaluation vs. Exact String Matching

• The Problem: How do we benchmark if the agent successfully fixed a bug? Traditional exact string matching fails because the LLM might use different variable names (e.g., x += 1 instead of x = x + 1), resulting in false negatives.
• The Solution: The evaluation suite uses Execution-Based Benchmarking. The benchmark dynamically runs automated unit tests or validation scripts containing assertion statements against the patched files. If the patched script exits with code 0, it is marked as a success.

---

📊 Evaluation Benchmark

The agent is evaluated against a curated dataset of scripts spanning 5 distinct error categories:

• NameError: Undefined variables, scope issues, and missing imports.
• IndexError: Off-by-one loop conditions and bounds checking.
• TypeError: Data type mismatches and unsupported operations.
• AttributeError: Typographical errors in object methods or calling methods on NoneType.
• Logic Bugs: Silent errors that require execution-based assertions to detect.

(Currently evaluated against an automated function-level testing benchmark suite inside eval_dataset/)

---

🛠️ Installation & Usage

Option 1: Standard Pip Installation (Recommended)

To install PyFix Agent locally in editable mode (which registers the CLI tool globally):

# Clone the repository
git clone https://github.com/yourusername/agent-debugging-loop.git
cd agent-debugging-loop

# Install package in editable mode
pip install -e .

This registers the global CLI tool pyfix-agent which can be executed from anywhere.

Option 2: Run as a Python Script

If you prefer to run it without installing the package:

pip install huggingface_hub
python pyfix_agent.py --script <path_to_script>

Configuration

Export your Hugging Face Hub token to your environment variables to ensure secure API access:

Bash (Linux/macOS):

export HF_TOKEN="your_huggingface_token_here"

PowerShell (Windows):

$env:HF_TOKEN="your_huggingface_token_here"

---

🚀 CLI Usage Guide

Point the agent at any broken Python script. Use the --verbose flag to watch the ReAct state machine's internal thought process.

# Run with default Qwen model
pyfix-agent --script my_broken_code.py --verbose --max_iter 5

# Run using a specific Hugging Face model
pyfix-agent --script my_broken_code.py --model "mistralai/Mixtral-8x7B-Instruct-v0.1" --max_iter 3

CLI Command Options

Argument	Type	Default	Description
--script	str	Required	Path to the broken Python script to debug
--max_iter	int	5	Maximum number of debugging iterations
--verbose	flag	False	Enable logging of reasoning, tracebacks, and raw LLM responses
--model	str	Qwen/Qwen2.5-72B-Instruct:cheapest	Model endpoint ID on Hugging Face Serverless API

Running the Evaluation Benchmark

To run the full evaluation suite against the benchmark:

python benchmark.py

---

📦 Packaging & Release Recommendations

For releasing version 1.0.0 of PyFix Agent as a CLI tool:

1. Direct Python Package (Recommended for Python users)

Distribute PyFix Agent as a Python package via PyPI.

• Build tool: Use build (pip install build) to compile source distribution .tar.gz and wheel .whl files.
• Upload tool: Use twine to publish the artifacts to PyPI.
• Install: Users can install it directly with pip install pyfix-agent or run isolated using pipx run pyfix-agent.

2. Standalone Binary Executable (Recommended for Non-Python users)

Compile the script into a standalone executable using PyInstaller.

• Compile:

  pip install pyinstaller
  pyinstaller --onefile --name pyfix-agent pyfix_agent.py
  

• Release Artifact: Upload the compiled executable (dist/pyfix-agent or dist/pyfix-agent.exe) directly as a release asset in your GitHub Releases.
• Note: The target environment still needs a Python interpreter installed to execute target scripts via sys.executable.