Task: "Add WebSocket support to the chat API"
(Given a real codebase with FastAPI routes, Pydantic schemas, and an existing REST chat endpoint.)

"Add a WebSocket endpoint.
 Use the websockets library.
 Create handlers.
 Add auth middleware."

Phase 1: Extend api/routes/chat.py
  - Reuse existing ChatService
  - Validate token with current auth layer
  - Test: pytest tests/api/test_chat_ws.py
Phase 2: Adapt schemas/chat.py

Preserve current ChatMessage shape
Verify: pydantic model_validate()

Where to start 🚀

[!IMPORTANT] fitz-forge plan runs local-first by default. A local model does the planning work, fitz-sage handles code retrieval, SQLite stores jobs and checkpoints, and optional API review only runs when you configure it.

pip install fitz-forge

fitz-forge prep
fitz-forge plan "Add WebSocket support to the chat API"

The result is a planning artifact: repository context, implementation status, architectural decisions, proposed code artifacts, review findings, and commands a human or coding agent can use to verify the work.

About

fitz-forge is a working LLM harness for agentic coding planning. It reads a real repository, gathers relevant context, decomposes the requested change, generates implementation artifacts, validates cross-file consistency, and writes a plan that a human or coding agent can execute.

⭐ The planning pipeline is deliberately decomposed. Instead of one giant prompt, local models work through focused retrieval, implementation checks, decision decomposition, decision resolution, synthesis, artifact generation, and review passes.

⭐ The harness carries the exactness. fitz-sage retrieval, structural indexes, AST passes, call graphs, grounding checks, and artifact-set closure checks catch classes of mistakes that prompt-only planning tends to miss.

⭐ Evaluation is built around golden-plan taxonomies. For benchmark tasks, a strong model such as Claude Sonnet helps author the ideal plan taxonomy, and generated plans are scored against that explicit answer key.

Yan Fitzner — (LinkedIn, GitHub, HuggingFace).

Why fitz-forge?

Planning artifact as the contract 🧾

Every job produces a markdown artifact with codebase context, implementation status, decisions, proposed artifacts, review findings, risks, and verification commands. The plan is meant to be inspected, handed to a coding agent, or used by a human engineer.

Harnessed local models 🧠

Local coding models are much more useful when the harness narrows each call to something the model can do reliably: inspect this context, resolve this decision, write this artifact, critique this stage.

Codebase-grounded context 🗂️ → Agent Context Gathering

fitz-sage retrieval, structural indexes, interface signatures, import context, and AST-derived summaries feed the planning pipeline before design decisions are made.

Senior-engineer review layer 🧑‍💼 → Senior Engineer Reviews

Narrow review passes critique decomposition, assumptions, architecture, design, semantic artifact quality, and artifact coverage. A failed review can regenerate the affected stage; a failed review pass leaves the original output unchanged.

Closed artifact sets 🔒

Artifact generation is checked as a set, not just one file at a time. Closure checks look for missing cross-file symbols, unresolved imports, invalid kwargs, async/sync misuse, and typed field access errors.

MCP-ready workflow 🔌

The CLI and MCP server share the same service layer. fitz-forge can run standalone, or sit behind Claude Desktop, Claude Code, Codex, or another MCP-capable client.

Local-first execution 🏠

Ollama, LM Studio, and llama.cpp are supported local backends. Jobs are stored in SQLite and long runs can be resumed from checkpoints.

Planning Artifact

fitz-forge treats the plan as the output contract.

It is not a guaranteed patch generator. The artifact code is intended to be concrete enough for a human or coding agent to inspect, adapt, and implement.

Pipeline

Pipeline Docs • Architecture • Configuration

fitz-forge runs a 10-stage planning pipeline with deterministic code analysis between LLM calls.

The important design choice is that a senior-engineer review layer wraps the pipeline. It runs narrow critique passes scoped to one stage's output, then regenerates the affected stage when a review flags issues.

     USER PROMPT
          │
          ▼
┌─── 🧑‍💼 SENIOR ENGINEER REVIEW LAYER ───────────────────────────────┐
│   detect → regenerate → re-review → keep whichever is better      │
│                                                                   │
│  ┌─────────────────────────────────────────────────────────┐      │
│  │                                                         │      │
│  │   1. Agent Context Gathering       [6-8 LLM]            │      │
│  │   2. Implementation Check          [1 LLM]              │      │
│  │   3. Call Graph Extraction         [0 · AST]            │      │
│  │   4. Decision Decomposition        [2-4 LLM]   ◂───────────────── decomposition review
│  │   5. Decision Resolution           [10-15 LLM]          │      │
│  │   6. Synthesis                     [~18 LLM]            │      │
│  │      ├─ context assumptions                    ◂───────────────── assumption review
│  │      ├─ architecture pick                      ◂───────────────── architecture review
│  │      └─ design spec                            ◂───────────────── design review
│  │   7. Artifact Generation           [3-8 LLM]            │      │
│  │      ├─ per-file code                          ◂───────────────── semantic review
│  │      └─ set-level coverage                     ◂───────────────── coverage review
│  │   8. Grounding Validation          [0-5 LLM]            │      │
│  │   9. Coherence Check               [1 LLM]              │      │
│  │  10. Render + Write                [0]                  │      │
│  │                                                         │      │
│  └─────────────────────────────────────────────────────────┘      │
│                                                                   │
│  Fail-safe: any review error keeps the original output unchanged. │
└───────────────────────────────────────────────────────────────────┘
          │
          ▼
    ~/.fitz-forge/plans/plan_<id>.md

[!NOTE] The current Python-heavy benchmark path is roughly 45-70 LLM calls and about 8-12 minutes on an RTX 5090. Hardware, model, source size, and optional review settings change that substantially.

Evaluation

Scorer V2 Spec • Example Taxonomy • Golden-Plan Authoring Harness

The core evaluation problem is simple:

How do you score a planning artifact when there is no single correct patch yet?

fitz-forge answers that with a task-specific golden-plan taxonomy. For each benchmark challenge, a frontier model such as Claude Sonnet is used to inspect the target repository and draft the ideal planning artifact: the right architecture, the files that matter, the expected implementation shape, and the common wrong approaches. That draft is reviewed into a taxonomy before local-model output is scored.

Example architecture tiers for a streaming feature:

The scorer combines deterministic checks with taxonomy classification. Deterministic checks catch exact failures such as missing required files, parse errors, NotImplementedError, sys.stdout, missing yield, fabricated symbols, unresolved imports, cross-artifact method mismatches, and invalid roadmap commands. Taxonomy classification then maps the plan's architecture and critical artifacts onto the golden answer key.

[!NOTE] Plans generated during normal use do not carry a score. The scorer is an offline evaluation harness for measuring the planning pipeline and comparing raw local-model output against the same model inside the harness.

pip install fitz-forge

pip install "fitz-forge[api-review]"  # Anthropic API review pass
pip install "fitz-forge[dev]"         # pytest, ruff, build tools

fitz-forge prep

fitz-forge plan "Build a plugin system for data transformations"

fitz-forge plan --detach "Build a plugin system for data transformations"
fitz-forge run
fitz-forge list
fitz-forge get <id>

git clone https://github.com/yafitzdev/fitz-forge.git
cd fitz-forge
pip install -e ".[dev]"

fitz-forge prep                 # Configure local model provider
fitz-forge plan "description"   # Create and run a planning job inline
fitz-forge plan --detach "..."  # Queue only
fitz-forge run                  # Process detached queued jobs
fitz-forge list                 # Show jobs
fitz-forge status <id>          # Check progress
fitz-forge get <id>             # Print completed plan
fitz-forge resume <id>          # Resume failed/interrupted job with live UI
fitz-forge retry <id>           # Re-queue failed/interrupted job
fitz-forge confirm <id>         # Approve optional API review
fitz-forge cancel <id>          # Skip optional API review
fitz-forge serve                # Start MCP server

QUEUED → RUNNING → COMPLETE
                  → AWAITING_REVIEW → QUEUED (confirm) / COMPLETE (cancel)
                  → FAILED / INTERRUPTED (both retryable)

{
  "mcpServers": {
    "fitz-forge": {
      "command": "fitz-forge",
      "args": ["serve"]
    }
  }
}

┌─────────────────────────────────────────────────────────────────┐
│                         fitz-forge                              │
├─────────────────────────────────────────────────────────────────┤
│  User Interfaces                                                │
│  CLI: prep | plan | run | list | status | get | serve           │
│  MCP: create_plan | check_status | get_plan | retry_job         │
├─────────────────────────────────────────────────────────────────┤
│  Shared Service Layer                                           │
│  fitz_forge/tools: CLI and MCP call the same operations          │
├─────────────────────────────────────────────────────────────────┤
│  Job Store                                                       │
│  SQLiteJobStore: queued jobs | state | checkpoints | plans       │
├─────────────────────────────────────────────────────────────────┤
│  Worker                                                         │
│  BackgroundWorker: sequential execution and crash recovery       │
├─────────────────────────────────────────────────────────────────┤
│  Planning Pipeline                                               │
│  retrieval | decisions | synthesis | artifacts | validation      │
├─────────────────────────────────────────────────────────────────┤
│  LLM Providers                                                   │
│  Ollama | LM Studio | llama.cpp | optional Anthropic review      │
└─────────────────────────────────────────────────────────────────┘

Development 🧪

pytest
ruff check fitz_forge/
ruff format --check fitz_forge/ tests/

This is alpha software. Treat CI failures as blockers before publishing a release or using benchmark numbers as project claims.

License

MIT

Links

• GitHub
• PyPI
• Changelog
• Contributing

Documentation:

• Feature Docs
• Architecture
• Configuration Reference
• Scorer V2 Spec
• Senior Engineer Reviews
• Agent Context Gathering
• Artifact Generation
• Grounding Validation
• Troubleshooting