crewforge

One crew of LLM agents, four recipes — code review, document Q&A, API scaffolding and content — on any OpenAI-compatible provider, and it fails loudly instead of pretending.

crewforge review src/                                  # six reviewers + a summary
crewforge ask handbook.pdf -q "How many leave days?" --verify
crewforge scaffold "Todo API: list, create and delete todos" -o todo-api
crewforge write "Why unit tests matter" -f markdown -f html

_{A real run of crewforge review: seven agents, ~25 s, provider-reported tokens. Recorded with VHS from docs/demo/review.tape using Gemini 3.8 Flash through an OpenAI-compatible gateway (the custom provider). The wait for the model is shown in full.}

Why one crew, four recipes

crewforge started as four separate command-line tools — codeguard, documind, apiforge and contentforge. Each had its own copy of the same scaffold: provider presets, an HTTP client, an agent base class, an asyncio pipeline and a token tracker. Running all four against a real model showed the copies had drifted and that each hid its failures differently:

• codeguard reported success when every review came back empty;
• documind's ask command was a stub;
• apiforge scored a spec with no endpoints 100/100;
• contentforge wrote the real API key into its YAML file, and sent ${OPENAI_API_KEY} literally as the key.

crewforge keeps one core — so a fix lands once — and turns each tool into a recipe: a set of agents plus the steps that connect them. The same core rules apply to every recipe: non-streaming requests, bounded retries, usage numbers taken only from the provider, keys only from the environment, and no run that reports success when an agent produced nothing.

60-second quickstart

pip install crewforge
# optional extras: PDF/DOCX ingestion, and the FastAPI app check in `validate`
pip install "crewforge[docs,api]"
# or the latest main: pip install "crewforge @ git+https://github.com/aimanmalib/crewforge.git"

export OPENAI_API_KEY=sk-...          # or any provider below
crewforge providers                   # list presets and the variable each one reads
crewforge review path/to/file.py

Any provider works the same way:

export GROQ_API_KEY=...
crewforge --provider groq review src/

# any other OpenAI-compatible server (vLLM, LM Studio, a gateway, ...)
crewforge --provider custom --base-url http://localhost:8000/v1 --model my-model ask notes.md -q "..."

crewforge init writes a crewforge.yaml you can pass with --config; the key in it is a ${VAR} reference, never the value.

Recipes

Each example below was run against a real model (a Gemini Flash model behind an OpenAI-compatible router) while building crewforge. Output is abbreviated; token tables are trimmed to their totals.

review

$ crewforge review app.py -f markdown -o review.md      # app.py builds SQL with an f-string
# crewforge review

**Decision:** REQUEST_CHANGES · **Quality score:** 25/100

### Top issues
1. Critical SQL injection vulnerability caused by string interpolation in database query (CWE-89)
2. Missing unit tests covering basic retrieval, missing records, and inputs with quotes
3. Missing docstring and incomplete type annotations for the 'name' parameter and return value

### Critical (3)
- **SQL Injection via string interpolation** (line 4) [CWE-89] — _security_: ...
  - Fix: conn.execute('SELECT * FROM users WHERE name = ?', (name,)).fetchall()
...
total   7 calls   22,366 tokens (provider-reported)

ask and summarize

$ crewforge ask kestrel.md -q "How many sensors does Project Kestrel use, and who leads it?" --verify --export session.md
✓ ingested kestrel.md (1 chunks)
Q: How many sensors does Project Kestrel use, and who leads it?
Project Kestrel uses 42 water-level sensors deployed along the Sarawak River, and the project is
led by Nadia Rahman [Source 1].
Confidence: high
Verification: verified (2/2 claims verified)
Exported to: session.md
total   2 calls   5,053 tokens (provider-reported)

$ crewforge summarize kestrel.md --length short
Summary: Project Kestrel is a river-monitoring pilot initiative launched in March 2025 by the
Kuching office under the leadership of Nadia Rahman. ...
total   1 call    2,821 tokens (provider-reported)

scaffold and validate

$ crewforge scaffold "Todo API: list todos, create a todo with a title, mark a todo done, and delete a todo. Bearer auth." -o todo-api
Wrote 11 files
todo-api/
  .env.example  API.md  Dockerfile  auth.py  main.py  models.py  openapi.json
  requirements.txt  routes.py  tests/conftest.py  tests/test_api.py
Validation: valid · 100/100 (4 operation(s) checked, 0 error(s), 0 warning(s))
App check: OK — app serves 5 route(s)
total   5 calls   20,711 tokens (provider-reported)

$ cd todo-api && pytest -q tests
4 passed

$ crewforge validate todo-api            # offline, no key needed
Validation: valid · 100/100 (4 operation(s) checked, 0 error(s), 0 warning(s))
App check: OK — app serves 5 route(s)

The first real run of this recipe produced a project whose own smoke test failed (DELETE /todos/1 correctly answered 404 and the generated test did not allow it); that was fixed with a regression test before this output was taken.

write

$ crewforge write "Why unit tests matter" --words 300 --no-seo --max-iter 1 -f markdown -f html -o article
╭──────────── crewforge write — Why unit tests matter ────────────╮
│ Article: 711 words · quality score 88/100                        │
│ Files:                                                           │
│   article/why-unit-tests-matter.md                               │
│   article/why-unit-tests-matter.html                             │
│   article/why-unit-tests-matter.publisher.json                   │
│ Metrics: article/metrics/20261020_095052.json                    │
╰──────────────────────────────────────────────────────────────────╯
total   6 calls   35,351 tokens (provider-reported)

Two honest caveats from that run: the model wrote more than twice the requested length (--words is a target given to the agents, not a hard limit), and the research agent has no web access, so statistics it quotes come from the model's training data. Review articles before publishing them.

Providers

The table is checked against the code by tests/test_readme.py.

Resolution order for the key, base URL and model: command-line flag or crewforge.yaml → CREWFORGE_API_KEY / CREWFORGE_BASE_URL / CREWFORGE_MODEL → the provider's own variables (e.g. OPENAI_BASE_URL) → the preset default. Pick the provider with --provider or CREWFORGE_PROVIDER; the default is openai. An unknown provider name is a configuration error, not a silent fallback.

Architecture

flowchart LR
    CLI["crewforge CLI<br/>review · ask · summarize · ingest<br/>scaffold · validate · write"] --> R

    subgraph R["Recipes (src/crewforge/recipes)"]
        RV[review]
        DC[docs]
        AP[api]
        CT[content]
    end

    R --> CREW

    subgraph CORE["Core (src/crewforge/core)"]
        CREW["Crew<br/>step() · parallel()<br/>names failed agents"]
        AG["Agent<br/>ask() · ask_json()<br/>rejects empty output"]
        PA["parsing<br/>tolerant JSON + schema"]
        CL["LLMClient<br/>stream=false · retries 429/5xx only"]
        US["UsageTracker<br/>provider-reported tokens"]
        CF["LLMConfig + provider presets<br/>keys from env only"]
        CREW --> AG
        AG --> PA
        AG --> CL
        AG --> US
        CL --> CF
    end

    CL -->|"/chat/completions"| P[("Any OpenAI-compatible provider")]

• core/client.py — one async client. It always sends "stream": false, retries only HTTP 429, 5xx and network/timeout errors (with capped exponential backoff that honours Retry-After), and fails on the first 401/403/4xx with a message naming the variable to fix. The key is only ever a request header.
• core/agent.py — an agent is a role, a system prompt and per-agent sampling defaults. ask() records usage and raises if the reply is blank; ask_json() additionally requires parseable JSON (optionally a pydantic schema).
• core/crew.py — runs agents as sequential steps or parallel stages and turns any failure into a CrewFailedError that lists every failed agent.
• core/usage.py — per-agent calls, tokens, retries and latency, taken from the provider's usage object. Calls without usage are counted as "not reported", never estimated.

Use the core directly:

import asyncio
from crewforge.core import Agent, Crew, LLMConfig


class Haiku(Agent):
    name = "poet"
    system_prompt = "You write one haiku. Nothing else."


async def main():
    async with Crew("demo", LLMConfig(provider="groq")) as crew:
        reply = await crew.step("poet", crew.agent(Haiku).ask("autumn in Kuching"))
        print(reply.content, crew.tracker.one_line())


asyncio.run(main())

Honest failure

A run whose agents return empty or unparseable output, or a validator that checked nothing, fails with exit code 1 and names the agent. It never prints a success message over a hollow result.

$ crewforge ask kestrel.md -q "Who leads Project Kestrel?"        # with a wrong key
error: docs failed
  ✗ agent 'answerer' (http): HTTP 401 from <base-url> (authentication failed — check OPENAI_API_KEY): API key required for remote API access

$ crewforge validate empty-spec.json
Validation: INVALID · 0/100 (0 operation(s) checked, 1 error(s), 0 warning(s))
  ERROR: No endpoints defined (paths is empty)

Exit codes: 0 success · 1 run failed (agent, validator or provider) · 2 usage or configuration error (bad flag, unknown provider, missing key).

The guards are covered by tests that were checked by removing each guard and watching them fail; see docs/porting.md.

How this was built

crewforge was built with AI coding assistance (Claude Code) under the direction of its author, Muhammad Aiman. It consolidates four earlier tools by the same author — codeguard, documind, apiforge and contentforge — whose test suites were carried over into this repository (see docs/porting.md for what was ported, re-targeted at the shared core, or dropped and why).

The evidence that it works is meant to be checkable: the test suite (run pytest; it needs no key and no network), CI on Python 3.10–3.13, and the real-provider runs shown above, two of which found bugs that were fixed with tests before the output was recorded.

Contributing

See CONTRIBUTING.md. In short: pip install -e ".[dev,docs]", pytest, ruff check . && ruff format --check ., and every new failure path needs a test that fails if the guard is removed.

Security

Please report vulnerabilities privately through GitHub private vulnerability reporting. See SECURITY.md for scope notes (what is sent to your provider, and what validate executes).

License

MIT © Muhammad Aiman

Built by Sasa Teguh · Malaysia