crewforge

One crew of LLM agents, five recipes — code review, document Q&A, API scaffolding, content and honest academic-writing help — on any OpenAI-compatible provider, and it fails loudly instead of pretending.

crewforge review src/                                  # six reviewers + a summary
crewforge ask handbook.pdf -q "How many leave days?" --verify
crewforge scaffold "Todo API: list, create and delete todos" -o todo-api
crewforge write "Why unit tests matter" -f markdown -f html
crewforge academic feedback draft.md --brief brief.md --language ms

_{A real run of crewforge review: seven agents, ~25 s, provider-reported tokens. Recorded with VHS from docs/demo/review.tape using Gemini 3.8 Flash through an OpenAI-compatible gateway (the custom provider). The wait for the model is shown in full.}

Why one crew, many recipes

crewforge started as four separate command-line tools — codeguard, documind, apiforge and contentforge. Each had its own copy of the same scaffold: provider presets, an HTTP client, an agent base class, an asyncio pipeline and a token tracker. Running all four against a real model showed the copies had drifted and that each hid its failures differently:

• codeguard reported success when every review came back empty;
• documind's ask command was a stub;
• apiforge scored a spec with no endpoints 100/100;
• contentforge wrote the real API key into its YAML file, and sent ${OPENAI_API_KEY} literally as the key.

crewforge keeps one core — so a fix lands once — and turns each tool into a recipe: a set of agents plus the steps that connect them. The same core rules apply to every recipe: non-streaming requests, bounded retries, usage numbers taken only from the provider, keys only from the environment, and no run that reports success when an agent produced nothing.

A fifth recipe, academic, replaces a standalone "academic writing pipeline" that drafted essays and marketed itself as AI-detector-resistant. Only the honest parts were kept; see docs/porting.md.

60-second quickstart

pip install crewforge
# optional extras: PDF/DOCX ingestion, and the FastAPI app check in `validate`
pip install "crewforge[docs,api]"
# or the latest main: pip install "crewforge @ git+https://github.com/aimanmalib/crewforge.git"

export OPENAI_API_KEY=sk-...          # or any provider below
crewforge providers                   # list presets and the variable each one reads
crewforge review path/to/file.py

Any provider works the same way:

export GROQ_API_KEY=...
crewforge --provider groq review src/

# any other OpenAI-compatible server (vLLM, LM Studio, a gateway, ...)
crewforge --provider custom --base-url http://localhost:8000/v1 --model my-model ask notes.md -q "..."

crewforge init writes a crewforge.yaml you can pass with --config; the key in it is a ${VAR} reference, never the value.

Recipes

Each example below was run against a real model (a Gemini Flash model behind an OpenAI-compatible router) while building crewforge. Output is abbreviated; token tables are trimmed to their totals.

review

$ crewforge review app.py -f markdown -o review.md      # app.py builds SQL with an f-string
# crewforge review

**Decision:** REQUEST_CHANGES · **Quality score:** 25/100

### Top issues
1. Critical SQL injection vulnerability caused by string interpolation in database query (CWE-89)
2. Missing unit tests covering basic retrieval, missing records, and inputs with quotes
3. Missing docstring and incomplete type annotations for the 'name' parameter and return value

### Critical (3)
- **SQL Injection via string interpolation** (line 4) [CWE-89] — _security_: ...
  - Fix: conn.execute('SELECT * FROM users WHERE name = ?', (name,)).fetchall()
...
total   7 calls   22,366 tokens (provider-reported)

ask and summarize

_{A real run of crewforge ask --verify: the answerer cites the passages it used as [Source N], a second agent checks every claim against those passages, and the question the document does not answer is answered as such. Provider-reported tokens. Recorded with VHS from docs/demo/ask.tape (the document is invented) using Gemini 3.8 Flash through an OpenAI-compatible gateway (the custom provider). The wait for the model is shown in full.}

$ crewforge ask kestrel.md -q "How many sensors does Project Kestrel use, and who leads it?" --verify --export session.md
✓ ingested kestrel.md (1 chunks)
Q: How many sensors does Project Kestrel use, and who leads it?
Project Kestrel uses 42 water-level sensors deployed along the Sarawak River, and the project is
led by Nadia Rahman [Source 1].
Confidence: high
Verification: verified (2/2 claims verified)
Exported to: session.md
total   2 calls   5,053 tokens (provider-reported)

$ crewforge summarize kestrel.md --length short
Summary: Project Kestrel is a river-monitoring pilot initiative launched in March 2025 by the
Kuching office under the leadership of Nadia Rahman. ...
total   1 call    2,821 tokens (provider-reported)

scaffold and validate

$ crewforge scaffold "Todo API: list todos, create a todo with a title, mark a todo done, and delete a todo. Bearer auth." -o todo-api
Wrote 11 files
todo-api/
  .env.example  API.md  Dockerfile  auth.py  main.py  models.py  openapi.json
  requirements.txt  routes.py  tests/conftest.py  tests/test_api.py
Validation: valid · 100/100 (4 operation(s) checked, 0 error(s), 0 warning(s))
App check: OK — app serves 5 route(s)
total   5 calls   20,711 tokens (provider-reported)

$ cd todo-api && pytest -q tests
4 passed

$ crewforge validate todo-api            # offline, no key needed
Validation: valid · 100/100 (4 operation(s) checked, 0 error(s), 0 warning(s))
App check: OK — app serves 5 route(s)

The first real run of this recipe produced a project whose own smoke test failed (DELETE /todos/1 correctly answered 404 and the generated test did not allow it); that was fixed with a regression test before this output was taken.

write

$ crewforge write "Why unit tests matter" --words 300 --no-seo --max-iter 1 -f markdown -f html -o article
╭──────────── crewforge write — Why unit tests matter ────────────╮
│ Article: 711 words · quality score 88/100                        │
│ Files:                                                           │
│   article/why-unit-tests-matter.md                               │
│   article/why-unit-tests-matter.html                             │
│   article/why-unit-tests-matter.publisher.json                   │
│ Metrics: article/metrics/20261020_095052.json                    │
╰──────────────────────────────────────────────────────────────────╯
total   6 calls   35,351 tokens (provider-reported)

Two honest caveats from that run: the model wrote more than twice the requested length (--words is a target given to the agents, not a hard limit), and the research agent has no web access, so statistics it quotes come from the model's training data. Review articles before publishing them.

academic

An honest assistant for a student's or researcher's own work. It plans, comments and formats; it does not write the assignment, does not invent references, and has no "humanise" or AI-detector-evasion feature. Output in English or Malaysian Malay (--language en|ms; Malay output is checked for Indonesian forms such as karena or kualitas). Every step is recorded (file names and hashes, never content or keys) so disclose can write the AI-use statement many universities now ask for.

_{A real run of crewforge academic feedback --language ms on a short essay draft in Malay. Each comment quotes the student's own sentence and suggests a fix; nothing is rewritten. The citation check after it uses no AI: it finds (Lim, 2024) cited in the text but missing from the reference list. Only the argument reviewer is run so the result fits on one screen. Recorded with VHS from docs/demo/academic.tape (draft, brief and reference are invented) using Gemini 3.8 Flash through an OpenAI-compatible gateway (the custom provider). The wait for the model is shown in full.}

$ crewforge academic plan brief.md -f markdown          # a 1,200-word TSL3013 essay brief
# Assignment plan: Regulated Integration over Prohibition: ...
- **Target length:** 1,200 words
- **Citation style:** APA 7th edition
## Research questions
1. What pedagogical benefits does generative AI offer secondary school students completing homework tasks?
...
| # | Section | Words | Purpose |
| 1 | Introduction | 150 | ... |   ...   Total: 1,200 words
## Suggested readings (UNVERIFIED)
> The AI proposed these. They may not exist, or may not say what is claimed. ...
- ⚠ **UNVERIFIED: proposed or altered by the AI, check it** — Holmes, W., & Tuomi, I. (2022). ...
total   2 calls   9,034 tokens (provider-reported)

$ crewforge academic feedback draft.md --brief brief.md -f markdown
## Argument
- **Paragraph 2** (major): “Everyone knows that tuition in Malaysia is very expensive.”
  - This is an unsupported generalization and an informal assertion rather than an evidence-based academic claim.
  - *Suggestion:* Replace the appeal to common knowledge with scholarly evidence or empirical data ...
## Citation check (no AI)
**Cited in the text but not in the reference list:**
- Lim, 2024
total   4 calls   17,892 tokens (provider-reported)

$ crewforge academic cite refs.txt --style ieee --language ms -f markdown
[4] W. Holmes, M. Bialik, and C. Fadel, *Artificial Intelligence in Education: ...*. Center for Curriculum Redesign, 2019, ISBN: 978-1-7941-1293-0.
  - ⚠ nombor semak ISBN salah: 9781794112930
  - Butiran yang perlu ditambah: tempat penerbitan

$ crewforge academic disclose --language ms             # no LLM, no key
## Pernyataan Penggunaan AI
Saya menggunakan crewforge academic (versi 0.1.0), sebuah alat berbantukan AI, dengan model gemini-3.8-flash (melalui gateway serasi OpenAI), bagi bahagian tugasan berikut:
- 22 Oktober 2026: Perancangan: alat ini menukar taklimat tugasan (brief-ms.md) kepada 3 soalan kajian, rangka 5 bahagian dan peruntukan perkataan. ...

How sources are labelled — there is no "verified" status, because the tool cannot check that a source exists:

A DOI the model proposes can be made up too, which is why even those stay unverified. All text the student sees lives in src/crewforge/recipes/academic/locales/{en,ms}.yaml; defaults (language, style, log file, input size, suggestion length) are the academic: section of crewforge.yaml.

Providers

The table is checked against the code by tests/test_readme.py.

Resolution order for the key, base URL and model: command-line flag or crewforge.yaml → CREWFORGE_API_KEY / CREWFORGE_BASE_URL / CREWFORGE_MODEL → the provider's own variables (e.g. OPENAI_BASE_URL) → the preset default. Pick the provider with --provider or CREWFORGE_PROVIDER; the default is openai. An unknown provider name is a configuration error, not a silent fallback.

Architecture

_{Source: docs/architecture.mmd (re-render with scripts/render_architecture.py).}

• core/client.py — one async client. It always sends "stream": false, retries only HTTP 429, 5xx and network/timeout errors (with capped exponential backoff that honours Retry-After), and fails on the first 401/403/4xx with a message naming the variable to fix. The key is only ever a request header.
• core/agent.py — an agent is a role, a system prompt and per-agent sampling defaults. ask() records usage and raises if the reply is blank; ask_json() additionally requires parseable JSON (optionally a pydantic schema).
• core/crew.py — runs agents as sequential steps or parallel stages and turns any failure into a CrewFailedError that lists every failed agent.
• core/usage.py — per-agent calls, tokens, retries and latency, taken from the provider's usage object. Calls without usage are counted as "not reported", never estimated.

Use the core directly:

import asyncio
from crewforge.core import Agent, Crew, LLMConfig


class Haiku(Agent):
    name = "poet"
    system_prompt = "You write one haiku. Nothing else."


async def main():
    async with Crew("demo", LLMConfig(provider="groq")) as crew:
        reply = await crew.step("poet", crew.agent(Haiku).ask("autumn in Kuching"))
        print(reply.content, crew.tracker.one_line())


asyncio.run(main())

Honest failure

A run whose agents return empty or unparseable output, or a validator that checked nothing, fails with exit code 1 and names the agent. It never prints a success message over a hollow result.

$ crewforge ask kestrel.md -q "Who leads Project Kestrel?"        # with a wrong key
error: docs failed
  ✗ agent 'answerer' (http): HTTP 401 from <base-url> (authentication failed — check OPENAI_API_KEY): API key required for remote API access

$ crewforge validate empty-spec.json
Validation: INVALID · 0/100 (0 operation(s) checked, 1 error(s), 0 warning(s))
  ERROR: No endpoints defined (paths is empty)

Exit codes: 0 success · 1 run failed (agent, validator or provider) · 2 usage or configuration error (bad flag, unknown provider, missing key).

The guards are covered by tests that were checked by removing each guard and watching them fail; see docs/porting.md.

How this was built

crewforge was built with AI coding assistance (Claude Code) under the direction of its author, Muhammad Aiman. It consolidates four earlier tools by the same author — codeguard, documind, apiforge and contentforge — whose test suites were carried over into this repository (see docs/porting.md for what was ported, re-targeted at the shared core, or dropped and why).

The evidence that it works is meant to be checkable: the test suite (run pytest; it needs no key and no network), CI on Python 3.10–3.14, and the real-provider runs shown above, two of which found bugs that were fixed with tests before the output was recorded.

Contributing

See CONTRIBUTING.md. In short: uv sync --extra dev --extra docs (versions from the committed uv.lock, as in CI), uv run pytest, uv run ruff check . && uv run ruff format --check ., and every new failure path needs a test that fails if the guard is removed.

Security

Please report vulnerabilities privately through GitHub private vulnerability reporting. See SECURITY.md for scope notes (what is sent to your provider, and what validate executes).

License

MIT © Muhammad Aiman

Built by Sasa Teguh · Malaysia