AgentForge
ReAct agents on open-weight LLMs — tools, memory, and an eval harness.
Pairs with ragforge-ml for retrieval and
turboquant-ml for quantized model serving.
Why AgentForge?
Most "agent framework" projects use proprietary models (GPT-4, Claude) behind a
DSL of Runnable.invoke() chains nobody can debug. AgentForge is the opposite:
ReAct loops on open-weight LLMs (Llama, Qwen, Mistral), with a small registry
of well-bounded tools, and an evaluation harness so you can measure whether
your agent is actually doing what you asked.
Three opinions:
- Open models first. Defaults work on
Qwen/Qwen2.5-3B-Instructand any chat-template HF model. No API key required. Plug in turboquant-ml to serve the model quantized. - ReAct, not magic. The loop is a 60-line function (
agent.py:run) that alternates Thought / Action / Observation steps. Easy to read, easy to debug. - Tools have hard boundaries. Python REPL runs in an AST-whitelisted sandbox; SQL is read-only; web search is rate-limited; RAG retrieval is delegated to ragforge-ml.
Features
| Stage | Default |
|---|---|
| LLM | Any HuggingFace chat-template model. Optional bnb-nf4 via turboquant-ml. |
| Loop | ReAct with max_steps, structured Thought/Action/Observation parser |
| Tools | calculator, python (sandboxed), web_search (DuckDuckGo), sql (read-only sqlite), rag (RAGforge) |
| Memory | In-memory conversation, persistent SQLite store |
| Eval | task_completion, tool_accuracy, step_efficiency, final_answer_match |
| Serve | FastAPI /ask, /tools, /health |
| CLI | agentforge ask / eval / tools / serve |
Installation
The PyPI distribution is agentforge-ml (the unsuffixed agentforge name was
taken by an unrelated project). Python import and CLI are just agentforge /
af:
pip install agentforge-ml # core
pip install "agentforge-ml[tools]" # + sympy + duckduckgo-search
pip install "agentforge-ml[rag]" # + ragforge-ml integration
pip install "agentforge-ml[quantized]" # + turboquant-ml NF4 path
pip install "agentforge-ml[serve]" # + FastAPI
pip install "agentforge-ml[all]" # everything
60-second tour
from agentforge import Agent
from agentforge.tools import Calculator, WebSearch, PythonREPL
agent = Agent.from_defaults(
model_id="Qwen/Qwen2.5-3B-Instruct",
tools=[Calculator(), PythonREPL(), WebSearch()],
)
result = agent.run("What is 47 * 1337, then take its square root?")
print(result.final_answer)
for step in result.steps:
print(f" [{step.tool}] {step.action_input!r} -> {step.observation!r}")
With RAG
from agentforge import Agent
from agentforge.tools import RAGTool
from ragforge import Pipeline
rag = Pipeline.from_defaults(model_id="Qwen/Qwen2.5-3B-Instruct")
rag.ingest(["docs/"])
agent = Agent.from_defaults(
model_id="Qwen/Qwen2.5-3B-Instruct",
tools=[RAGTool(rag)],
)
print(agent.run("What is our company refund policy?").final_answer)
CLI
af ask "What is 17 squared?" --tools calculator
af ask "Latest CVE for log4j?" --tools web_search
af eval data/eval_set.jsonl --tools calculator,python_repl
af serve --tools calculator,python_repl --port 8080
ReAct loop, in a picture
question -> [LLM] Thought + Action -> [Tool] Observation
^ |
|_______________________________________|
up to max_steps
If the LLM emits Final Answer: the loop exits. Otherwise it loops until
max_steps. The parser is forgiving: it tolerates whitespace and case but
falls back to the last completed step on truncation.
Eval harness
Built-in, pure Python, no judge model required:
| Metric | What it measures |
|---|---|
task_completion |
Did the agent produce a Final Answer:? |
final_answer_match |
Does the answer contain the ground-truth string (case-folded substring)? |
tool_accuracy |
Of the steps, what fraction used the expected tool? |
step_efficiency |
ground_truth_steps / actual_steps, clipped to [0, 1] |
af eval examples/eval_set.jsonl --tools all
+--------------------+--------+
| metric | mean |
+--------------------+--------+
| task_completion | 0.95 |
| final_answer_match | 0.81 |
| tool_accuracy | 0.88 |
| step_efficiency | 0.72 |
+--------------------+--------+
n=80 · p50=2.4s · p95=8.1s
Architecture
agentforge/
├── core/ # ReAct loop + parser + prompts
├── tools/ # registry, calculator, python repl, web search, sql, rag
├── memory/ # conversation, persistent sqlite
├── llm/ # HuggingFace causal LM wrapper
├── eval/ # 4 metrics + orchestrator
├── serve/ # FastAPI app
└── cli.py # af / agentforge
Every stage is a small module behind a small interface (LLM, Tool,
Memory) — swap any of them in two lines.
Roadmap
- ReAct loop with structured parsing
- Tool protocol + registry
- 5 built-in tools (calculator, python, web, sql, rag)
- Persistent SQLite memory
- Eval: task completion, final-answer match, tool accuracy, step efficiency
- FastAPI server + Typer CLI
- turboquant-ml integration (NF4 / GPTQ / AWQ models)
- Plan-and-execute pattern alongside ReAct
- Streaming step output in
/ask - Tool-use chat templates (Qwen tool format, Llama-3 tool format)
- Multi-agent coordination
License
MIT.
Metadata
Release files for agentforge-ml 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentforge_ml-0.1.0.tar.gz | 31.1 kB | Without provenance | Details for agentforge_ml-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentforge_ml-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for agentforge_ml-0.1.0-py3-none-any.whl |
Total release size: 60.7 kB
Release files / agentforge_ml-0.1.0.tar.gz
| Download URL | agentforge_ml-0.1.0.tar.gz |
|---|---|
| Size | 31.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
03b56da5a658da80cecf3f8e63aa9d37bbcca2e9dddd29555506b4f2c5dd8014
|
|
BLAKE2b-256 checksum How to use checksums |
4e69bcb6e3a4a7411c6735bb9e774765522b559d615136f84d96822e6f8ffda5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Release files / agentforge_ml-0.1.0-py3-none-any.whl
| Download URL | agentforge_ml-0.1.0-py3-none-any.whl |
|---|---|
| Size | 29.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3bfe9f4604392c54abcf3063bf5d2e57001eea84d6963d64556c97777cb7106e
|
|
BLAKE2b-256 checksum How to use checksums |
7f96c668b4ad0d7c19d87882add24a0859cbd9de9b56ca06420064de2271e0c2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Security for agentforge-ml 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.