effgen 0.2.5

A comprehensive framework for building agents with Small Language Models

effGen

Build AI Agents with Small Language Models

Fast • Efficient • Powerful

---

📰 News & Updates

	Date	Update
🚀	18 May 2026	v0.2.5 Released: 13 new free tools — PubMed, ArXiv, SemanticScholar, RSS, News, YouTubeTranscript, YouTubeMetadata, Reddit, HackerNews, Translate, LanguageDetect, QRGenerate, QRRead. 44+ built-in tools total. See changelog
🚀	14 May 2026	v0.2.4 Released: ModelRouter with CostBased/LatencyBased/FirstAvailable policies, transparent provider failover, cross-process SQLite rate-limit coordination, persistent cost tracker + effgen cost dashboard CLI. See changelog
🚀	4 May 2026	v0.2.3 Released: 5 new cloud backends (Groq, Together AI, Fireworks, Replicate, HuggingFace Inference) — 9 providers total. Unified ProviderRegistry, effgen doctor auth check, backend parity matrix. See changelog
🚀	25 Apr 2026	v0.2.1 Released: Cerebras backend (4 free-tier models, streaming, native tool-calling, rate-limit coordinator, cost tracking) + OpenAI gpt-5/gpt-5.4-nano/o-series with reasoning_effort, prompt caching, structured outputs v2, and OpenAI native tools (web_search, code_interpreter, file_search). See changelog
🚀	9 Apr 2026	v0.2.0 Released: Major release — native tool calling, guardrails, multi-agent orchestration, RAG pipeline, 31 tools, eval framework, production API server, MLX Apple Silicon support, Python & TypeScript SDKs. See changelog
🍎	8 Apr 2026	MLX & Apple Silicon support merged (PR #4): Native Metal GPU acceleration via MLX & MLX-VLM backends. pip install effgen[mlx]
🔧	25 Mar 2026	v0.1.3 Released: Verification hardening — smarter loop detection, "skip the tool" prompting, model-aware token counting, sub-agent depth limits, circuit breaker persistence. See changelog
🔧	12 Mar 2026	v0.1.2 Released: Test-driven hardening — 10 example agents, 19 bug fixes, cross-model compatibility matrix (11 models, 73% pass rate). See changelog
🔒	6 Mar 2026	v0.1.1 Released: Stabilization — fixed license/metadata consistency, improved error handling, added 6 examples, expanded test suite. See changelog
🎉	1 Mar 2026	v0.1.0 Released: Major feature release — 14 built-in tools, agent presets, plugin system, real streaming, memory integration, ACP/MCP protocols, CI/CD, and comprehensive test suite. See changelog
🔧	3 Feb 2026	v0.0.2 Released: vLLM backend fixes with automatic chat template support, GPU memory control, improved OOM error handling, and multi-model family compatibility
📄	2 Feb 2026	Preprint available: EffGen: Enabling Small Language Models as Capable Autonomous Agents
🚀	31 Jan 2026	Initial release of effGen framework (v0.0.1)

---

🤔 What is effGen?

effGen transforms Small Language Models into powerful AI agents. While most frameworks require massive LLMs, effGen is optimized from the ground up for efficient, smaller models — delivering fast, capable agents without the compute overhead.

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator, PythonREPL

# Load a small but mighty model
model = load_model("Qwen/Qwen2.5-1.5B-Instruct", quantization="4bit")

# Create agent with tools
config = AgentConfig(
    name="math_agent",
    model=model,
    tools=[Calculator(), PythonREPL()]
)
agent = Agent(config=config)

# Run computation
result = agent.run("What is 24344 * 334?")
print(f"Answer: {result.output}")

---

⚡ Installation

Requires Python 3.10 or newer. Tested on Python 3.10, 3.11, 3.12, 3.13.

📦 From PyPI (Recommended)

pip install effgen

🍎 Apple Silicon (MLX)

pip install effgen[mlx]          # Text models on Apple Silicon
pip install effgen[mlx-vlm]      # Vision-Language models on Apple Silicon

🚀 With vLLM for Faster Inference

pip install effgen[vllm]

📊 Optional Extras

pip install effgen[cerebras]  # Cerebras inference backend (cerebras-cloud-sdk)
pip install effgen[rag]       # RAG pipeline (sentence-transformers, faiss-cpu)
pip install effgen[finance]   # Finance tools (yfinance)
pip install effgen[data]      # Data science tools (matplotlib, plotly)
pip install effgen[eval]      # Evaluation (rouge-score, nltk)
pip install effgen[gguf]      # GGUF model support (llama-cpp-python)

🔧 From Source

git clone https://github.com/ctrl-gaurav/effGen.git
cd effGen

# Quick install
./install.sh

# Full install (includes vLLM + dev tools)
./install.sh --full

# Manual install
pip install -e .

---

🚀 Quick Start

💻 CLI Usage

# Run a task
effgen run "What is the capital of France?"

# Interactive chat
effgen chat

# Start API server
effgen serve --port 8000

# Interactive wizard
effgen

🐍 Python API

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

# Load model
model = load_model("Qwen/Qwen2.5-1.5B-Instruct", quantization="4bit")

# Configure agent
config = AgentConfig(
    name="calculator_agent",
    model=model,
    tools=[Calculator()],
    system_prompt="You are a helpful math assistant."
)

# Create and run
agent = Agent(config=config)
result = agent.run("Calculate 15% tip on $85.50")
print(result.output)

---

✨ Features

🧠 SLM Optimized Small models

🍎 Apple Silicon MLX + Metal GPU

🛡️ Guardrails PII, injection, safety

📚 RAG Pipeline Ingest, search, cite

👥 Multi-Agent DAG workflows

🔧 31 Tools + MCP/A2A/ACP

🏭 Production API OpenAI-compat

---

🆕 ModelRouter — Smart Multi-Provider Routing (v0.2.4)

Route requests across 9 cloud providers automatically — pick the cheapest, fastest, or first available:

from effgen import PolicyBasedRouter, RoutingContext, CostBasedPolicy, LatencyBasedPolicy
from effgen.models.capabilities import Capability

# Build a router: try fastest first, fall back to cheapest
router = PolicyBasedRouter(policies=[LatencyBasedPolicy(), CostBasedPolicy()])

ctx = RoutingContext(
    prompt_tokens_estimate=500,
    user_budget_usd=0.01,       # stay within $0.01
    latency_budget_ms=3000,     # need response in under 3s
    required_capabilities={Capability.chat},
)

decision = router.route(ctx)
print(decision.chosen)      # e.g., ProviderModelPair("cerebras", "llama3.1-8b")
print(decision.eliminated)  # [(pair, reason), ...] — fully explainable

Transparent failover — route_and_execute retries on rate-limits, 5xx errors, or timeouts and seamlessly moves to the next-best provider.

Cost dashboard — track every API call:

effgen cost today          # per-provider per-model table
effgen cost week           # rolling 7-day view
effgen cost set-budget 1.0 # set $1/day cap

---

🎯 Agent Presets

Get started instantly with ready-to-use agent configurations:

from effgen import load_model
from effgen.presets import create_agent

model = load_model("Qwen/Qwen2.5-3B-Instruct", quantization="4bit")

# One-line agent creation
math_agent = create_agent("math", model)       # Calculator + PythonREPL
research_agent = create_agent("research", model) # WebSearch + URLFetch + Wikipedia
coding_agent = create_agent("coding", model)     # CodeExecutor + PythonREPL + FileOps + Bash
general_agent = create_agent("general", model)   # All 11 tools
minimal_agent = create_agent("minimal", model)   # Direct inference, no tools

# CLI preset support
effgen run --preset math "What is sqrt(144)?"
effgen run --preset research "Tell me about quantum computing"

---

🛠️ Built-in Tools (31)

🔢 Calculator Math & Units	🌐 WebSearch DuckDuckGo	💻 CodeExecutor Sandboxed	🐍 PythonREPL Interactive	📁 FileOps Read/Write	🔍 Retrieval RAG + BM25	🎯 AgenticSearch ripgrep
🖥️ BashTool Shell Cmds	🌤️ WeatherTool Open-Meteo	📋 JSONTool Query/Validate	🕐 DateTimeTool Timezones	📝 TextProcessing Regex/Count	🔗 URLFetch Web Scrape	📖 Wikipedia Free API

---

📚 Examples

python examples/basic/basic_agent.py               # Basic agent (Transformers backend)

python examples/basic/basic_agent_vllm.py          # Basic agent (vLLM backend - 5-10x faster)

python examples/web_retrieval/web_agent.py         # Web search agent

python examples/web_retrieval/retrieval_agent.py   # RAG-based retrieval

python examples/web_retrieval/agentic_search_agent.py # Grep-based agentic search

📖 More Examples

Multi-Tool Agent

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator, WebSearch, PythonREPL

model = load_model("Qwen/Qwen2.5-3B-Instruct")

config = AgentConfig(
    name="research_agent",
    model=model,
    tools=[Calculator(), WebSearch(), PythonREPL()],
    system_prompt="You are a research assistant."
)

agent = Agent(config=config)
result = agent.run("Search for the population of Tokyo and calculate what percentage it is of Japan's total population")

Streaming

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

model = load_model("Qwen/Qwen2.5-3B-Instruct", quantization="4bit")
agent = Agent(config=AgentConfig(
    name="stream_demo", model=model,
    tools=[Calculator()], enable_streaming=True
))

for token in agent.stream("What is 2 + 2?"):
    print(token, end="", flush=True)

Memory (Multi-Turn)

agent = Agent(config=AgentConfig(
    name="memory_demo", model=model,
    tools=[], enable_memory=True
))

agent.run("My name is Alice and I'm working on quantum computing.")
result = agent.run("What's my name and what am I working on?")
# → "Your name is Alice and you're working on quantum computing."

Retrieval Agent (RAG)

from effgen.tools.builtin import Retrieval

retrieval_tool = Retrieval(knowledge_base_path="./docs")
config = AgentConfig(name="qa_agent", model=model, tools=[retrieval_tool])
agent = Agent(config=config)
result = agent.run("What does the documentation say about configuration?")

---

🔒 Security

🐳 Docker Sandbox Isolated execution

🛡️ Input Validation Auto sanitization

⚡ Rate Limiting Configurable limits

📋 For security policies and vulnerability reporting, see SECURITY.md

---

📖 Citation

If you use effGen in your research, please cite our paper:

@software{srivastava2026effgen,
      title={effGen: Enabling Small Language Models as Capable Autonomous Agents},
      author={Gaurav Srivastava and Aafiya Hussain and Chi Wang and Yingyan Celine Lin and Xuan Wang},
      year={2026},
      eprint={2602.00887},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.00887},
}

---

🔗 Links

---

📄 License

Apache License 2.0 — see LICENSE for details.

---

Made with ❤️ for the AI community