flash-attn is not in [all] on purpose: its own setup.py imports torch before pip's isolated build environment has torch installed (a well-known upstream bug), so bundling it would break pip install effgen[all] for everyone. Install it in two steps instead:

pip install effgen[all]                       # step 1: gets torch + the rest
pip install flash-attn --no-build-isolation   # step 2: reuses the torch from step 1

git clone https://github.com/ctrl-gaurav/effGen.git
cd effGen

./install.sh            # quick install
./install.sh --full     # full install (includes vLLM + dev tools)
pip install -e .        # manual editable install

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

# Native Metal GPU, unified memory, no CPU-GPU transfer
model = load_model("LiquidAI/LFM2.5-1.2B-Instruct-MLX-8bit", engine="mlx")

agent = Agent(config=AgentConfig(name="mlx_agent", model=model, tools=[Calculator()]))
result = agent.run("What is sqrt(144) + 2^10?")
print(result.output)

A run that stops making progress is asked for its answer, and every run says how it ended. response.termination is "done", "not_possible", "stuck", "tool_failed" or "error". Ten changes are visible to existing code, and the public surface grew from 251 names to 253 with nothing removed or renamed.

• How a run ends. max_turns_without_progress now defaults to 2: after two turns that bring no new tool result, the run is asked for its answer, and a run about to stop on a loop guard gets one closing request with its calls and results. A run whose tool keeps failing on its own side ends tool_failed, which raises RunStoppedError under the default raise_on_error=True, where 1.2.0 returned the model's own text. None restores 1.2.0's loop.
• How a tool call is read. recover_lost_tool_calls now defaults to True: a call written as a Python literal, with raw line breaks or unescaped quotes, or with arguments sent as a string, is read and run; a call missing a required argument is asked for again rather than dispatched empty. A tool's error is never read as a repeated result, and never returned as the answer.
• Measured, not assumed. The first agent with tools for a model served behind base_url, or run on a local engine, measures once what that model does with a tool, stores it in ~/.effgen/capabilities.json, and tool_calling_mode="auto" follows it: a model that answers from memory while holding a search tool is made to call it. effgen doctor shows what was measured; capability_probe=False or EFFGEN_CAPABILITY_PROBE=0 turns it off. New: probe_tool_calling and ToolCallingProbe.
• One agent, many conversations. Overlapping run(), run_async() and stream() calls given session= each read and record only their own conversation, and stream() on an agent bound to a session saves the turn there.
• reasoning_effort reaches a model behind base_url, and Groq.

from effgen import AgentConfig
from effgen.core.agent import TERMINATIONS

config = AgentConfig(model="Qwen/Qwen2.5-1.5B-Instruct", base_url="http://127.0.0.1:8000/v1")
print(config.max_turns_without_progress)   # 2: a run with no new result is asked for its answer
print(config.recover_lost_tool_calls)      # True: a broken tool call is read before it is reported
print(config.capability_probe)             # True: a served or local model is measured once
print(TERMINATIONS)                        # the values response.termination can take

as_in_1_2 = AgentConfig(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    base_url="http://127.0.0.1:8000/v1",
    max_turns_without_progress=None,
    recover_lost_tool_calls=False,
    capability_probe=False,
)

pip install --upgrade effgen
effgen --version

Where it falls short. On task types kept out of this release's development, the accuracy gain over 1.2.0 comes from tasks whose tool fails; elsewhere it is flat. Small models given a search tool cost noticeably more per run, because they are made to use it, and tool-using runs still make more model calls than they need to.

What it cost. Against 1.2.0, the larger model measured makes slightly fewer model calls, sends fewer prompt tokens and takes less wall time, with accuracy up. The smaller model makes half again as many model calls and takes nearly twice the wall time, nearly all of it from being made to use its search tool. No cloud model was measured at full size.

Full v1.3.0 changelog

Every run now keeps a ledger of what it spent and where its time went. response.ledger is a RunLedger: model and tool calls, prompt, completion and cached tokens, cost, and wall time split into model, tool, caller, child and framework time. It matches what goes over the wire. Fourteen changes are visible to existing code, and the public surface grew from 250 names to 251 with nothing removed or renamed.

• A model's invented tool result is never the answer. A tool-holding turn is sent the stop sequence "\nObservation:", and a written action runs while whatever the model wrote after it is discarded. An adapter whose provider rejects stop beside tools declares supports_stop_with_tools() as False, and the text is cut locally instead.
• Prompt caching you can see. A run with tools keeps one request shape on a provider with a prompt cache; cache_system_prompt and cache_tools now work on Anthropic; cached tokens are read on five more providers and priced at the cached rate where the catalog carries one.
• Less of the framework's own text in a request. Tool rules are stated once, a repeated tool result is sent once, and the tool contracts stop asking for text the task did not ask for: requests are shorter, with every answer and call count unchanged. AgentConfig(answer_style=...) asks for a shorter or a fuller answer; the default states nothing.
• Cost that reads true. A model with no published price reads None, not $0; a server reached with base_url= records as openai_compatible; a spent cap no longer refuses local, free or self-hosted calls, and a refusal raises BudgetExceededError. The spend ledger stops growing at 250,000 rows with every total kept exact.
• The loop. A tool that keeps returning new results is no longer withdrawn at 12 calls; run(max_iterations=N) moves the loop's thresholds too; reasoning_effort reaches run() and run_async(). New: max_turns_without_progress= and recover_lost_tool_calls=, opt-in in 1.2.0 and on by default from 1.3.0.
• Local and streamed runs. Concurrent streams keep their own tool arguments; openai:<id> with base_url= sends the id without the prefix; concurrent agents share one in-process vLLM engine; a GGUF run reuses its cache across turns.
• effgen bench. Run a suite of your own tasks against a model, and compare two runs with a noise band beside every difference.

from effgen import Agent, AgentConfig
from effgen.tools.builtin import Calculator

agent = Agent(AgentConfig(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    base_url="http://127.0.0.1:8000/v1",
    tools=[Calculator()],
))
response = agent.run("What is 17 * 23?")
ledger = response.ledger

print(response.output)
print(ledger.llm_calls, ledger.tool_calls, ledger.prompt_tokens, ledger.completion_tokens)
print(f"model {ledger.model_wait_s:.2f} s, tools {ledger.tool_wait_s:.3f} s, "
      f"framework {ledger.framework_s * 1000:.1f} ms")
print(ledger.cost_usd)   # None: a model you serve yourself has no published price

from effgen import AgentConfig

config = AgentConfig(model="openai:gpt-5-nano", answer_style="brief")
print(config.answer_style)                 # brief: one line, stated last
print(config.max_turns_without_progress)   # 2 from 1.3.0; None (off) in 1.2.0
print(config.recover_lost_tool_calls)      # True from 1.3.0; False (off) in 1.2.0

effgen bench init
effgen bench run bench-suite.yaml --model Qwen/Qwen2.5-1.5B-Instruct --base-url http://127.0.0.1:8000/v1 --out runs/a

pip install --upgrade effgen
effgen --version

What it cost. Against 1.1.0, accuracy holds on every task measured after the final fix. Answers that need no tool are far shorter. Arithmetic and math tasks that use a calculator write shorter answers but make more model calls and send more prompt tokens, and question-answering tasks with a search tool search more often and answer more questions correctly. Tool-using runs still make more model calls than they need to, and reducing that is the focus of the next release. No cloud model was measured at full size.

Full v1.2.0 changelog

A run now keeps its conversation as typed steps instead of one growing string. That one change is the release: the loop builds an AgentThread, the prompt is rendered from it, the checkpoint stores it, and the caller can read it. Eleven changes are visible to existing code, and the public surface grew from 225 names to 250 with nothing removed or renamed.

from effgen import Agent, AgentConfig, thread_as_text

agent = Agent(AgentConfig(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    base_url="http://127.0.0.1:8000/v1",
))
response = agent.run("What is 17 * 23?")

print(response.output)
print(thread_as_text(response.thread))        # the run, step by step
print(response.metadata["context_budget"])    # what it was allowed to send

from effgen import AgentConfig

print(AgentConfig(model="openai:gpt-5-nano").prompt_protocol)   # auto
print(AgentConfig(model="openai:gpt-5-nano", prompt_protocol="messages").prompt_protocol)

pip install --upgrade effgen
effgen --version

What it cost. On the same ten public sample sets as the 1.0.1 baseline, a run sends 26% fewer prompt tokens at 1.5B and 18% fewer at 7B, and makes about 16% fewer model calls. It is not faster. Mean accuracy moved −2.67 (1.5B) and −3.53 (7B) with sets weighted equally, −0.42 and −1.48 with samples weighted equally; three sets got worse and two got better outside every band we computed. No cloud model was measured.

Full v1.1.0 changelog

This release fixes how the framework reports what a run did, what it puts in a prompt, and what its own bookkeeping costs. Four changes are visible to existing code, and one of them changes what success means for a run that stopped part way.

from effgen import Agent, AgentConfig, RunStoppedError

agent = Agent(AgentConfig(model="openai:gpt-5-nano"))
try:
    response = agent.run("What is 17 * 23?")
    print(response.outcome, response.stop_reason)
    print(response.text)
except RunStoppedError as exc:
    print(exc.stop_reason)
    print(exc.partial.text if exc.partial else "nothing to report")

effgen runs list --status stopped      # runs that ended before an answer was written
effgen cost prune --older-than-days 30 --dry-run

What it cost. A run makes 37% more model calls and sends 57% more prompt tokens than 1.0.0, and two retrieval sets got worse. The full measurement is in the changelog.

Full v1.0.1 changelog

v1.0.0 is about control over where a model runs and visibility into what a run did — drive any server speaking the OpenAI protocol, read back the calls a run made, extend the agent loop — and it adds the surfaces that make a run easy to drive, watch and share. The largest and least visible part of the release is a pass over everything that used to report the wrong thing confidently: a failed run now says so, an unpriced model reports no cost, and a turn that did nothing is not a success. Three changes are breaking, each with a one-line migration in the changelog.

from effgen.models import load_model

model = load_model(
    "Qwen/Qwen2.5-7B-Instruct",
    provider="openai_compatible",
    base_url="http://127.0.0.1:8000/v1",
)

effgen code "add a --dry-run flag to the importer"   # diffs first, writes on your word
effgen models browse --vision --min-context 128000 --sort price-out
effgen battle "Explain gradient clipping" -m groq:openai/gpt-oss-20b,gemini:gemini-3.1-flash-lite
effgen top                                           # terminal mission control

Full v1.0.0 changelog →

v0.3.2 keeps sanding down the edges — this time for a reliability engineer, a trust auditor, a security engineer, an ETL engineer, a clinical analyst, an SRE, a localizer, a CI gatekeeper, a non-technical operator, a game writer, a plugin author, a FinOps owner, and a document specialist. No new providers or subsystems — the surfaces you already reach for are now more predictable, and every quiet trap now surfaces a clear, typed error. No breaking API changes — every change is additive.

from effgen import PIIGuardrail, get_guardrail_preset

# Redaction that covers the labeled clinical identifiers, plus site-specific patterns.
g = PIIGuardrail(action="redact", custom_patterns=[(r"MRN[:#]\s*\d+", "[MRN REDACTED]")])
print(g.check("Jane Doe  DOB: 1980-02-14  MRN: 55123").modified_content)
# "[NAME REDACTED]  DOB: [DOB REDACTED]  MRN: [MRN REDACTED]"

chain = get_guardrail_preset("phi")   # redaction + fail-closed strict mode

effgen batch --input tickets.jsonl --output out.jsonl -m groq:openai/gpt-oss-20b --schema schema.json
effgen eval --suite cases.jsonl -m groq:openai/gpt-oss-20b --fail-under 0.9   # exit 1 if it drops
effgen compare --models "groq:openai/gpt-oss-20b,gemini:gemini-3.1-flash-lite" --suite cases.jsonl --optimize cost
effgen run "What was Q3 revenue?" --file report.pdf -m groq:openai/gpt-oss-20b

Full v0.3.2 changelog →

from effgen import load_model
from effgen.presets import create_agent

model = load_model("Qwen/Qwen2.5-3B-Instruct", quantization="4bit")

# One-line agent creation
math_agent = create_agent("math", model)        # Calculator + PythonREPL
research_agent = create_agent("research", model) # WebSearch + URLFetch + Wikipedia + academic
coding_agent = create_agent("coding", model)     # CodeExecutor + PythonREPL + FileOps + Bash
general_agent = create_agent("general", model)   # Broad built-in tool suite
rag_agent = create_agent("rag", model, knowledge_base="./docs/")  # RAG pipeline
minimal_agent = create_agent("minimal", model)   # Direct inference, no tools

# CLI preset support
effgen run --preset math "What is sqrt(144)?"
effgen run --preset research "Tell me about quantum computing"

9 presets: math · research · coding · general · rag · minimal · multimodal · notify · media

Browse quickstart snippets for all 66 tools in the full tool gallery.

effGen ships a curated catalog of 35 reusable prompt templates across 8 domains, each with a golden evaluation test and CLI access. Browse the full gallery.

effgen prompts list                            # browse all 35 templates
effgen prompts show research.paper_summary.v1  # inspect a template
effgen prompts eval                            # run golden eval (no model needed)
effgen prompts playground                      # interactive REPL

from effgen.prompts.library import registry

# Get and render a template
p = registry.get("coding.code_review.v1")
prompt = p.template(code="def add(a, b): return a + b", language="python")

# Search templates
cot_prompts = registry.search(variant="cot")
sql_prompts = registry.search(domain="data")

Legal and medical templates enforce a mandatory non-advice disclaimer in every rendered output, verified by unit tests.

effGen supports 9 cloud inference providers, any server that speaks the OpenAI protocol, and 4 local backends, tested across 11+ model families:

from effgen import load_model, Agent
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

# Any of the 9 cloud providers
model = load_model("openai/gpt-oss-20b", provider="groq")          # Groq
# model = load_model("meta-llama/Llama-3.3-70B-Instruct-Turbo", provider="together")
# model = load_model("Qwen/Qwen2.5-72B-Instruct", provider="hf")

agent = Agent(config=AgentConfig(name="agent", model=model, tools=[Calculator()]))
result = agent.run("What is (17 * 23) + sqrt(144)?")
print(result.output)  # → 403

effgen doctor   # see which provider API keys are configured

Top Recommended Models

Full matrix with 11 models × 10 agents: compatibility_matrix.md

Multi-Tool Agent

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator, WebSearch, PythonREPL

model = load_model("Qwen/Qwen2.5-3B-Instruct")
config = AgentConfig(
    name="research_agent",
    model=model,
    tools=[Calculator(), WebSearch(), PythonREPL()],
    system_prompt="You are a research assistant.",
)
agent = Agent(config=config)
result = agent.run("Search for the population of Tokyo and calculate what percentage it is of Japan's total population")

Streaming

from effgen import Agent, load_model
from effgen.core.agent import AgentConfig
from effgen.tools.builtin import Calculator

model = load_model("Qwen/Qwen2.5-3B-Instruct", quantization="4bit")
agent = Agent(config=AgentConfig(
    name="stream_demo", model=model,
    tools=[Calculator()], enable_streaming=True,
))
for token in agent.stream("What is 2 + 2?"):
    print(token, end="", flush=True)

Memory (Multi-Turn)

agent = Agent(config=AgentConfig(
    name="memory_demo", model=model,
    tools=[], enable_memory=True,
))
agent.run("My name is Alice and I'm working on quantum computing.")
result = agent.run("What's my name and what am I working on?")
# → "Your name is Alice and you're working on quantum computing."

Retrieval Agent (RAG)

from effgen.tools.builtin import Retrieval

retrieval_tool = Retrieval(knowledge_base_path="./docs")
config = AgentConfig(name="qa_agent", model=model, tools=[retrieval_tool])
agent = Agent(config=config)
result = agent.run("What does the documentation say about configuration?")

effGen ships deployment recipes for every major target.

docker build -f deploy/docker/Dockerfile -t effgen .
docker run -p 8000:8000 --env-file .env effgen
curl http://localhost:8000/health

helm lint deploy/k8s/helm/effgen/
helm install effgen deploy/k8s/helm/effgen/

cd deploy/aws_lambda
sam build && sam deploy --guided

cd deploy/cloudflare
wrangler deploy   # staging: wrangler deploy --env staging

Secret scanning. Gitleaks pre-commit hook + CI workflow (secret-scan.yml) catch secrets before they reach the repo:

pip install pre-commit && pre-commit install

Sandboxed code execution. CodeExecutor defaults to SubprocessSandbox (rootless user-namespace, network blocked, isolated /tmp) or DockerSandbox when Docker is available. To opt out (not recommended):

EFFGEN_SANDBOX_BACKEND=off effgen run ...   # a loud warning is emitted

API server auth. Protect the server with OAuth2/OIDC (Auth0, Keycloak, Cognito — any OIDC provider):

export EFFGEN_OIDC_ISSUER=https://your-tenant.auth0.com/
export EFFGEN_OIDC_CLIENT_ID=your-client-id
export EFFGEN_OIDC_JWKS_URI=https://your-tenant.auth0.com/.well-known/jwks.json
effgen serve --port 8000

📋 See SECURITY.md for policies and vulnerability reporting, plus docs/server/auth.md, docs/server/rbac.md, and docs/server/audit.md.