svarog-crucible 0.1.0

Crucible. The touchstone — benchmark and verification tool for multi-agent dispatch.

Crucible

The touchstone for multi-agent dispatch — fire that tests your metal

---

In a smithy, the crucible is where raw ore meets fire — a test of what's true metal and what's dross.

Existing benchmarks measure what an agent produces. None measure how the work is divided and gathered.

SWE-bench    → Can a single agent write code?
GAIA         → Can an agent answer questions?
τ-bench      → Can an agent browse?
AgentBench   → Can an agent use tools?
BFCL         → Can an agent call functions?

Crucible asks a different question:

When you spawn 5 agents in parallel, how much do they cost? Is nested (depth=2) more efficient than flat (depth=1)? Which pattern wins: 3 deep or 5 shallow?

---

For whom?

You are...	You ask...	Crucible answers
Building an agent framework	Which dispatch pattern should I use?	Real metrics for 6 patterns
Choosing between CrewAI / LangGraph / AutoGen	Which architecture costs less?	Per-pattern cost breakdown
Shipping a production multi-agent system	What's the cheapest reliable pattern?	Producer-Reviewer: $0.05, 0 errors
Writing a research paper on agents	Where's the evidence?	19 sub-agents, 3+ runs each, live API calls, raw data

---

Quick Start

git clone https://github.com/SvarogForge/crucible
cd crucible
hermes -f scenarios/flat_5.txt

3 minutes later you have real metrics for 6 dispatch patterns.

---

Results

19 sub-agents, 6 patterns, live API calls. Zero simulation.

Pattern	Time	Cost $	Errors	Coverage
Flat	2:29 ⚡	$0.12	0	~60%
Nested	7:14	$0.14	0	~90%
Sequential	4:15	$0.19	0	~70%
Hybrid	7:47	$0.53	8	~75%
Producer-Reviewer	5:20	$0.05 💰	0	~82%
Consensus	4:52	$0.12	0	~90% 🛡️

Picking a Pattern

Need	Try	Why
Fastest execution	Flat	2:29 — 2-3× faster than alternatives
Deepest coverage	Nested	85-95% completeness at depth=2
Lowest cost	Producer-Reviewer	$0.05 — built-in review pass
Most reliable	Consensus	3 couriers → 1 result, variation <0.3%
Balanced	Sequential	4:15, $0.19 — predictable

---

Methodology

Principles

1. Identical topics — only the architecture differs
2. Identical toolsets — web + terminal for everyone
3. Multiple runs — minimum 3 per scenario
4. Automatic metrics — real API returns cost, tokens, tool_trace
5. No simulations — only live API calls

Research Topics

Topic	Data Source
Top-5 OS AI Agent Frameworks by GitHub stars	GitHub API
Local TTS Solutions (CosyVoice, Piper, XTTS, Silero)	GitHub API
Top-5 SDXL Checkpoints by downloads (July 2026)	CivitAI API
RAG Architectures (LlamaIndex, LangChain, Qdrant)	GitHub API
AI Agent Monitoring (LangSmith, W&B, Arize)	GitHub API

7 Core Metrics

Metric	Objectivity	Collection
Wall-clock Time (s)	★★★★★	Timer
Total Tokens	★★★★★	API usage
Cost ($)	★★★★★	tokens × rate
Quality Score (1-10)	★★★☆☆	LLM-as-Judge
Coverage (%)	★★★★☆	Checklist eval
Success Rate (%)	★★★★★	status pass/fail
Efficiency (quality/cost)	★★★★☆	derived

---

Consensus Verification

3 independent agents, same question. Results:

Framework	Agent 1	Agent 2	Agent 3	Variation
LangGraph	36,650	36,650	36,650	0%
CrewAI	55,000	55,033	55,033	~0.06%
AutoGen	59,500	59,538	59,538	~0.06%
Pydantic AI	18,300	18,250	18,300	~0.27%
Semantic Kernel	28,300	28,275	28,300	~0.09%

Sub-agents produce consistent, reproducible data. Variation under 0.3% across all tested frameworks.

---

Setup

JSON Schema

{
  "test_id": "crucible-flat-20260707",
  "pattern": "flat",
  "model": "deepseek/deepseek-v4-flash",
  "config": { "max_concurrent_children": 5, "repeats": 3 }
}

---

Why Crucible?

2026. There are at least 15 active agent frameworks with different dispatch strategies. CrewAI uses flat delegation. LangGraph has graph-based routing. AutoGen has nested conversations. Pydantic AI uses sequential plans.

Which one is cheaper? Which one covers more ground? Which one should you build on?

Nobody knew. Not until the fire was lit.

Crucible is the first benchmark that isolates dispatch architecture from agent capability. Same topics. Same tools. Same models. Only the pattern differs. The result: real, reproducible data on what works and what costs.

---

🔥 From the forge-fire

Crucible is part of SvarogForge — a family of tools forged for Hermes Agent.

Project	Description
⚡ Talaria	Seven-league messengers for market research
🔥 Crucible	Touchstone for quality & benchmarks (you are here)
⚒️ Forge	The smithy itself — AI-powered project forge

📄 License

MIT — open for everyone.

---

⭐ Star on GitHub · 🐦 Follow @iMonstra · 💬 Join Discussions

Built for Hermes Agent · MIT · Contributions welcome