agentic-fleet 0.6.0

Multi-agent orchestration system built with Microsoft Agent Framework's Magentic Fleet pattern

⚠️ Active Development Notice APIs, signatures, and execution semantics can change between minor versions. Pin a version tag for production usage.

---

AgenticFleet – DSPy‑Enhanced Multi‑Agent Orchestration

AgenticFleet is a hybrid DSPy + Microsoft agent-framework runtime that delivers a self‑optimizing fleet of specialized AI agents. DSPy handles task analysis, routing, progress & quality assessment; agent-framework provides robust orchestration primitives, event streaming, and tool execution. Together they enable delegated, sequential, parallel, and handoff‑driven workflows with iterative refinement loops.

---

Table of Contents

• AgenticFleet – DSPy‑Enhanced Multi‑Agent Orchestration
  • Table of Contents
  • Key Features
  • Architecture Overview
  • Directory Layout
  • Installation
    • Python (uv recommended)
    • Standard pip
    • Optional Frontend
    • Playwright (Browser Tool)
  • Configuration & Environment
  • Quick Start
    • TUI / CLI
    • Python API
    • Streaming
  • Execution Modes
  • Agents
  • DSPy Optimization
  • Observability & History
  • Evaluation & Self-Improvement
  • Testing & Quality
  • Troubleshooting
  • Contributing
  • License
  • Acknowledgments
  • Related Documentation

---

Key Features

• Adaptive Routing – DSPy supervisor analyzes tasks and decides agent roster + execution mode (delegated / sequential / parallel).
• Quality Loops – Automatic Judge / Reviewer refinement when quality score drops below configurable threshold.
• Tool‑Aware Decisions – Signatures include tool context; Supervisor recommends tool usage (code interpreter, search, browser, etc.).
• Streaming Events – Emits OpenAI Responses‑compatible events for real‑time TUI / web UI updates.
• Self‑Improvement – GEPA + BootstrapFewShot compilation refines routing from curated examples & execution history.
• YAML‑Driven – Central workflow_config.yaml governs models, thresholds, agents, tracing, evaluation toggles.
• Rich Ergonomics – Typer CLI (console.py), dspy-fleet command, optional Vite frontend, history analytics scripts.
• Safe Fallbacks – Graceful degradation when DSPy unavailable (heuristic routing & quality scoring).
• Extensible Toolkit – Add agents, tools, signatures, evaluation metrics with minimal boilerplate.

---

Architecture Overview

Four‑phase pipeline:

Task → [1] DSPy Analysis → [2] DSPy Routing → [3] Agent Execution → [4] Quality / Judge Assessment → (Optional Refinement)

Phase	Responsibility	Source
Analysis	Extract goals, complexity, constraints	dspy_modules/supervisor.py (analyze_task)
Routing	Pick agents + execution mode, tools	dspy_modules/supervisor.py (route_task)
Execution	Orchestrate agents & tools; stream events	workflows/supervisor_workflow.py
Quality	Score output, recommend improvements	dspy_modules/supervisor.py (assess_quality + Judge)

Refinement triggers when score < threshold (default 8 or judge threshold ≥ 7). Handoffs coordinate multi‑agent chains via HandoffManager.

Consult: docs/developers/architecture.md & docs/guides/quick-reference.md.

---

Directory Layout

Path	Purpose
config/workflow_config.yaml	Models, agents, thresholds, tracing, evaluation flags
src/agentic_fleet/dspy_modules/	DSPy Signatures & Supervisor implementation
src/agentic_fleet/workflows/	SupervisorWorkflow, handoff & exceptions
src/agentic_fleet/agents/	Specialist configurations & factory
src/agentic_fleet/tools/	Tool adapters: Tavily, Browser, Hosted Interpreter, MCP
src/agentic_fleet/utils/	Compiler cache, GEPA optimizer, history, tracing, registry
src/agentic_fleet/evaluation/	Metrics & evaluator engine
src/agentic_fleet/console.py	Rich / Typer CLI (dspy-fleet)
examples/	Minimal workflow samples
scripts/	Analysis, self-improvement, dataset generation
logs/	Execution history, compilation artifacts
frontend/	Optional Vite + React streaming UI

---

Installation

Python (uv recommended)

git clone https://github.com/Qredence/agentic-fleet.git
cd agentic-fleet
uv pip install -r requirements.txt
uv pip install -e .

Standard pip

pip install -r requirements.txt
pip install -e .

Optional Frontend

make frontend-install          # installs Node dependencies
make dev                       # runs backend + frontend dev servers

Playwright (Browser Tool)

playwright install chromium

---

Configuration & Environment

Create .env (or copy .env.example):

OPENAI_API_KEY=sk-...          # Required for all model calls
TAVILY_API_KEY=tvly-...        # Enables web search for Researcher agent
DSPY_COMPILE=true              # Toggle DSPy compilation (true/false)
OPENAI_BASE_URL=https://...    # Optional custom endpoint
LANGFUSE_PUBLIC_KEY=...        # Optional observability
LANGFUSE_SECRET_KEY=...

Key YAML knobs (workflow_config.yaml):

• dspy.model – Supervisor model (e.g. gpt-5-mini)
• dspy.optimization.metric_threshold – Minimum routing accuracy
• workflow.supervisor.max_rounds – Conversation turn limit
• workflow.supervisor.enable_streaming – Event streaming toggle
• agents.* – Per-agent model + temperature + tools
• evaluation.* – Batch evaluation settings

---

Quick Start

TUI / CLI

dspy-fleet                          # Launch interactive console

uv run python console.py run -m "Research 2024 AI funding trends" --verbose
uv run python console.py analyze --dataset data/evaluation_tasks.jsonl

Python API

import asyncio
from agentic_fleet.workflows import create_supervisor_workflow

async def main():
		workflow = await create_supervisor_workflow(compile_dspy=True)
		result = await workflow.run("Summarize transformer architecture evolution")
		print(result["result"])  # final output
		print(result["quality"]) # quality assessment details

asyncio.run(main())

Streaming

async for event in workflow.run_stream("Compare AWS vs Azure AI offerings"):
		# Handle MagenticAgentMessageEvent / WorkflowOutputEvent
		print(event)

---

Execution Modes

Mode	Description	Use Case
Delegated	Single agent manages entire task	Focused research, simple writeups
Sequential	Output of one feeds next	Research → Analyze → Write report
Parallel	Multiple agents concurrently; synthesis afterwards	Multi‑source comparisons
Handoff Chains	Explicit role transitions with artifacts	Complex coding + verification flows

Supervisor chooses based on task structure + examples; can be overridden via configuration or future explicit flags.

---

Agents

Core specialists: Researcher, Analyst, Writer, Reviewer, Judge (quality). Extended handoff specialists: Planner, Executor, Coder, Verifier, Generator.

See AGENTS.md for detailed roles, tool usage, configuration examples, and selection guidelines.

---

DSPy Optimization

Training examples live in src/agentic_fleet/data/supervisor_examples.json:

{
  "task": "Research the latest AI advances",
  "team": "Researcher: web search\nAnalyst: code + data",
  "assigned_to": "Researcher,Analyst",
  "mode": "sequential"
}

Compilation (BootstrapFewShot + GEPA) occurs on first run (if DSPY_COMPILE=true). Cache stored under logs/compiled_supervisor.pkl. Refresh via:

uv run python manage_cache.py --clear

---

Observability & History

• History: Structured events appended to logs/execution_history.jsonl.
• Tracing: Enable OpenTelemetry in YAML; export to AI Toolkit / OTLP endpoint.
• Logging: Adjustable log level via env (AGENTIC_FLEET_LOG_LEVEL=DEBUG).
• Analysis: scripts/analyze_history.py --all surfaces aggregate metrics.

---

Evaluation & Self-Improvement

Run batch evaluations against curated tasks:

uv run python console.py analyze --dataset data/evaluation_tasks.jsonl

Generate evaluation datasets from history:

uv run python scripts/create_history_evaluation.py

Self‑improve routing by folding high‑quality history examples back into DSPy training:

uv run python scripts/self_improve.py --max 50

---

Testing & Quality

make check                 # lint (Ruff), format (Black), type‑check (mypy)
make test                  # run pytest suite
PYTHONPATH=. uv run pytest tests/workflows/test_supervisor_workflow.py::test_supervisor_workflow -q

Key test domains: routing accuracy, tool registry integration, judge refinement, lazy compilation, tracing hooks.

---

Troubleshooting

Symptom	Cause	Fix
Missing web citations	TAVILY_API_KEY unset	Export key or set in .env
Slow first run	DSPy compilation	Enable cache; reduce max_bootstrapped_demos
No streaming output	enable_streaming=false	Toggle in YAML
Low quality score	Insufficient examples	Add training examples; rerun compilation
Tool warning	Name mismatch	Verify tool name & registry entry

Detailed guides: docs/users/troubleshooting.md, docs/guides/dspy-optimizer.md.

---

Contributing

1. Fork / branch (breaking-refactor for large changes)
2. Add or update tests (prefer focused unit tests over broad integration when possible)
3. Run make check and ensure no style / type errors
4. Update docs (README, AGENTS.md, or relevant guide) for user‑visible changes
5. Submit PR with clear rationale & architectural notes (link to docs/developers/architecture.md sections if modifying internals)

Please see: docs/developers/contributing.md.

---

License

MIT License – see LICENSE.

---

Acknowledgments

• Microsoft agent-framework – Orchestration, events & tool interfaces
• DSPy (Stanford NLP) – Prompt optimization & structured signatures
• Tavily – Reliable, citation‑rich web search
• OpenAI Responses – Event paradigm enabling unified CLI/TUI/frontend streaming

---

Related Documentation

• Getting Started: docs/users/getting-started.md
• Configuration: docs/users/configuration.md
• Architecture Deep Dive: docs/developers/architecture.md
• Quick Reference: docs/guides/quick-reference.md
• DSPy Optimization: docs/guides/dspy-optimizer.md
• Evaluation: docs/guides/evaluation.md
• Tracing: docs/guides/tracing.md
• Self Improvement: docs/users/self-improvement.md
• Troubleshooting: docs/users/troubleshooting.md

---

Happy hacking! 🚀