shadow-engine 0.1.0

A self-improving background agent with persistent codebase knowledge graph and parallel experimentation

🧠 Shadow Engineer

A self-improving background agent with persistent codebase knowledge graph and parallel experimentation.

---

No existing background agent framework does this. Every agent today (Ramp's Inspect, Open-Inspect, Copilot, Claude Code) treats every session as a blank slate. Session 100 is no smarter than Session 1. Shadow Engineer remembers — and gets smarter with every session.

---

Table of Contents

1. What Is Shadow Engineer?
2. The Problem It Solves
3. Architecture
4. Installation
5. Quick Start (CLI)
6. REST API Integration
7. Python SDK
8. Open-Inspect Integration
9. Deployment
10. Configuration Reference
11. Supported Languages
12. Project Structure
13. FAQ
14. License

---

What Is Shadow Engineer?

Shadow Engineer is a learning layer that sits on top of any background coding agent (Open-Inspect, Claude Code, custom agents). It provides three capabilities that no other framework offers:

Engine	What It Does	Unique Value
Knowledge Graph	Indexes your codebase into a persistent, searchable semantic graph	Agents start with informed context — no more fumbling through code
Laboratory	Spawns N parallel agent sessions with different strategies and picks the winner	Not one attempt — choose from proven solutions
Learning Engine	Analyzes every session to extract patterns, track efficacy, and suggest approaches	Session 100 is smarter than Session 1 — compounding intelligence

The Compounding Moat

Session 1:   No context  | No patterns  | No approach data  | 40% success rate (guess)
Session 10:  10 symbols  | 3 patterns   | 2 approaches      | 55% success rate
Session 50:  50 symbols  | 8 patterns   | 5 approaches      | 70% success rate
Session 200: 100+ symbols | 20+ patterns | 10+ approaches   | 80%+ success rate

Every session makes the next one smarter. This is the defensible moat that no competitor ships.

---

The Problem It Solves

Current background coding agents treat every session independently:

Problem	Without Shadow Engineer	With Shadow Engineer
No memory	Agent fumbles through codebase every time	Agent starts with relevant context from the knowledge graph
No learning	Same mistakes repeated across sessions	Pattern extraction + efficacy tracking prevents repeat failures
Single attempt	One approach, one model — if it fails, start over	N parallel experiments, winner picked automatically
No codebase understanding	"What file handles authentication?" every session	Semantic search: "authentication" → auth/service.py

---

Architecture

                    ┌──────────────────────────────────────────┐
                    │           SHADOW ENGINEER                 │
                    │                                           │
 User sends task    │  ┌──────────────┐  ┌──────────────────┐  │
        │           │  │  Knowledge   │  │    Laboratory    │  │
        ▼           │  │    Graph     │  │                  │  │
 ┌──────────┐       │  │              │  │  3 variants:     │  │
 │ Classify │───────┼──│ • 211 symbols│  │  • Targeted Fix  │  │
 │ problem  │       │  │ • 26 files   │  │  • Root Cause    │  │
 └────┬─────┘       │  │ • 3 patterns │  │  • Defense Depth │  │
      │             │  └──────┬───────┘  └────────┬─────────┘  │
 ┌────▼─────┐       │         │                   │            │
 │  Build   │       │         │                   ▼            │
 │ context  │───────┼─────────┼──▶  Agent Prompt + Approach    │
 └────┬─────┘       │         │                                │
      │             │         │     ┌──────────────────┐       │
      │             │         │     │     Learning     │       │
      ▼             │         │     │      Engine      │       │
 ┌──────────┐       │         │     │                  │       │
 │  Agent   │       │         │     │ • Pattern extract│       │
 │ executes │       │         └─────│ • Efficacy track │       │
 │  task    │       │               │ • Failure analyze│       │
 └────┬─────┘       │               └────────┬─────────┘       │
      │             │                        │                 │
      ▼             │                        ▼                 │
 ┌──────────┐       │           ┌──────────────────────┐      │
 │  Ingest  │───────┼──────────▶│  Next session is     │      │
 │  result  │       │           │  SMARTER than before  │      │
 └──────────┘       │           └──────────────────────┘      │
                    └──────────────────────────────────────────┘

Three Engines

1. Knowledge Graph — "Remember"

• 7 languages indexed (Python, TypeScript, JavaScript, Go, Rust)
• Semantic search via ChromaDB vector embeddings
• Dependency mapping — "If I change UserService.authenticate(), what breaks?"
• Impact analysis — BFS up the dependency chain
• Context injection — relevant symbols, conventions, and approaches injected into agent prompts

2. Laboratory — "Experiment"

• 12 strategy templates — problem-type-aware (bug fix → "Targeted Fix" + "Root Cause + Guard" + "Defense in Depth")
• Configurable scoring — logistic curve normalization, no arbitrary cliffs
• 4 winner modes — best performing, smallest change, fastest execution, first to pass
• Concurrent execution — semaphore-limited parallel spawning

3. Learning Engine — "Improve"

• Pattern extraction — infers testing conventions, change scope, code review quality
• Efficacy tracking — running averages for success rate, duration, tokens
• Failure analysis — understands why approaches fail
• Approach suggestion — recommends historically-best approach and model
• Confidence scores — every classification returns (type, 0.0–1.0) not just a label

---

Installation

Prerequisites

• Python 3.12 or later
• (Optional) Redis for production rate limiting
• (Optional) Ollama for local LLM testing

Option 1: pip (recommended)

pip install shadow-engine

# With optional Redis support
pip install "shadow-engine[redis]"

Option 2: From source

git clone https://github.com/shadow-engine/shadow-engine.git
cd shadow-engine
pip install -e ".[dev]"

Option 3: Docker

docker compose -f docker/docker-compose.yml up -d

This starts both Shadow Engineer and a Redis instance for rate limiting.

---

Quick Start (CLI)

1. Index Your Codebase

cd /path/to/your/project
shadow-engine bootstrap

Output:

Bootstrapped: 347 symbols, 52 files indexed

2. Search for Symbols

shadow-engine search "authenticate"

Output:

[function] authenticate_user — src/auth/service.py
  Authenticates a user with email and password. Returns JWT token.
[method] authenticate — src/auth/middleware.py
  Middleware that checks the Authorization header for valid JWT.

3. Get AI-Ready Context

shadow-engine context "fix the login rate-limiting bug"

Output (injects into agent prompts):

## Codebase Knowledge Graph Context (Semantic)

### Semantically Relevant Symbols

- **authenticate_user** (`function`) in `src/auth/service.py` (relevance: 0.89)
  Authenticate a user from a JWT token.
  Depends on: UserModel, TokenService

- **login_handler** (`function`) in `src/auth/views.py` (relevance: 0.82)
  Handle login POST requests. Validates credentials and returns session token.
  Complexity: 6.5

### Learned Codebase Conventions
- **error_handling**: Auth errors return 401 with JSON body {error: string, code: string}
- **testing**: Agent writes tests alongside code changes.

### Historically Effective Approaches
- **Targeted Fix**: 85% success rate (17/20) — best model: claude-sonnet-4-6

4. Analyze Change Impact

shadow-engine impact "UserService"

Output:

{
  "symbol": {"name": "UserService", "kind": "class", "file_path": "src/services/user.py"},
  "dependencies": ["Database", "CacheClient", "EmailService"],
  "direct_dependents": ["AuthController", "ProfileController", "AdminController"],
  "total_affected_symbols": 18
}

5. Get Approach Suggestion

shadow-engine suggest "add a search feature for products"

Output:

{
  "problem_type": "feature",
  "classification_confidence": 0.85,
  "recommended_approach": "Extensible Design",
  "expected_success_rate": 0.78,
  "best_model": "claude-sonnet-4-6"
}

6. Create Parallel Experiments

shadow-engine experiment "refactor the billing module" --variants 3

7. Record Session Results

shadow-engine record \
  --session-id "session-abc123" \
  --outcome "success" \
  --prompt "fix the login rate-limiting bug" \
  --approach "Targeted Fix" \
  --model "claude-sonnet-4-6" \
  --files "src/auth/service.py" "tests/auth/test_service.py" \
  --tests-passed 12 --tests-failed 0 \
  --duration 45.2 --tokens 8500

8. View Improvement Report

shadow-engine report

---

REST API Integration

Start the server:

# Development
uvicorn shadow_engine.api_server.server:app --reload

# Production
shadow-engine-server
# or
docker compose -f docker/docker-compose.yml up -d

Authentication (optional)

export SHADOW_ENGINE_API_KEY="your-secret-key"
curl -H "X-API-Key: your-secret-key" http://localhost:8000/health

Core Workflow

# 1. Index the codebase
curl -X POST http://localhost:8000/bootstrap

# 2. Get context for an agent prompt
curl "http://localhost:8000/context?task=fix+the+login+rate+limiting+bug"

# 3. Get approach suggestion
curl "http://localhost:8000/suggest?task=fix+the+login+rate+limiting+bug"

# 4. Create an experiment batch
curl -X POST "http://localhost:8000/experiment?task=refactor+auth&variants=3"

# 5. Record session result
curl -X POST http://localhost:8000/sessions/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "sess-001",
    "outcome": "success",
    "prompt": "fix the login bug",
    "approach": "Targeted Fix",
    "model": "claude-sonnet-4-6",
    "files_changed": ["src/auth.py", "tests/test_auth.py"],
    "tests_passed": 10,
    "tests_failed": 0,
    "duration_seconds": 30.0,
    "token_count": 5000
  }'

# 6. View improvement report
curl http://localhost:8000/report

# 7. Check operational metrics
curl http://localhost:8000/metrics

Full API Reference

All 11 endpoints are documented in API_DOCS.md with request/response schemas, field descriptions, and curl examples. Interactive Swagger UI at http://localhost:8000/docs.

---

Python SDK

import httpx

class ShadowEngineClient:
    """Minimal Python client for Shadow Engine REST API."""
    
    def __init__(self, base_url: str = "http://localhost:8000", api_key: str | None = None):
        self.base_url = base_url.rstrip("/")
        self.headers = {"X-API-Key": api_key} if api_key else {}

    def bootstrap(self, repo: str = ".") -> dict:
        r = httpx.post(f"{self.base_url}/bootstrap", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def get_context(self, task: str, repo: str = ".") -> str:
        r = httpx.get(f"{self.base_url}/context", params={"task": task, "repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()["context"]

    def search(self, query: str, kind: str | None = None, repo: str = ".") -> dict:
        params = {"query": query, "repo": repo}
        if kind: params["kind"] = kind
        r = httpx.get(f"{self.base_url}/search", params=params, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def suggest(self, task: str, repo: str = ".") -> dict:
        r = httpx.get(f"{self.base_url}/suggest", params={"task": task, "repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def create_experiment(self, task: str, variants: int = 3, repo: str = ".") -> dict:
        r = httpx.post(f"{self.base_url}/experiment", params={"task": task, "variants": variants, "repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def ingest_session(self, **kwargs) -> dict:
        r = httpx.post(f"{self.base_url}/sessions/ingest", json=kwargs, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def get_report(self, repo: str = ".") -> str:
        r = httpx.get(f"{self.base_url}/report", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.text

    def get_stats(self, repo: str = ".") -> dict:
        r = httpx.get(f"{self.base_url}/stats", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def get_metrics(self, repo: str = ".") -> dict:
        r = httpx.get(f"{self.base_url}/metrics", params={"repo": repo}, headers=self.headers)
        r.raise_for_status()
        return r.json()

    def health(self) -> dict:
        r = httpx.get(f"{self.base_url}/health")
        r.raise_for_status()
        return r.json()

Usage Example

# Initialize
client = ShadowEngineClient(base_url="http://localhost:8000", api_key="your-secret-key")

# Bootstrap the codebase
result = client.bootstrap()
print(f"Indexed {result['symbols_indexed']} symbols")

# Get context for an agent
context = client.get_context("fix the login rate-limiting bug")

# Get approach suggestion
suggestion = client.suggest("fix the login rate-limiting bug")
print(f"Recommended: {suggestion['recommended_approach']} ({suggestion['expected_success_rate']:.0%} expected)")

# Record a completed session
client.ingest_session(
    session_id="sess-001",
    outcome="success",
    prompt="fix the login bug",
    approach="Targeted Fix",
    model="claude-sonnet-4-6",
    files_changed=["src/auth.py", "tests/test_auth.py"],
    tests_passed=10,
    tests_failed=0,
    duration_seconds=30.0,
    token_count=5000,
)

# After several sessions, view the improvement
print(client.get_report())
print(f"Overall success rate: {client.get_stats()['overall_success_rate']:.0%}")

---

Open-Inspect Integration

Shadow Engineer provides an async bridge that plugs directly into Open-Inspect's session lifecycle:

from shadow_engine.integrations.openinspect import OpenInspectBridge

# Initialize
bridge = OpenInspectBridge(repo_path="/path/to/your/repo")
bridge.bootstrap_if_needed()

# Before session spawn — enrich the prompt with knowledge graph context
config = {
    "prompt": "fix the login rate-limiting bug",
    "repository": "my-repo",
}
enriched = await bridge.enrich_session_config(config)
# enriched["prompt"] now contains knowledge graph context
# enriched["suggested_approach"] = "Targeted Fix"
# enriched["suggested_model"] = "claude-sonnet-4-6"
# enriched["problem_type"] = "bug_fix"
# enriched["classification_confidence"] = 0.95

# After session completes — ingest the result for learning
result = {
    "session_id": "sess-abc123",
    "outcome": "completed",
    "prompt": "fix the login rate-limiting bug",
    "approach": "Targeted Fix",
    "model": "claude-sonnet-4-6",
    "pr_url": "https://github.com/myorg/myrepo/pull/142",
    "files_changed": ["src/auth.py", "tests/test_auth.py"],
    "tests_passed": 12,
    "tests_failed": 0,
    "duration_seconds": 45.2,
    "token_count": 8500,
}
ingestion = await bridge.ingest_session_result(result)

---

Deployment

Docker (Production)

# With Redis for rate limiting
docker compose -f docker/docker-compose.yml up -d

# Without Redis (in-memory rate limiting)
docker build -t shadow-engine -f docker/Dockerfile .
docker run -p 8000:8000 \
  -e SHADOW_ENGINE_API_KEY=your-secret \
  -v /data/shadow-engine:/home/shadow/data \
  shadow-engine

Environment Variables

Variable	Default	Description
SHADOW_ENGINE_API_KEY	"" (disabled)	API key for authentication. Set to enable
SHADOW_ENGINE_REDIS_URL	redis://redis:6379	Redis URL for rate limiting
SHADOW_ENGINE_RATE_LIMIT	100	Max requests per window
SHADOW_ENGINE_RATE_WINDOW	60	Rate limit window in seconds
SHADOW_ENGINE_STORAGE_PATH	/home/shadow/data	Persistent storage directory
SHADOW_ENGINE_PORT	8000	API server port
SHADOW_ENGINE_HOST	0.0.0.0	API server host

Production Checklist

See FINDINGS_REPORT.md §6.2 for a detailed 12-point production readiness checklist.

---

Configuration Reference

Scoring Configuration

Customize how experiment variants are scored:

from shadow_engine.laboratory.experiment import ScoringConfig

config = ScoringConfig(
    test_pass_weight=0.50,      # Prioritize test passing
    change_size_weight=0.15,    # Less emphasis on change size
    speed_weight=0.20,          # More emphasis on speed
    token_efficiency_weight=0.05,
    file_scope_weight=0.10,
)

Model Configuration

from shadow_engine.main import ShadowEngine

engine = ShadowEngine(
    storage_path="./.shadow-engine",
    repo_path="./my-project",
    use_sqlite=True,    # Production: SQLite WAL mode (default)
    use_chroma=True,    # ChromaDB semantic search (default)
)

---

Supported Languages

Language	Extensions	Symbols Extracted
Python	.py	Functions, methods, classes
TypeScript	.ts, .tsx	Functions, classes, interfaces, enums, type aliases
JavaScript	.js, .jsx	Functions, classes, constants
Go	.go	Functions, methods, structs, interfaces
Rust	.rs	Functions, structs, traits, enums, type aliases

---

Project Structure

shadow-engine/
├── README.md                        # You are here
├── CHANGELOG.md                     # Version history
├── API_DOCS.md                      # Full REST API reference (1,070 lines)
├── FINDINGS_REPORT.md               # Research, efficiency, competitive analysis
├── pyproject.toml                   # Package dependencies + entry points
├── docker/
│   ├── Dockerfile                   # Production container
│   └── docker-compose.yml           # Redis + app stack
├── .github/workflows/ci.yml         # CI/CD pipeline
├── scripts/real_demo.py             # End-to-end real LLM demo
├── src/shadow_engine/
│   ├── main.py                      # Orchestrator + CLI (11 commands)
│   ├── knowledge_graph/
│   │   ├── models.py                # 10 Pydantic data models
│   │   ├── indexer.py               # 7-language AST parser
│   │   └── store.py                 # JSON backend (legacy)
│   ├── sqlite_store/db.py           # SQLite WAL backend (production default)
│   ├── chroma_store/vector_store.py # ChromaDB semantic search
│   ├── laboratory/experiment.py     # Experiment runner + scoring
│   ├── learning/engine.py           # Pattern extraction + efficacy tracking
│   ├── async_lab/executor.py        # Concurrent experiment execution
│   ├── api_server/server.py         # FastAPI REST server
│   ├── integrations/openinspect.py  # Open-Inspect async bridge
│   └── redis_limiter/               # Redis rate limiter
└── tests/
    ├── test_knowledge_graph.py       # 33 tests
    ├── test_laboratory.py            # 19 tests
    ├── test_learning.py              # 12 tests
    └── test_integration.py           # 16 tests

---

FAQ

Q: How is this different from Open-Inspect? A: Open-Inspect is a background agent framework — it spawns sandboxes and runs coding sessions. Shadow Engineer is a learning layer that sits on top. It adds cross-session memory, parallel experimentation, and compounding intelligence that Open-Inspect (and every other agent framework) lacks.

Q: Can I use this without Open-Inspect? A: Yes. Shadow Engineer works with any background agent via its REST API or CLI. The Open-Inspect bridge is an optional integration.

Q: Does it require a GPU? A: No. ChromaDB uses CPU embeddings by default. GPU is only needed if you want to use a custom embedding model.

Q: What scale does this support? A: SQLite WAL mode supports 100K+ sessions. For larger scale, the findings report recommends PostgreSQL as a future backend.

Q: Is this ready for production? A: See the FINDINGS_REPORT.md for a detailed assessment. Verdict: A- — production-grade for internal teams, beta-quality for public release.

---

License

MIT — Build on it. Ship it. Make agents smarter.

---

Inspired by Ramp's Inspect and Open-Inspect.