cortex-engine 0.1.0

AI Operating System for Coding — routes queries across 30-50+ LLMs with GPU scheduling, LRU caching, evaluation, and feedback-driven routing.

🧠 Cortex-Engine

An AI Operating System for coding — routes queries across 30–50+ models, manages GPU scheduling, caching, evaluation, and continuous feedback.

models = processes   |   GPU = CPU   |   router = scheduler   |   kernel = control plane

---

Architecture

User Query
    │
    ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Cortex-Engine API  (FastAPI)                  │
│                                                                   │
│  POST /inference                                                  │
│       │                                                           │
│       ▼                                                           │
│  ┌─────────────────────────────────────────────────────────┐     │
│  │              Model Orchestrator (central brain)          │     │
│  │                                                          │     │
│  │  ① Cache Check ──────────► Redis LRU Cache              │     │
│  │       │ miss                                             │     │
│  │       ▼                                                  │     │
│  │  ② Router Engine ─────────► Cluster Detection           │     │
│  │       │                     Model Selection              │     │
│  │       ▼                     Confidence Score            │     │
│  │  ③ Scheduler ──────────────► GPU Assignment             │     │
│  │       │                     Priority Queue              │     │
│  │       ▼                                                  │     │
│  │  ④ Model Worker ──────────► vLLM / Triton (Phase 2)     │     │
│  │       │                                                  │     │
│  │       ▼                                                  │     │
│  │  ⑤ Evaluator ─────────────► Static Analysis             │     │
│  │       │                     LLM Grading (Phase 2)       │     │
│  │       ▼                                                  │     │
│  │  ⑥ Feedback Log ──────────► Redis (rolling 10k)         │     │
│  └─────────────────────────────────────────────────────────┘     │
└─────────────────────────────────────────────────────────────────┘
         │
         ▼
    Model Registry (Redis Hash)
    7 seed models → 50+ in Phase 3

---

Latency Targets

Component	Target
Router	< 50 ms
Cache check	< 5 ms
Scheduling	< 10 ms
Total overhead	< 100 ms

---

Project Structure

cortex_engine/
├── main.py                  ← FastAPI app, lifespan, middleware, health/metrics
├── config.py                ← Pydantic Settings (env-based)
├── dependencies.py          ← All Depends() providers
│
├── models/
│   └── schemas.py           ← All Pydantic request/response models + enums
│
├── services/
│   ├── registry.py          ← Redis-backed model registry (7 seed models)
│   ├── router.py            ← Keyword/heuristic router with tiebreak scoring
│   ├── cache_manager.py     ← LRU response cache + warm pool + eviction log
│   ├── scheduler.py         ← Priority queue scheduler (Redis sorted sets)
│   ├── evaluator.py         ← Static analysis + LLM-grade stub
│   ├── feedback.py          ← Rolling feedback log + accuracy stats
│   └── orchestrator.py      ← Central brain: routes→schedules→infers→evals
│
├── routers/
│   ├── inference.py         ← POST /inference, GET /inference/route-preview
│   └── api.py               ← /registry CRUD, /admin (queue/cache/feedback)
│
├── tests/
│   ├── conftest.py          ← Async fixtures with fakeredis (no real Redis needed)
│   ├── test_registry.py     ← 11 tests
│   ├── test_router.py       ← 11 tests
│   ├── test_cache.py        ← 7 tests
│   ├── test_evaluator.py    ← 6 tests
│   ├── test_scheduler.py    ← 7 tests
│   ├── test_orchestrator.py ← 11 tests (integration)
│   └── test_feedback.py     ← 4 tests  (57 total → all pass)
│
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── pytest.ini
└── .env.example

---

Quickstart

Option A — Local (needs Redis)

# 1. Clone & install
git clone <repo>
cd cortex_engine
pip install -r requirements.txt

# 2. Start Redis
docker run -d -p 6379:6379 redis:7.4-alpine

# 3. Configure
cp .env.example .env

# 4. Run
uvicorn main:app --reload --port 8000

Option B — Docker Compose (recommended)

docker compose up --build
# Optional: include Redis Commander UI
docker compose --profile dev up --build

Run Tests (no Redis needed)

pip install -r requirements.txt
pytest tests/ -v

---

API Reference

POST /inference

Route a query to the best model and get a response.

// Request
{
  "query": "Write a pytest test for a function that reverses a string",
  "preferred_type": null,       // optional: debugging|testing|explanation|...
  "preferred_model": null,      // optional: override routing entirely
  "max_tokens": 2048,
  "temperature": 0.2,
  "evaluate": true
}

// Response
{
  "request_id": "a1b2c3d4e5f6",
  "output": "...",
  "model_used": "starcoder2-7b",
  "route": {
    "cluster": "testing",
    "selected_model": "starcoder2-7b",
    "confidence": 0.72,
    "fallback_models": ["phi-3-mini"],
    "routing_latency_ms": 1.4
  },
  "evaluation": {
    "success": true,
    "score": 0.8,
    "method": "static_analysis",
    "details": "has_content=✓; has_code=✓; no_apology=✓; syntax_ok=✓; keyword_coverage=✓"
  },
  "total_latency_ms": 14.2,
  "cached": false,
  "tokens_used": 87
}

GET /inference/route-preview?query=...

Dry-run: see which model would be selected without running inference.

GET /registry/

List all registered models with status and metadata.

POST /registry/

Register a new model.

PATCH /registry/{model_name}/status

Set model status: available | loading | busy | offline

GET /admin/queue

Current GPU queue depth, running jobs, per-GPU load counts.

GET /admin/cache

Cache hit rate, warm pool, recent evictions.

GET /admin/feedback

Routing accuracy stats and recent feedback log.

GET /health

Liveness check — Redis connectivity + model counts.

GET /metrics

Full system metrics: routing accuracy, cache hit rate, GPU loads, queue depth.

---

Cluster → Model Routing

Cluster	Trigger Keywords	Models
debugging	error, exception, traceback, crash, fix bug	codellama-7b
testing	pytest, jest, unit test, mock, assert	starcoder2-7b
explanation	explain, what is, how does, document	mistral-7b-instruct
refactoring	refactor, optimize, clean up, simplify	qwen-coder-14b
python	python, django, fastapi, flask, .py	qwen-coder-7b, qwen-coder-14b
general_code	code, class, implement, build	deepseek-coder-6.7b
fallback	everything else	phi-3-mini

---

Seed Models (Phase 1 MVP)

Model	Cluster	Size	Latency	GPU
qwen-coder-7b	python	7B	250ms	gpu-0
qwen-coder-14b	python	14B	400ms	gpu-1
deepseek-coder-6.7b	general_code	6.7B	220ms	gpu-0
codellama-7b	debugging	7B	270ms	gpu-1
mistral-7b-instruct	explanation	7B	300ms	gpu-2
starcoder2-7b	testing	7B	260ms	gpu-2
phi-3-mini	fallback	3.8B	150ms	gpu-3

---

Roadmap

Phase 1 (now — MVP)

• FastAPI kernel with all core services
• Redis-backed model registry (7 models)
• Keyword/heuristic router with cluster detection
• LRU response cache + warm pool + eviction log
• Priority queue GPU scheduler (Redis sorted sets)
• Static analysis evaluator
• Rolling feedback system + accuracy tracking
• 59 tests (all passing, no real Redis needed)
• Docker Compose stack

Phase 2

• Swap inference stub → real vLLM HTTP calls
• Embedding-based routing (BGE/E5 + FAISS/Qdrant)
• 15 models across more language clusters
• Ray Serve for distributed model workers
• LLM judge model for evaluation (phi-3-mini)

Phase 3

• 30–50+ models
• Self-improving router (daily retraining from feedback)
• Cost-aware routing (balance quality vs. $/token)
• Multi-model collaboration (chain models)
• PostgreSQL for persistent metadata
• Reinforcement learning router

---

Environment Variables

Variable	Default	Description
REDIS_URL	redis://localhost:6379/0	Redis connection URL
REDIS_MAX_CONNECTIONS	50	Connection pool size
PORT	8000	API server port
WORKERS	1	Uvicorn worker count
CACHE_TTL_SECONDS	3600	Response cache TTL
ENABLE_EVALUATION	true	Run evaluator on outputs
ENABLE_FEEDBACK	true	Log routing feedback

---

Tech Stack

Layer	Tech
API	FastAPI + Uvicorn
Cache / State	Redis 7 (asyncio)
Scheduling	Redis sorted sets
Config	Pydantic Settings
Testing	pytest-asyncio + fakeredis
Containers	Docker + Docker Compose
Phase 2 Serving	vLLM + Triton
Phase 2 Orchestration	Ray Serve + Kubernetes
Phase 2 Vector DB	FAISS / Qdrant
Phase 3 Metadata	PostgreSQL + SQLAlchemy