planwise 2.0.0

PlanWise Router — intelligent tool routing layer for large tool catalogs

Token Titans — PlanWise Router

Submission for the Crest Data AI Innovation Challenge (Problem 02 — Intelligent Tool Router for Large Tool Catalogs).

Team Information

• Team Name: Token Titans
• Team Lead: Rikin Solanki
• Members: Rikin Solanki, Devika Pedada, Karan Gondaliya, Yaksh Rajvanshi

Project Overview

An AI agent handed 60+ tool schemas at once wastes tokens on descriptions it'll never use, and picks the wrong tool when several look alike (same inputs, different job; v1/v2 pairs). PlanWise is a routing layer that sits between the user's query and the catalog: it reads the query, writes a step-by-step plan, and hands the agent only the 3–6 tools it actually needs, in the right order, plus a plan hint.

Offline, every tool is quality-gated (including a self-retrieval test — a tool must rank #1 for its own example questions or it's rejected) and indexed (embeddings + BM25 + twin/version graphs). At runtime, one LLM call turns the query into abstract steps that never reference tool names — so the planner can't hallucinate a tool — and each step is scored on four signals, the standout being output fit (what a tool returns vs. what the step needs), which is what tells apart "twin" tools with identical inputs. Two additional, optional LLM-judgement passes — before binding (does the plan itself make sense?) and after (does the bound tool actually satisfy the step?) — catch what pure scoring can't, and both fail open to the deterministic path if no LLM gateway is configured. Everything ships as four pluggable surfaces: an HTTP gateway, a Python middleware wrapper, an MCP server, and a judge-facing demo web console.

Measured: 79.8% exact tool-sequence match (104 queries, 16-tool catalog) and 64.67% on a harder 150-query/70-tool run, with 88–97% token savings vs. shipping the full catalog. Full methodology and reproduction steps: docs/LLD.md §16.

Demo Video

TBD — link to follow before submission (Crest Google Drive, Crest-wide read access enabled).

Live URL

TBD — not yet deployed. Deployment is a two-piece split (FastAPI backend on Render, React/Vite console on Vercel) with a full walkthrough in DEPLOY.md.

Technical Design Document / Architecture Decision Record

• docs/TDD.md — the formal design doc: problem statement, goals/non-goals, requirements traceability, architecture rationale, NFRs, testing strategy, risk register.
• docs/HLD.md — high-level design: rules, system overview, how it scores against the judging criteria.
• docs/LLD.md — low-level design: exact algorithms, data models, module-by-module detail, current eval numbers.
• docs/architecture-diagram.html — visual architecture diagram.
• docs/ONE_PAGER.md — a one-page summary of the whole project.

Tech Stack

• Core engine: Python 3.11+, Pydantic, NumPy (cosine similarity, no vector DB needed at this scale), rank-bm25, jsonschema
• Semantic embeddings: sentence-transformers (bge-small-en-v1.5, local CPU; deterministic hashing fallback for offline/CI)
• LLM planner + verification: httpx against any OpenAI-compatible chat gateway — Crest's AWS Bedrock proxy by default, Google Gemini as an automatic fallback provider
• Tier 1 — HTTP gateway: FastAPI, Uvicorn
• Tier 3 — MCP server: mcp SDK
• Demo console: React 18, Vite 5, Tailwind CSS v4, Framer Motion
• Eval/trace tooling: Streamlit (trace viewer)
• Deployment: Docker (backend, Render), Vercel (frontend)

Setup & Run Instructions

Full guide: docs/USER_GUIDE.md. Quick version:

pip install -e ".[dev]"                # everything, from a checkout

# Route one query against the bundled sample catalog — works fully offline, no key needed
planwise route "Pull last quarter's revenue and email the chart to [e-mail]"

# Run the evaluation harness (with the naive top-5-similarity baseline)
planwise eval --baseline

# Run the tests
python -m pytest tests/

For the LLM-enhanced path (LLM planner + both verifier passes + real semantic embeddings):

pip install -e ".[llm,embeddings]"
export PLANWISE_LLM_API_KEY=sk-...                     # your gateway key
export PLANWISE_LLM_CERT=/path/to/apiproxy.cdsys.crt   # only if pointed at an internal-CA gateway
planwise route "..."

Start the HTTP gateway (Tier 1):

pip install -e ".[server,embeddings]"
planwise serve --host 0.0.0.0 --port 8000
curl -s localhost:8000/healthz

Run the demo web console locally:

planwise serve --port 8000          # terminal 1 — the backend
cd web && npm install && npm run dev   # terminal 2 → http://localhost:5173

Docker:

docker build -t planwise .
docker run -p 8000:8000 -e PLANWISE_LLM_API_KEY=... planwise

Additional Links / Resources

• docs/USER_GUIDE.md — complete guide across all four surfaces (CLI, HTTP gateway, Python middleware, MCP server, demo console), configuration reference, running evals, troubleshooting
• docs/http-gateway.md / docs/http-gateway-user-guide.md — the HTTP gateway's full REST reference, sequence diagrams, and design rationale
• DEPLOY.md — Render + Vercel deployment walkthrough and judging-day checklist
• results/ — saved evaluation run reports (results_v4.md, results_v9.md)
• data/ — ground-truth query sets and eval catalogs used for the numbers above
• Repository: gitlab.com/mygroup4322601/planwise

---

Appendix: repository layout

• planwise/engine/ — the routing engine (pure library: intake, index, planner, scorer, binder, assembler, two LLM verifier passes)
• planwise/middleware/ — self-healing execution wrapper (Tier 2)
• planwise/server/ — HTTP gateway (Tier 1); planwise/mcp_server.py — MCP surface (Tier 3)
• web/ — judge-facing demo console (React + Vite)
• eval/ — evaluation harness, stress suites, baselines; scripts/run_validation.py — the day-to-day accuracy report
• planwise/data/catalog.json — sample catalog bundled with the package (the default when no --catalog is given)
• data/ — ground-truth queries + eval catalogs
• tests/ — 10 files, ~72 tests, deterministic and fully offline (hashing embeddings, rules planner — no downloads, no API keys)