forge-cloud

OpenAI-compatible reasoning-aware inference proxy for Qwen3.6.

Point your OpenAI client at forge-cloud instead of directly at vLLM/SGLang/Ollama. The proxy routes thinking mode based on query complexity, swaps sampling parameters to match the mode, normalizes backend flags, and tags responses with routing metadata.

What it does

1. Receives a standard /v1/chat/completions request
2. Classifies query complexity (simple/moderate/complex)
3. Decides thinking mode (think vs no_think) with correct sampling params
4. Normalizes the enable_thinking flag for the target backend (vLLM nested, DashScope top-level, llama.cpp server-side)
5. Forwards to the user's configured backend
6. Tags the response with routing metadata and estimated token split (thinking vs response)

The proxy does not run inference. It configures and monitors it.

Install

pip install forge-cloud

Quick start

# Set admin key and backend URL
export FORGE_ADMIN_KEY=my-secret
export FORGE_BACKEND_URL=http://localhost:8000
export FORGE_BACKEND_TYPE=vllm

# Start the proxy
forge-cloud

Create an API key:

curl -X POST http://localhost:8741/v1/keys \
  -H "Authorization: Bearer my-secret" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-app"}'
# Returns: {"key": "fk-...", "name": "my-app", "tier": "free", ...}

Use it like any OpenAI endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8741/v1",
    api_key="fk-..."  # key from above
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B",
    messages=[{"role": "user", "content": "refactor this module"}],
)
print(response.choices[0].message.content)

Response metadata

Every response includes a forge field with routing metadata and estimated token counts:

{
  "id": "chatcmpl-test",
  "choices": ["..."],
  "usage": {"...": 0},
  "forge": {
    "thinking_mode": "think",
    "complexity": "complex",
    "backend": "vllm",
    "sampling_profile": "thinking",
    "thinking_tokens": 450,
    "response_tokens": 120
  }
}

Endpoints

Configuration

All settings are environment variables with FORGE_ prefix:

Per-key backend override

Each API key can have its own backend URL and type:

curl -X POST http://localhost:8741/v1/keys \
  -H "Authorization: Bearer my-secret" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "sglang-user",
    "tier": "paid",
    "backend_url": "http://sglang-server:30000",
    "backend_type": "sglang"
  }'

Tiers

• Free: 1,000 requests/day, single backend target
• Paid: no rate limit, per-key backend routing

Streaming

Streaming is supported. Set stream: true in the request and the proxy forwards the SSE stream from the backend.

Note: Streaming responses do not include forge metadata (thinking_mode, complexity, token counts). The proxy cannot inspect the response until the stream completes, so the forge field is only present in non-streaming responses.

Dependencies

• qwen-think -- thinking session manager (routing, budget, sampling)
• FastAPI + uvicorn
• httpx -- async HTTP client for backend forwarding
• aiosqlite -- async SQLite for API keys and usage tracking

License

Apache-2.0