nvd-claude-proxy 0.3.4

Run Claude Code (and any Anthropic SDK client) on NVIDIA NIM models via a local proxy.

nvd-claude-proxy

Run Claude Code — and any Anthropic SDK client — on free NVIDIA NIM models.

A local HTTP proxy that speaks the Anthropic Messages API and forwards requests to https://integrate.api.nvidia.com/v1 (NVIDIA NIM / build.nvidia.com). Point your ANTHROPIC_BASE_URL at the proxy and your tools work unchanged while inference runs on Nemotron Ultra, Qwen3, DeepSeek-R1, or any other NIM model.

---

Install

# Recommended: isolated global install
pipx install nvd-claude-proxy

# Or plain pip
pip install nvd-claude-proxy

# Optional extras
pip install "nvd-claude-proxy[metrics]"   # Prometheus /metrics endpoint
pip install "nvd-claude-proxy[pdf]"       # PDF document block extraction
pip install "nvd-claude-proxy[full]"      # everything above

---

Quick start — ncp CLI (recommended)

# First run: save your API key permanently
ncp init
# → prompts for NVIDIA_API_KEY and saves to ~/.config/nvd-claude-proxy/.env

# Launch proxy + Claude Code in one command
ncp code

That's it. ncp code starts the proxy in the background, waits until it is ready, then launches claude. When Claude exits the proxy stops cleanly.

All ncp commands

Command	Description
ncp code	Start proxy → launch Claude Code
ncp proxy	Start proxy only (foreground)
ncp init	Save NVIDIA_API_KEY to global config
ncp models list	Show all configured model aliases
ncp kill	Terminate any stuck proxy process on port 8788
ncp version	Print version

Pass --api-key nvapi-… to any command for a one-shot override without saving.

---

Quick start — manual

# 1. Get a free API key at https://build.nvidia.com  (no credit card required)
export NVIDIA_API_KEY=nvapi-...

# 2. Start the proxy (default port 8788)
nvd-claude-proxy

In another shell:

export ANTHROPIC_BASE_URL=http://localhost:8788
export ANTHROPIC_API_KEY=not-used          # any non-empty string works
export ANTHROPIC_MODEL=claude-opus-4-7     # → Nemotron Ultra 253B
export ANTHROPIC_SMALL_FAST_MODEL=claude-haiku-4-5  # → Nemotron Nano 9B v2
claude

---

Default model mapping

Claude alias	NVIDIA NIM model	Notes
claude-opus-4-7	nvidia/llama-3.1-nemotron-ultra-253b-v1	Reasoning, best quality
claude-sonnet-4-6	nvidia/llama-3.3-nemotron-super-49b-v1	Balanced
claude-haiku-4-5	nvidia/nvidia-nemotron-nano-9b-v2	Fast, small
claude-opus-4-7-vision	meta/llama-4-maverick-17b-128e-instruct	Vision-capable
claude-qwen3	qwen/qwen3-235b-a22b	Qwen3 thinking
claude-r1	deepseek-ai/deepseek-r1	DeepSeek-R1

Legacy Claude 3.x names (claude-3-5-sonnet-*, claude-3-opus-*, etc.) are automatically routed to the matching tier via prefix fallbacks.

Override by setting MODEL_CONFIG_PATH=/path/to/your/models.yaml.

---

Environment variables

Variable	Default	Description
NVIDIA_API_KEY	required	nvapi-… key from build.nvidia.com
NVIDIA_BASE_URL	https://integrate.api.nvidia.com/v1	Override for self-hosted NIM
PROXY_HOST	127.0.0.1	Bind address (0.0.0.0 for Docker/remote)
PROXY_PORT	8788	Bind port
PROXY_API_KEY	(unset)	Require clients to present this key as Bearer token
LOG_LEVEL	INFO	DEBUG / INFO / WARNING / ERROR
MODEL_CONFIG_PATH	(bundled)	Path to a custom models.yaml
REQUEST_TIMEOUT_SECONDS	600	Total request timeout (long for reasoning streams)
MAX_RETRIES	2	Upstream retry budget for transient 5xx
RATE_LIMIT_RPM	0 (off)	Per-client sliding-window requests/minute; 0 = off
MAX_REQUEST_BODY_MB	0 (off)	Reject bodies larger than this; 0 = unlimited

Variables can be placed in:

• .env in the current directory
• ~/.config/nvd-claude-proxy/.env (written by ncp init)

---

API endpoints

Method	Path	Purpose
POST	/v1/messages	Anthropic Messages — streaming & non-streaming
POST	/v1/messages/count_tokens	Approximate token count (cl100k_base)
GET	/v1/models	List model aliases
GET	/v1/models/{id}	Single model lookup
GET	/healthz	Liveness probe
GET	/metrics	Prometheus metrics ([metrics] extra)
POST	/v1/messages/batches	501 stub (not supported by NIM)
POST	/v1/files	501 stub

---

Features

• Full Anthropic SDK compatibility — anthropic-version header, correct SSE Content-Type, proper message_start token counts
• Streaming — strict Anthropic SSE event ordering with keepalive ping events every 15 s
• Tool use — Anthropic tool definitions → OpenAI function-calling; parallel tool calls; schema sanitization for NIM
• Reasoning / thinking — thinking.budget_tokens enforced; <think> tags stripped from non-streaming
• Vision — JPEG/PNG pass-through; GIF/WEBP transcoded to PNG
• PDF documents — base64 PDF blocks extracted to plain text (requires [pdf] extra)
• Model failover — automatic retry on 5xx with the next model in the configured chain
• Context overflow guard — pre-flight check returns a clean 400 before the request reaches NVIDIA when input exceeds the model's window
• Shared connection pool — one httpx.AsyncClient for all requests (no per-request TLS setup)
• SIGHUP reload — kill -HUP <pid> reloads models.yaml without restart
• Sliding-window rate limiter — per-client, keyed on metadata.user_id or IP
• Prometheus metrics — request count, token usage, latency histograms
• Cost estimation — cost_usd_est field in every structured log line

---

Custom model config

Create a models.yaml (start from the bundled default):

defaults:
  big: "my-model"
  small: "my-model"

aliases:
  my-model:
    nvidia_id: "org/my-nim-model"
    supports_tools: true
    supports_vision: false
    supports_reasoning: false
    max_context: 131072
    max_output: 16384
    failover_to: []

prefix_fallbacks:
  "claude-": "my-model"

MODEL_CONFIG_PATH=./my_models.yaml nvd-claude-proxy
# or
ncp code --model-config ./my_models.yaml

---

Docker

docker run --rm -p 8788:8788 \
  -e NVIDIA_API_KEY=nvapi-... \
  ghcr.io/khiwn/nvd-claude-proxy:latest

Or clone the repo and run:

cp .env.example .env      # fill in NVIDIA_API_KEY
docker compose up

---

Troubleshooting

Symptom	Fix
NVIDIA_API_KEY Field required	Run ncp init to save your key globally
proxy did not start in time	Run ncp kill then ncp code again
132183 input tokens > 131072	Context overflow — proxy now returns a clean 400 with explanation
429 rate_limit_error	Hit NIM free-tier 40 RPM cap; wait 60 s or upgrade your NIM plan
Claude Code shows tool errors	Check ncp models list — server tools are silently dropped

---

Known limitations

• Prompt caching is silently ignored (NVIDIA NIM has no equivalent).
• thinking.signature is proxy-local — do not forward proxy-generated thinking blocks to the real Anthropic API.
• DeepSeek-R1 + tool use is unreliable; use Nemotron models for agentic workloads.
• Anthropic server tools (web_search, bash, computer, code_execution, memory) are silently dropped.
• Batch and Files APIs return 501 — NIM has no equivalent.

---

Development

git clone https://github.com/khiwn/nvd-claude-proxy
cd nvd-claude-proxy
cp .env.example .env      # fill in NVIDIA_API_KEY
make dev                  # pip install -e ".[dev,full]"
make test                 # pytest
make lint                 # ruff + mypy
make run                  # uvicorn on :8788

---

License

MIT — see LICENSE.