llmdup
From a Kubernetes cluster to a working llm-d endpoint, in one command.
A k9s-style terminal UI for people. A JSON CLI and an MCP server for scripts and agents. One core, three front ends, zero copied manifests.
$ llmdup local up --yes
==> [2/9] prerequisites.gaie
customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io created
==> [5/9] deploy.standalone
==> [6/9] local.router_small
==> [7/9] local.simulator
deployment.apps/llmdup-sim created
... 0/3 pods ready, 2 model server pods, 0/2 deployments available
... 3/3 pods ready, 2 model server pods, 2/2 deployments available
==> [9/9] verify.tests.request
{"id":"cmpl-02a38584-…","model":"llmdup-sim","object":"text_completion","choices":[{"text":"I am fine, how are you today? …"}]}
ok: up optimized-baseline: all steps ok
That is a real run: the llm-d router, the Gateway API Inference Extension and two simulated model servers on a laptop kind cluster, answering /v1/completions through the router. No GPU, about two minutes.
Why llmdup
llm-d 1.0 ships every well-lit path as a machine-readable recipe: a guide.yaml with variables, a support matrix and ordered prerequisites, deploy, verify and cleanup steps, rendered by the release's own scripts/guide.py. llmdup drives that recipe. It writes no install commands of its own.
| By hand from the README | llmdup | |
|---|---|---|
| Get the guides | git clone, check out the release branch |
Downloads the pinned tag once, verifies its commit, caches it |
| Pick variables | Read the support table, export a dozen variables | Detects accelerator, provider and model; validates every value against the guide |
| Run the steps | Copy blocks, skip the ones that do not apply | Plans exactly the steps for your mode and choices, in guide order |
| Cluster-wide changes | Easy to miss | Flagged, shown with the real values, run only after you confirm |
| Know it worked | Watch pods, craft a curl | Waits for ready pods, sends the guide's own smoke test |
| Tear down | Find the cleanup block, hope it is complete | Runs the guide's cleanup, then checks nothing is left |
| Automate it | Parse prose | One JSON document per command, stable exit codes, an MCP server |
Install
uv tool install llmdup # or: pip install llmdup
llmdup version
Needs Python 3.14 or newer (uv installs it for you), kubectl and helm. For the local demo, also kind with Docker or Podman.
Try it without a GPU
llmdup local up --yes # kind cluster "llmdup" + optimized-baseline + llm-d-inference-sim
llmdup status # router and simulator pods, endpoint, how to reach it
llmdup test # run the guide's verify tests again
llmdup down --yes # the guide's cleanup, then a check that nothing is left
llmdup local down --yes # delete the kind cluster
The local profile is the optimized-baseline plan with three changes: the model servers become two llm-d-inference-sim replicas, the router is shrunk to fit a laptop, and the Hugging Face secret is skipped because the simulator downloads nothing. With Podman, set KIND_EXPERIMENTAL_PROVIDER=podman.
The same profile runs on any cluster, to try the router, a Gateway and the whole flow before you have GPUs:
llmdup up optimized-baseline --profile local --context my-cluster --yes # standalone
llmdup up optimized-baseline --profile local --mode gateway --context my-cluster --yes
Deploy on your cluster
Preflight first. Example on a GKE node pool with eight NVIDIA GPUs:
$ llmdup doctor optimized-baseline --context my-cluster
[ok ] Tools: kubectl and helm found
[ok ] Cluster reachable: context my-cluster answers
[ok ] Kubernetes version: v1.34.2-gke.1100000
[ok ] Accelerators: gpu: 8 x nvidia.com/gpu
[warn] Inference Extension CRDs: inferencepools.inference.networking.k8s.io not installed
fix: the plan installs them in its cluster-scoped prerequisites step, after confirmation
[ok ] Guide support: gpu + vllm: validated
[ok ] Capacity: 4 nvidia.com/gpu of 8
[ok ] HF_TOKEN: set
[ok ] Existing release: no release optimized-baseline in llm-d-optimized-baseline
[ok ] Existing stack: no llmdup stack in llm-d-optimized-baseline
[skip] Gateway provider: mode standalone
$ llmdup plan optimized-baseline --accelerator gpu
plan: up optimized-baseline namespace llm-d-optimized-baseline mode standalone llm-d v1.0.0
# STEP TAGS
1 prerequisites.gaie cluster-scoped
2 prerequisites.namespace
3 prerequisites.secrets
4 deploy.router_values
5 deploy.standalone mode
6 deploy.modelserver.standard manual option
7 verify.endpoint.standalone mode
8 verify.tests.request sends traffic
needs confirmation: prerequisites.gaie
llmdup up optimized-baseline --context my-cluster --dry-run # the exact script, nothing runs
llmdup up optimized-baseline --context my-cluster --yes # doctor, deploy, wait, smoke test
llmdup status --context my-cluster
llmdup down --context my-cluster --namespace llm-d-optimized-baseline --yes
Choices you leave out come from detection first, then from the guide's defaults: the accelerator from allocatable resources and DRA DeviceClasses, INFRA_PROVIDER=gke on GKE node pools, MODEL from the support matrix. A value the guide does not allow exits with code 2 and lists the allowed ones. --mode gateway and every accelerator and engine pair in a guide's matrix go through the same path.
Gateway mode expects the Gateway llm-d-inference-gateway in the stack's namespace, created the way llm-d's gateway docs describe; doctor checks it. Before the smoke test, llmdup waits until the Gateway is programmed and has an address. On a GKE Gateway (gke-l7-* class) llmdup adds two steps. gateway.provider switches the router chart to provider.name=gke as llm-d's GKE doc does, because GKE rejects the route the guide's provider.name=none creates. verify.gateway_route retries one completion through the Gateway until it answers, because GKE's load balancer starts routing a minute or two after the Gateway is programmed. Gateway mode on GKE also needs the Network Services API (networkservices.googleapis.com) enabled in the project.
Every guide in llm-d v1.0.0
All 27 guides with a guide.yaml load and plan for up, test and down, checked against the real v1.0.0 release by the network test suite:
llmdup guides list
GUIDE SECTION WHAT IT DOES MODES SUPPORT
agentic-api foundations Serve the Stateful OpenAI Responses API standalone gpu+vllm: validated
multimodal-serving foundations Serve Multimodal Models standalone gpu+vllm: validated
omni-serving foundations Serve Omni Models standalone gpu+vllmomni: community
optimized-baseline foundations Deploy an Optimized Baseline standalone,gateway gpu+vllm: validated
p2p-kv-cache-sharing foundations Enable P2P Prefix Cache Sharing standalone gpu+vllm: community
pd-disaggregation foundations Deploy Prefill/Decode Disaggregation standalone gpu+vllm: validated
precise-prefix-cache-routing foundations Enable Precise Prefix-Cache Aware Routing standalone gpu+vllm: validated
predicted-latency-routing foundations Route Requests by Predicted Latency standalone gpu+vllm: validated
tiered-prefix-cache foundations Configure a Tiered Prefix Cache standalone gpu+vllm: validated
wide-ep foundations Scale MoE Models with Wide Expert Parallelism standalone gpu+vllm: validated
deepseek-v4 models DeepSeek-V4 standalone gpu+vllm: community
glm-5-2 models GLM-5.2 standalone gpu+vllm: community
nemotron-3-ultra models NVIDIA Nemotron 3 Ultra standalone gpu+vllm: community
qwen3-5-397b models Qwen3.5-397B standalone tpu+vllm: community
async-multitenant operations Enforce Tenant Quotas and Priorities - -
keda-epp operations Scale on Queue Depth and Saturation - -
keda-epp-token-aware operations Scale on Token Backlog (Experimental) - -
kueue-rebalancing operations Share a GPU Budget Across Models (Experimental) - -
slo-aware operations Scale to Latency SLOs - -
fast-model-actuation-keda operations-startup Scale from Zero with FMA and KEDA standalone -
fast-model-actuation operations-startup Reuse Warm Model Servers (FMA) standalone -
modelexpress-p2p operations-startup Transfer Weights Peer-to-Peer (ModelExpress) standalone,gateway -
pod-snapshot operations-startup Restore from Pod Snapshots standalone,gateway -
flow-control operations-traffic Queue and Prioritize Requests (Flow Control) standalone,gateway -
multi-inference-pool operations-traffic Deploy Multiple Inference Pools - -
turn-priority-fairness unlisted Turn-Priority Fairness standalone,gateway -
wva unlisted Autoscaling Workloads with HPA and WVA Metrics - -
For scripts and agents
Every subcommand runs without prompts when given --yes, and with --json prints exactly one JSON document on stdout. Progress and logs go to stderr.
{"schema_version": 1, "ok": true, "command": "up", "data": {"...": "..."}, "errors": []}
| Exit code | Meaning |
|---|---|
| 0 | Success |
| 1 | A step failed; data names the step, its command and its last 40 log lines |
| 2 | Bad usage, or a value the guide does not allow; the hint lists the allowed values |
| 3 | doctor found a blocking check |
| 4 | Confirmation needed, but --yes is absent and there is no TTY |
MCP server
claude mcp add llmdup -- llmdup mcp
Eight tools over stdio: list_guides, show_guide, doctor, plan, up, status, test, down. They return the same models as the CLI's data. up and down are two-step by design: without confirm: true they change nothing and return the plan, the target kube context and the exact commands, and the confirmed call must name that same context, so the change lands on the cluster the user approved. skills/llmdup/SKILL.md teaches an agent the CLI and these rules.
Terminal UI
llmdup with no arguments opens a keyboard-only UI in the style of k9s. It calls the same core functions as the CLI, so a person at the keyboard and an agent over MCP get the same behaviour.
Plan |
Deploy |
Doctor |
Stacks |
Filter |
Help |
- Plan: variables with their source, ordered steps with tags, and the exact command of the selected step.
eedits a value,ctrl-ssaves the script,udeploys. - Deploy: step progress on the left, the live log on the right. It runs in a background worker, so leaving the view does not stop it;
ctrl-kcancels. - Doctor: every preflight check with what was found and how to fix it;
ccopies the fix. - Stacks: pods, endpoint and last test of each deployed stack.
tsends a test request,ltails the logs,mreads the model-server metrics,shift-fport-forwards,ctrl-dtears down. - Filter:
/cachenarrows any table as you type;/!textinverts,/-f textis fuzzy. - Help: every key, grouped, generated from the same key map as the header hints, so the two never drift apart.
Keys follow k9s wherever the meaning is the same: j/k to move, enter to open, esc to go back, [ and ] for view history, : for commands such as :plan optimized-baseline, :stacks or :ctx, ? for help. Every cluster-wide step and every teardown opens a dialog with the exact commands. The HF_TOKEN value is never drawn, only set or not set.
Screens above are the app on a real GKE 1.36 cluster with one L4 GPU node: guides, doctor and plan for that GPU, then a deploy, test and tear down of the simulator profile, all driven from the UI. The layout also fits a 100 x 30 terminal.
How it works
flowchart LR
subgraph front["Front ends: collect input, show results"]
TUI["Terminal UI<br/>Textual"]
CLI["CLI<br/>Typer, --json"]
MCP["MCP server<br/>+ agent skill"]
end
subgraph core["core/ (Pydantic in, Pydantic out)"]
G["guides + planner"]
D["detect + doctor"]
R["runner"]
S["state + status"]
end
REL["llm-d v1.0.0<br/>guide.yaml + scripts/guide.py<br/>cached, commit-checked"]
K["kube.py<br/>kubectl · helm · kind · bash<br/>pinned to one context"]
C[("Kubernetes")]
TUI --> core
CLI --> core
MCP --> core
G -- reads --> REL
R --> K --> C
D --> K
S --> K
- Fetch. The tarball of the pinned tag is downloaded once into
~/.cache/llmdup/llm-d/<ref>/, unpacked with tarfile'sdatafilter and checked against the expected commit. - Plan. The release's own
guide.pyis imported. Each step path is classified (cluster-scoped,mode,manual option,sends traffic,skipped), and each surviving step is emitted withGuide.emit()using your variables. - Run. Steps run one by one in a fresh working directory. Each step's exports carry into the next, and the run stops at the first failure with the command and its last 40 lines.
- Wait and verify. Between deploy and verify, llmdup waits for the guide's model-server pods and the router's pods, then runs the guide's first test.
- Remember. A
llmdup-stateConfigMap in the stack's namespace records guide, variables, mode, llm-d ref and commit, sostatus,testanddownwork from any machine. A deploy cut short shows aspartial.
Safety by construction
- Your context, pinned. The kube context is resolved once per command and passed to every
kubectl,helmand bash step. Switching context in another terminal cannot redirect a running deploy. - Ask first. Cluster-scoped steps (CRDs, ClusterRoles, wherever a guide puts them) and every delete need
--yes, a dialog in the UI, orconfirm: truein MCP, and the confirmation shows the context, namespace and commands with real values. - Secrets stay in the environment.
HF_TOKENand every variable a guide marks sensitive are read from the environment only.--setrefuses them, and their values are redacted from logs, JSON, state and the screen. - Honest teardown.
downruns every cleanup command, then checks the namespace, Helm releases and pods. Anything it cannot verify counts as left behind, and the state is kept so you can try again. - One door to the cluster. Only
core/kube.pystarts processes. SIGTERM and SIGHUP stop the running step instead of orphaning it.
Development
uv sync
uv run pytest # offline: unit, planner, doctor, runner, MCP, UI pilot and snapshot tests
uv run pytest -m network # the real llm-d v1.0.0 release, all 27 guides
uv run pytest -m e2e # kind + simulator, CLI and UI; refuses any context not starting with kind-llmdup
uv run ruff check . && uv run ruff format --check . && uv run ty check
CI runs lint, types and tests on Python 3.14, 3.15 and the free-threaded 3.14t, plus the upstream and end-to-end suites on main and nightly. PLAN.md is the design, PROGRESS.md tracks each phase and every place where upstream llm-d differs from the plan.
Not in 0.1: creating cloud clusters, choosing a configuration from a workload or SLA, benchmarks, stacking several guides in one namespace, upgrades between llm-d versions, and guides without a guide.yaml.
License
Apache-2.0, like llm-d. Test fixtures copied from llm-d are listed in NOTICE.
Metadata
Release files for llmdup 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llmdup-0.1.0.tar.gz | 97.0 kB | Without provenance | Details for llmdup-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llmdup-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for llmdup-0.1.0-py3-none-any.whl |
Total release size: 210.6 kB
Release files / llmdup-0.1.0.tar.gz
| Download URL | llmdup-0.1.0.tar.gz |
|---|---|
| Size | 97.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
518387951e91ea0fa0c6b3e2e6ab331cb0dc1e01e6d0437561c863ea39b63b8a
|
|
BLAKE2b-256 checksum How to use checksums |
1e9cf9eb65da27cf7118fbf4d8c201ac0295c960007e25a222a28e982cd48217
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / llmdup-0.1.0-py3-none-any.whl
| Download URL | llmdup-0.1.0-py3-none-any.whl |
|---|---|
| Size | 113.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
41d658d50f0e8de10a2b9eef535a3e58fe242920aae6e3af67e083256cbb7775
|
|
BLAKE2b-256 checksum How to use checksums |
e66decdc7f6ee9e167b7729474257792647eb9d73dd833dfdca33412f85a3756
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for llmdup 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.