Skip to main content

llmdup

From a Kubernetes cluster to a working llm-d endpoint, in one command.

A k9s-style terminal UI for people. A JSON CLI and an MCP server for scripts and agents. One core, three front ends, zero copied manifests.

License Python llm-d Guides GPU


llmdup terminal UI: the guides view, with support for the detected accelerator
$ llmdup local up --yes
==> [2/9] prerequisites.gaie
    customresourcedefinition.apiextensions.k8s.io/inferencepools.inference.networking.k8s.io created
==> [5/9] deploy.standalone
==> [6/9] local.router_small
==> [7/9] local.simulator
    deployment.apps/llmdup-sim created
... 0/3 pods ready, 2 model server pods, 0/2 deployments available
... 3/3 pods ready, 2 model server pods, 2/2 deployments available
==> [9/9] verify.tests.request
    {"id":"cmpl-02a38584-…","model":"llmdup-sim","object":"text_completion","choices":[{"text":"I am fine, how are you today? …"}]}
ok: up optimized-baseline: all steps ok

That is a real run: the llm-d router, the Gateway API Inference Extension and two simulated model servers on a laptop kind cluster, answering /v1/completions through the router. No GPU, about two minutes.


Why llmdup

llm-d 1.0 ships every well-lit path as a machine-readable recipe: a guide.yaml with variables, a support matrix and ordered prerequisites, deploy, verify and cleanup steps, rendered by the release's own scripts/guide.py. llmdup drives that recipe. It writes no install commands of its own.

By hand from the README llmdup
Get the guides git clone, check out the release branch Downloads the pinned tag once, verifies its commit, caches it
Pick variables Read the support table, export a dozen variables Detects accelerator, provider and model; validates every value against the guide
Run the steps Copy blocks, skip the ones that do not apply Plans exactly the steps for your mode and choices, in guide order
Cluster-wide changes Easy to miss Flagged, shown with the real values, run only after you confirm
Know it worked Watch pods, craft a curl Waits for ready pods, sends the guide's own smoke test
Tear down Find the cleanup block, hope it is complete Runs the guide's cleanup, then checks nothing is left
Automate it Parse prose One JSON document per command, stable exit codes, an MCP server

Install

uv tool install llmdup        # or: pip install llmdup
llmdup version

Needs Python 3.14 or newer (uv installs it for you), kubectl and helm. For the local demo, also kind with Docker or Podman.

Try it without a GPU

llmdup local up --yes         # kind cluster "llmdup" + optimized-baseline + llm-d-inference-sim
llmdup status                 # router and simulator pods, endpoint, how to reach it
llmdup test                   # run the guide's verify tests again
llmdup down --yes             # the guide's cleanup, then a check that nothing is left
llmdup local down --yes       # delete the kind cluster

The local profile is the optimized-baseline plan with three changes: the model servers become two llm-d-inference-sim replicas, the router is shrunk to fit a laptop, and the Hugging Face secret is skipped because the simulator downloads nothing. With Podman, set KIND_EXPERIMENTAL_PROVIDER=podman.

The same profile runs on any cluster, to try the router, a Gateway and the whole flow before you have GPUs:

llmdup up optimized-baseline --profile local --context my-cluster --yes              # standalone
llmdup up optimized-baseline --profile local --mode gateway --context my-cluster --yes

Deploy on your cluster

Preflight first. Example on a GKE node pool with eight NVIDIA GPUs:

$ llmdup doctor optimized-baseline --context my-cluster
[ok  ] Tools: kubectl and helm found
[ok  ] Cluster reachable: context my-cluster answers
[ok  ] Kubernetes version: v1.34.2-gke.1100000
[ok  ] Accelerators: gpu: 8 x nvidia.com/gpu
[warn] Inference Extension CRDs: inferencepools.inference.networking.k8s.io not installed
       fix: the plan installs them in its cluster-scoped prerequisites step, after confirmation
[ok  ] Guide support: gpu + vllm: validated
[ok  ] Capacity: 4 nvidia.com/gpu of 8
[ok  ] HF_TOKEN: set
[ok  ] Existing release: no release optimized-baseline in llm-d-optimized-baseline
[ok  ] Existing stack: no llmdup stack in llm-d-optimized-baseline
[skip] Gateway provider: mode standalone
$ llmdup plan optimized-baseline --accelerator gpu
plan: up optimized-baseline  namespace llm-d-optimized-baseline  mode standalone  llm-d v1.0.0

#  STEP                         TAGS
1  prerequisites.gaie           cluster-scoped
2  prerequisites.namespace
3  prerequisites.secrets
4  deploy.router_values
5  deploy.standalone            mode
6  deploy.modelserver.standard  manual option
7  verify.endpoint.standalone   mode
8  verify.tests.request         sends traffic

needs confirmation: prerequisites.gaie
llmdup up optimized-baseline --context my-cluster --dry-run   # the exact script, nothing runs
llmdup up optimized-baseline --context my-cluster --yes       # doctor, deploy, wait, smoke test
llmdup status --context my-cluster
llmdup down --context my-cluster --namespace llm-d-optimized-baseline --yes

Choices you leave out come from detection first, then from the guide's defaults: the accelerator from allocatable resources and DRA DeviceClasses, INFRA_PROVIDER=gke on GKE node pools, MODEL from the support matrix. A value the guide does not allow exits with code 2 and lists the allowed ones. --mode gateway and every accelerator and engine pair in a guide's matrix go through the same path.

Gateway mode expects the Gateway llm-d-inference-gateway in the stack's namespace, created the way llm-d's gateway docs describe; doctor checks it. Before the smoke test, llmdup waits until the Gateway is programmed and has an address. On a GKE Gateway (gke-l7-* class) llmdup adds two steps. gateway.provider switches the router chart to provider.name=gke as llm-d's GKE doc does, because GKE rejects the route the guide's provider.name=none creates. verify.gateway_route retries one completion through the Gateway until it answers, because GKE's load balancer starts routing a minute or two after the Gateway is programmed. Gateway mode on GKE also needs the Network Services API (networkservices.googleapis.com) enabled in the project.

Every guide in llm-d v1.0.0

All 27 guides with a guide.yaml load and plan for up, test and down, checked against the real v1.0.0 release by the network test suite:

llmdup guides list
GUIDE                         SECTION             WHAT IT DOES                                     MODES               SUPPORT
agentic-api                   foundations         Serve the Stateful OpenAI Responses API          standalone          gpu+vllm: validated
multimodal-serving            foundations         Serve Multimodal Models                          standalone          gpu+vllm: validated
omni-serving                  foundations         Serve Omni Models                                standalone          gpu+vllmomni: community
optimized-baseline            foundations         Deploy an Optimized Baseline                     standalone,gateway  gpu+vllm: validated
p2p-kv-cache-sharing          foundations         Enable P2P Prefix Cache Sharing                  standalone          gpu+vllm: community
pd-disaggregation             foundations         Deploy Prefill/Decode Disaggregation             standalone          gpu+vllm: validated
precise-prefix-cache-routing  foundations         Enable Precise Prefix-Cache Aware Routing        standalone          gpu+vllm: validated
predicted-latency-routing     foundations         Route Requests by Predicted Latency              standalone          gpu+vllm: validated
tiered-prefix-cache           foundations         Configure a Tiered Prefix Cache                  standalone          gpu+vllm: validated
wide-ep                       foundations         Scale MoE Models with Wide Expert Parallelism    standalone          gpu+vllm: validated
deepseek-v4                   models              DeepSeek-V4                                      standalone          gpu+vllm: community
glm-5-2                       models              GLM-5.2                                          standalone          gpu+vllm: community
nemotron-3-ultra              models              NVIDIA Nemotron 3 Ultra                          standalone          gpu+vllm: community
qwen3-5-397b                  models              Qwen3.5-397B                                     standalone          tpu+vllm: community
async-multitenant             operations          Enforce Tenant Quotas and Priorities             -                   -
keda-epp                      operations          Scale on Queue Depth and Saturation              -                   -
keda-epp-token-aware          operations          Scale on Token Backlog (Experimental)            -                   -
kueue-rebalancing             operations          Share a GPU Budget Across Models (Experimental)  -                   -
slo-aware                     operations          Scale to Latency SLOs                            -                   -
fast-model-actuation-keda     operations-startup  Scale from Zero with FMA and KEDA                standalone          -
fast-model-actuation          operations-startup  Reuse Warm Model Servers (FMA)                   standalone          -
modelexpress-p2p              operations-startup  Transfer Weights Peer-to-Peer (ModelExpress)     standalone,gateway  -
pod-snapshot                  operations-startup  Restore from Pod Snapshots                       standalone,gateway  -
flow-control                  operations-traffic  Queue and Prioritize Requests (Flow Control)     standalone,gateway  -
multi-inference-pool          operations-traffic  Deploy Multiple Inference Pools                  -                   -
turn-priority-fairness        unlisted            Turn-Priority Fairness                           standalone,gateway  -
wva                           unlisted            Autoscaling Workloads with HPA and WVA Metrics   -                   -

For scripts and agents

Every subcommand runs without prompts when given --yes, and with --json prints exactly one JSON document on stdout. Progress and logs go to stderr.

{"schema_version": 1, "ok": true, "command": "up", "data": {"...": "..."}, "errors": []}
Exit code Meaning
0 Success
1 A step failed; data names the step, its command and its last 40 log lines
2 Bad usage, or a value the guide does not allow; the hint lists the allowed values
3 doctor found a blocking check
4 Confirmation needed, but --yes is absent and there is no TTY

MCP server

claude mcp add llmdup -- llmdup mcp

Eight tools over stdio: list_guides, show_guide, doctor, plan, up, status, test, down. They return the same models as the CLI's data. up and down are two-step by design: without confirm: true they change nothing and return the plan, the target kube context and the exact commands, and the confirmed call must name that same context, so the change lands on the cluster the user approved. skills/llmdup/SKILL.md teaches an agent the CLI and these rules.

Terminal UI

llmdup with no arguments opens a keyboard-only UI in the style of k9s. It calls the same core functions as the CLI, so a person at the keyboard and an agent over MCP get the same behaviour.

Plan view
Plan
Deploy view
Deploy
Doctor view
Doctor
Stacks view
Stacks
Filter
Filter
Help view
Help
  • Plan: variables with their source, ordered steps with tags, and the exact command of the selected step. e edits a value, ctrl-s saves the script, u deploys.
  • Deploy: step progress on the left, the live log on the right. It runs in a background worker, so leaving the view does not stop it; ctrl-k cancels.
  • Doctor: every preflight check with what was found and how to fix it; c copies the fix.
  • Stacks: pods, endpoint and last test of each deployed stack. t sends a test request, l tails the logs, m reads the model-server metrics, shift-f port-forwards, ctrl-d tears down.
  • Filter: /cache narrows any table as you type; /!text inverts, /-f text is fuzzy.
  • Help: every key, grouped, generated from the same key map as the header hints, so the two never drift apart.

Keys follow k9s wherever the meaning is the same: j/k to move, enter to open, esc to go back, [ and ] for view history, : for commands such as :plan optimized-baseline, :stacks or :ctx, ? for help. Every cluster-wide step and every teardown opens a dialog with the exact commands. The HF_TOKEN value is never drawn, only set or not set.

Screens above are the app on a real GKE 1.36 cluster with one L4 GPU node: guides, doctor and plan for that GPU, then a deploy, test and tear down of the simulator profile, all driven from the UI. The layout also fits a 100 x 30 terminal.

How it works

flowchart LR
  subgraph front["Front ends: collect input, show results"]
    TUI["Terminal UI<br/>Textual"]
    CLI["CLI<br/>Typer, --json"]
    MCP["MCP server<br/>+ agent skill"]
  end
  subgraph core["core/ (Pydantic in, Pydantic out)"]
    G["guides + planner"]
    D["detect + doctor"]
    R["runner"]
    S["state + status"]
  end
  REL["llm-d v1.0.0<br/>guide.yaml + scripts/guide.py<br/>cached, commit-checked"]
  K["kube.py<br/>kubectl · helm · kind · bash<br/>pinned to one context"]
  C[("Kubernetes")]
  TUI --> core
  CLI --> core
  MCP --> core
  G -- reads --> REL
  R --> K --> C
  D --> K
  S --> K
  1. Fetch. The tarball of the pinned tag is downloaded once into ~/.cache/llmdup/llm-d/<ref>/, unpacked with tarfile's data filter and checked against the expected commit.
  2. Plan. The release's own guide.py is imported. Each step path is classified (cluster-scoped, mode, manual option, sends traffic, skipped), and each surviving step is emitted with Guide.emit() using your variables.
  3. Run. Steps run one by one in a fresh working directory. Each step's exports carry into the next, and the run stops at the first failure with the command and its last 40 lines.
  4. Wait and verify. Between deploy and verify, llmdup waits for the guide's model-server pods and the router's pods, then runs the guide's first test.
  5. Remember. A llmdup-state ConfigMap in the stack's namespace records guide, variables, mode, llm-d ref and commit, so status, test and down work from any machine. A deploy cut short shows as partial.

Safety by construction

  • Your context, pinned. The kube context is resolved once per command and passed to every kubectl, helm and bash step. Switching context in another terminal cannot redirect a running deploy.
  • Ask first. Cluster-scoped steps (CRDs, ClusterRoles, wherever a guide puts them) and every delete need --yes, a dialog in the UI, or confirm: true in MCP, and the confirmation shows the context, namespace and commands with real values.
  • Secrets stay in the environment. HF_TOKEN and every variable a guide marks sensitive are read from the environment only. --set refuses them, and their values are redacted from logs, JSON, state and the screen.
  • Honest teardown. down runs every cleanup command, then checks the namespace, Helm releases and pods. Anything it cannot verify counts as left behind, and the state is kept so you can try again.
  • One door to the cluster. Only core/kube.py starts processes. SIGTERM and SIGHUP stop the running step instead of orphaning it.

Development

uv sync
uv run pytest                  # offline: unit, planner, doctor, runner, MCP, UI pilot and snapshot tests
uv run pytest -m network       # the real llm-d v1.0.0 release, all 27 guides
uv run pytest -m e2e           # kind + simulator, CLI and UI; refuses any context not starting with kind-llmdup
uv run ruff check . && uv run ruff format --check . && uv run ty check

CI runs lint, types and tests on Python 3.14, 3.15 and the free-threaded 3.14t, plus the upstream and end-to-end suites on main and nightly. PLAN.md is the design, PROGRESS.md tracks each phase and every place where upstream llm-d differs from the plan.

Not in 0.1: creating cloud clusters, choosing a configuration from a workload or SLA, benchmarks, stacking several guides in one namespace, upgrades between llm-d versions, and guides without a guide.yaml.

License

Apache-2.0, like llm-d. Test fixtures copied from llm-d are listed in NOTICE.

Metadata

Release files for llmdup 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llmdup 0.1.0
File Size Uploaded Provenance Actions
llmdup-0.1.0.tar.gz 97.0 kB Without provenance Details for llmdup-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llmdup 0.1.0
File Interpreter ABI Platform Actions
llmdup-0.1.0-py3-none-any.whl Python 3 none any Details for llmdup-0.1.0-py3-none-any.whl

Total release size: 210.6 kB

Release files / llmdup-0.1.0.tar.gz

Download URL llmdup-0.1.0.tar.gz
Size 97.0 kB
Tags Source
SHA-256 checksum
How to use checksums
518387951e91ea0fa0c6b3e2e6ab331cb0dc1e01e6d0437561c863ea39b63b8a
BLAKE2b-256 checksum
How to use checksums
1e9cf9eb65da27cf7118fbf4d8c201ac0295c960007e25a222a28e982cd48217
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / llmdup-0.1.0-py3-none-any.whl

Download URL llmdup-0.1.0-py3-none-any.whl
Size 113.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
41d658d50f0e8de10a2b9eef535a3e58fe242920aae6e3af67e083256cbb7775
BLAKE2b-256 checksum
How to use checksums
e66decdc7f6ee9e167b7729474257792647eb9d73dd833dfdca33412f85a3756
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for llmdup 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page