Working Set

Tools for scaling local LLM deployments: how many concurrent users or agents a given GPU configuration can keep warm, where KV cache, decode bandwidth, and prefill compute each become the binding constraint, and which knob (topology, dtypes, max_num_seqs, prompt caching) buys the most headroom for agentic coding workloads.

Start with the interactive explorer — live sliders for the workload, model (Qwen3.8-27B / 35B-A3B / Mistral-Medium-3.5 / GLM-5.3 / DeepSeek-V4-Flash / Qwen3.8-Flash-Next / GLM-5.3-Flash), GPU (H200 / B300), weight & KV dtypes, and DP × TP topology. It answers as a decision tool: a binding-constraint verdict, a deploy recipe (vLLM flags), the bill (€/GPU-hour and €/kWh sliders: hardware plus a duty-cycle power model), a sensitivity panel showing which assumption would flip the decision, the steady-state decode point (how many sessions are actually decoding at your load, and how fast each one runs — Little's law, not the all-warm stress test), shareable links that encode the whole configuration, and a "Test these hypotheses" button that hands out the configuration on screen as a workingset.toml — feed it to ws test below and measure the real limits on a live vLLM endpoint.

The workingset package

The model behind the explorer is a Python package (src/workingset/, the source of truth; the explorer's JS mirrors it). It ships a CLI:

uv run ws init --model Q38FN --gpu B300 --tp 8 --weight-dtype nvfp4   # writes workingset.toml
uv run ws predict workingset.toml        # the four ceilings, which one binds, the operating point
uv run ws predict workingset.toml --json # the same as a run record
uv run ws hypotheses                     # the H-* and what each one needs
uv run ws test workingset.toml --dry-run # the plan, the sampler self-check, no requests
uv run ws test workingset.toml --exclusive --out run.json   # measure it
uv run ws report run.json                # re-print the verdicts
uv run ws models                         # model / GPU keys
uv run pytest                            # self-checks + config round-trips

No checkout needed — the explorer's workingset.toml runs straight from git (--from carries the package because workingset publishes one console script, ws):

uvx --from git+https://github.com/T0mSIlver/working-set ws predict workingset.toml
uvx --from git+https://github.com/T0mSIlver/working-set ws test workingset.toml --dry-run
uvx --from git+https://github.com/T0mSIlver/working-set ws test workingset.toml --all --exclusive --out run.json

After the PyPI release the same commands shorten to uvx --from workingset ws ….

Predictions live in no file: ws predict recomputes them from the config every time, so a config can never carry a number the code did not produce. A harness .py downloaded from the explorer before the package existed still loads (its CONFIG block is extracted).

ws test puts the predictions to a live endpoint, one falsifiable hypothesis at a time. Without --exclusive it runs only the hypotheses that need a handful of requests (miss TTFT, the inter-token gap distribution, the steady decode point) and lists the rest as skipped — a hypothesis that has to generate its own population is never measured against someone else's load. With --exclusive it drives the geometric load ladder once, and every ceiling reads from it. --burst N adds the correlated-flush probe (B*).

Contents

• docs/writeup.md — baseline study: KV-cache capacity and the prompt-caching / offload / max_num_seqs trade-offs.

• docs/scenarios.md — extended scenario model: multi-GPU topologies, MoE vs dense, subagent workloads, the cost of a cache miss, and cold-spike tolerance.

• scripts/ — everything is reproducible:

  uv run ws selfcheck                # the shared model's self-checks (src/workingset/model.py)
  uv run scripts/scenarios.py        # renders the scenario figures
  uv run scripts/tables.py           # regenerates every number in docs/scenarios.md
  

• research/ — sourced constants for each model and GPU.

• interactive/ — the explorer, a dependency-free page mirroring the Python model: index.html holds the markup and styles, src/*.js the model and the charts as ES modules (src/main.js is the entry and lists the layering). Browsers refuse module scripts from file://, so serve the folder to open it locally:

  python3 -m http.server 8000 --directory interactive   # then http://localhost:8000
  

Method, calibration, and caveats are laid out in the docs above.

MIT licensed; see LICENSE.