BidKV

Framework-portable KV cache request scheduling primitive.

中文文档

Overview

bidkv is a zero-dependency Python package that addresses the victim-selection problem under KV cache pressure: when KV memory is exhausted, which request should be preempted?

The core idea is to evict the request that frees the most KV space per unit of quality loss, maximising utility:

$$U(r, \delta) = \frac{r}{\delta + \varepsilon}, \quad \varepsilon = 10^{-3}$$

where $r$ = tokens freed, $\delta$ = surrogate disruption estimate.

BidKV does not compress tokens — it only controls who gets preempted. The actual eviction is performed by the framework's native preempt + recompute path (vLLM) or RadixCache eviction (SGLang).

Ecosystem classification

BidKV is a scheduler-local victim-selection policy component. It is not a KV store, transport, connector, compression mechanism, or external state system. The repository also contains framework adapters and experiment tooling, but those delivery surfaces do not change the policy's runtime role.

See .vllm-hust/repository-profile.json for the machine-readable boundary and migration contract.

Module Layout

Baseline Strategies

Configuration

from bidkv import BidKVConfig

# Default: all bid logic bypassed (safe to import without activating)
config = BidKVConfig(enabled=False)

# Enable BidKV scheduling
config = BidKVConfig(enabled=True)
assert config.is_active

# Kill switch: immediately bypasses all logic even when enabled=True
config = BidKVConfig(enabled=True, kill_switch=True)
assert not config.is_active

Adding a Custom Strategy

from bidkv import (
    BaselineRegistry,
    BidKVStrategy,
    PreemptEvictStrategy, LargestFirstStrategy,
    StaticRandomStrategy, PreemptEvictSJFStrategy,
)

# Register all built-in strategies at once
registry = BaselineRegistry()
registry.create_default_registry()

# Or register selectively
registry2 = BaselineRegistry()
registry2.register(BidKVStrategy())
registry2.register(PreemptEvictStrategy())

strategy = registry2.get("bidkv")
print(strategy.name)              # "bidkv"
print(registry2.list_strategies())  # ["bidkv", "preempt-evict"]

Running Experiments

# vLLM: 5 strategies × mixed workload × 3 rates × 3 runs
HF_HUB_OFFLINE=1 python -m bidkv.experiments.vllm.runner \
    --strategies "preempt-evict,preempt-evict-sjf,static-random,largest-first,bidkv" \
    --workloads mixed \
    --mixed-rates 2.0,3.8,5.7 \
    --runs 3 \
    --output-dir results/vllm_experiment \
    --gpu-memory-utilization 0.5 \
    --num-gpu-blocks-override 600 \
    --max-num-seqs 32

# SGLang: 3 strategies
HF_HUB_OFFLINE=1 python -m bidkv.experiments.sglang.runner \
    --strategies "sglang_default,slack_aware,bidkv" \
    --workloads mixed \
    --runs 3 \
    --output-dir results/sglang_experiment

Framework Integration (vLLM)

With sibling vllm-hust-dev-hub, the repository manifest reduces activation to one command. The dev hub installs the package, verifies the exact entry point, and supplies the native selector configuration:

cd ../vllm-hust-dev-hub
./manage.sh restart --optimization bidkv

For vllm-hust, install BidKV into the same Python environment. The runtime discovers it through the vllm.victim_selector entry point. Installing the package is inert by default; explicitly select and enable BidKV when serving:

python -m pip install -e . --no-deps

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enforce-eager \
    --port 8000 \
    --additional-config '{
      "victim_selector_plugin": "bidkv",
      "enable_utility_victim_selection": true,
      "utility_strategy": "bidkv",
      "utility_kv_gate": 0.95
    }'

Verify discovery before launching:

python - <<'PY'
from importlib.metadata import entry_points

for ep in entry_points(group="vllm.victim_selector"):
    print(ep.name, "->", ep.value)
PY

Environment variables with the BIDKV_UTILITY_ prefix provide equivalent runtime configuration. BIDKV_STRATEGY belongs to the legacy experiment adapter, which monkey-patches the scheduler. Do not combine it with the native victim-selector integration.

Experimental typed bundle path

BidKV also ships bidkv/manifests/vllm-hust-extension-v1.json for the experimental vLLM-HUST Extension Bundle v1 path. This manifest describes BidKV as a scheduler policy; it does not describe a KV store, connector, or external system. The wheel registers the static manifest through vllm.extension_bundles; registration does not import BidKV or enable scheduling behavior. Inspect and validate the installed Bundle, then select org.vllm-hust.bidkv/victim-selector through additional_config.victim_selector_component.

vllm plugin inspect org.vllm-hust.bidkv
vllm plugin validate org.vllm-hust.bidkv

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --extension org.vllm-hust.bidkv \
    --additional-config '{
      "victim_selector_component": "org.vllm-hust.bidkv/victim-selector",
      "enable_utility_victim_selection": true,
      "utility_strategy": "bidkv",
      "utility_kv_gate": 0.95
    }'

Do not enable the typed manifest and the legacy vllm.victim_selector provider as two independent implementations. The typed scheduler materializer takes precedence when this Bundle is admitted. To roll back, remove --extension and victim_selector_component, select victim_selector_plugin=bidkv if the legacy path is desired, and start a fresh process. To bypass both paths during an incident, set victim_selector_plugin_disabled=true in additional_config, which selects the upstream-compatible no-op policy.

This path remains experimental until matched legacy-versus-typed scheduler traces verify victim choices, metrics, failures, and rollback against an exact vLLM-HUST revision.

Legacy experiment adapter

Use this path only to reproduce the historical multi-strategy experiments:

BIDKV_STRATEGY=bidkv python -m bidkv.experiments.vllm.serve \
    --model meta-llama/Llama-3.1-8B-Instruct --enforce-eager --port 8000

Zero Dependencies

bidkv depends only on the Python standard library — no torch, numpy, vllm, or sglang.

Install

pip install -e .

# development mode
pip install -e ".[dev]"

Testing

python -m pytest tests/ -v

License

Apache-2.0