Docs	Website	Models on Hugging Face	Quickstart	Citation

kLLM

kLLM pulls and runs models on SGLang and vLLM, with a workflow in the manner of Ollama. It can also seal a model under a key. A sealed model keeps its tensor names, shapes and data types, so the stock engine serves it unmodified. With the right key, it produces the original model's output; with a wrong key, the text is meaningless and the image is noise.

MiniMax H3: the same prompt with the key (left) and without the key (right). Without the key, the prompt reaches the model as garbled tokens and the video shows an unrelated scene. The full videos are on the kLLM website.

Quickstart

Install

kLLM 0.1.0 is available on PyPI as anet-kllm. It requires Linux, an NVIDIA GPU and Python 3.10 or later.

pip install 'anet-kllm[seal,serve]'     # SGLang; use 'anet-kllm[seal,vllm]' for vLLM
kllm doctor                        # checks the driver, CUDA, disk space and engine packages

Alternatively, install from source:

git clone https://github.com/ANetResearch/kLLM && cd kLLM
pip install -e '.[seal,serve]'     # SGLang; use '.[seal,vllm]' for vLLM

Run a catalog model

Catalog models are public models sealed so that anyone can try kLLM. Everyone who submits a model's access form receives the same key, so a catalog model is a demonstration, not a private model.

1. Open the model's Hugging Face repository, such as ANetResearch/Qwen3-8B-kllm. Sign in and submit the access form. Approval is immediate.
2. Download the model and its key, kllm.key, which is included in the repository and requires no passphrase.
3. Run or serve the model with the key.

hf auth login                                          # or set HF_TOKEN
hf download ANetResearch/Qwen3-8B-kllm --local-dir Qwen3-8B-kllm
kllm serve Qwen3-8B-kllm --key Qwen3-8B-kllm/kllm.key  # OpenAI-compatible API on port 8000; add --engine vllm for vLLM

Alternatively, pull and run the model through the local store:

kllm pull qwen3:8b-sealed                              # or: kllm pull hf://ANetResearch/Qwen3-8B-kllm
kllm run qwen3:8b-sealed                               # uses the kllm.key pulled with the model

Seal your own model

To keep a model private, seal it under a new key:

kllm key new mykey                                     # prompts for a passphrase
kllm pull qwen3:8b
kllm create qwen3-mine --from qwen3:8b --key mykey --transform perm
kllm run qwen3-mine --key mykey

For a local Hugging Face checkpoint directory, create a key file with kllm keygen -o my.key, seal the checkpoint with kllm seal models/Qwen3-8B -o models/Qwen3-8B-sealed --key my.key, and serve it with kllm serve models/Qwen3-8B-sealed --key my.key.

kllm run qwen3:8b runs a library model without sealing it. kllm ps lists running models; kllm stop stops one.

Catalog

The catalog contains 19 sealed models at huggingface.co/ANetResearch. Each is sealed from a public base model and keeps that model's license. kllm search sealed lists the models and their library tags.

Model	Base model	Modality	Size
Qwen3-0.6B-kllm	Qwen/Qwen3-0.6B	Text	1.5 GB
Qwen3-1.7B-kllm	Qwen/Qwen3-1.7B	Text	4.1 GB
Qwen3-4B-kllm	Qwen/Qwen3-4B	Text	8.8 GB
Qwen3-4B-Instruct-2507-kllm	Qwen/Qwen3-4B-Instruct-2507	Text	8.8 GB
Qwen3-8B-kllm	Qwen/Qwen3-8B	Text	16.4 GB
Qwen3-14B-kllm	Qwen/Qwen3-14B	Text	29.5 GB
Qwen3-32B-kllm	Qwen/Qwen3-32B	Text	65.5 GB
Qwen3-30B-A3B-Instruct-2507-kllm	Qwen/Qwen3-30B-A3B-Instruct-2507	Text, mixture of experts	61.1 GB
Qwen3-30B-A3B-Instruct-2507-FP8-kllm	Qwen/Qwen3-30B-A3B-Instruct-2507-FP8	Text, mixture of experts, FP8	31.2 GB
Qwen3-Coder-30B-A3B-Instruct-FP8-kllm	Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8	Text, mixture of experts, FP8	31.2 GB
Qwen2.5-7B-Instruct-kllm	Qwen/Qwen2.5-7B-Instruct	Text	15.2 GB
Qwen2.5-Coder-7B-Instruct-kllm	Qwen/Qwen2.5-Coder-7B-Instruct	Text	15.2 GB
DeepSeek-R1-Distill-Qwen-7B-kllm	deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	Text	15.2 GB
DeepSeek-V4-Flash-kllm	deepseek-ai/DeepSeek-V4-Flash	Text, mixture of experts	159.6 GB
DeepSeek-V4-Pro-kllm	deepseek-ai/DeepSeek-V4-Pro	Text, mixture of experts	864.7 GB
GLM-5.3-kllm	zai-org/GLM-5.3	Text, mixture of experts	755.7 GB
GLM-5.3-Flash-kllm	zai-org/GLM-5.3-Flash	Text and image input, mixture of experts	328.4 GB
Z-Image-kllm	Tongyi-MAI/Z-Image	Image generation	20.5 GB
Z-Image-Turbo-kllm	Tongyi-MAI/Z-Image-Turbo	Image generation	32.9 GB

Supported architectures

Any model that the selected engine supports can be served unsealed. Sealing supports the architectures below.

Family	Architecture	Modality	Transforms	Engines
Qwen3	Qwen3ForCausalLM	Text	orth, perm (FP8: perm)	SGLang, vLLM
Qwen3-MoE	Qwen3MoeForCausalLM	Text, mixture of experts	orth, perm (FP8: perm)	SGLang, vLLM
Qwen2 and Qwen2.5, including DeepSeek-R1-Distill-Qwen	Qwen2ForCausalLM	Text	orth, perm	SGLang, vLLM
GLM-5 (5, 5.1, 5.2, 5.3)	GlmMoeDsaForCausalLM	Text, mixture of experts	perm	SGLang, vLLM
DeepSeek V4 (Flash, Pro)	DeepseekV4ForCausalLM	Text, mixture of experts	perm	SGLang, vLLM
GLM-5.3-Flash	Glm5NextForConditionalGeneration	Text and image input, mixture of experts	perm	SGLang, vLLM
MiniMax H3	FL2VA	Video with audio	orth, perm	SGLang
Z-Image	ZImagePipeline	Image generation	orth, perm	kLLM latent server

To request another model, open a model request or run kllm request MODEL_ID.

Editions

kLLM Community is the open-source edition in this repository. For every model, it keeps the efficiency of the stock engine. kLLM Enterprise is a separate runtime written in Rust and Go, with dedicated optimizations for selected models and workloads, such as MiniMax H3 and agent-team workloads. These optimizations are not open source. Contact the team through kllm.agentnetwork.org.cn/enterprise or at hi@anet0.com.

Documentation

The documentation covers installation, the catalog, sealing, serving, the command-line reference and guides for each model family. For contributions, see CONTRIBUTING.md.

Citation

@inproceedings{yuan2025scx,
  title     = {{SCX}: Stateless {KV}-Cache Encoding for Cloud-Scale Confidential Transformer Serving},
  author    = {Yuan, Mu and Zhang, Lan and Zeng, Liekang and Jiang, Siyang and Yang, Bufang and Duan, Di and Xing, Guoliang},
  booktitle = {Proceedings of the ACM SIGCOMM 2025 Conference},
  year      = {2025},
  doi       = {10.1145/3718958.3750509}
}

@inproceedings{yuan2026stip,
  title     = {{STIP}: Three-Party Privacy-Preserving and Lossless Inference for Large Transformers in Production},
  author    = {Yuan, Mu and Zhang, Lan and Cheng, Yihang and Song, Miao-Hui and Xing, Guoliang and Li, Xiang-Yang},
  booktitle = {Network and Distributed System Security (NDSS) Symposium 2026},
  address   = {San Diego, CA},
  year      = {2026},
  doi       = {10.14722/ndss.2026.230035}
}

License

kLLM is licensed under Apache-2.0. Each catalog model retains its base model's license.

---

kLLM is a project of Agent Network.