TLLM

Quickstart | Disk bottleneck solved | Configurations | MacOS | Example notebooks | FAQ

TLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) — the largest open-source model released to date — on under 4GB, because sparse MoE models stream one expert at a time rather than a whole layer.

TLLM fixes the biggest real-world limitation of layer-streaming inference: the per-token disk re-read (see Disk bottleneck solved).

Disk bottleneck solved

Layer-streaming inference reads one layer at a time from disk to GPU and evicts it immediately after the layer runs. That saves VRAM, but it means every single generated token re-reads essentially the entire model from disk. For a 70B model (~140GB in bf16) that is 0.5–2 tokens/second on a 4GB GPU versus 15–30 with the model in VRAM — and for Kimi K3 some users report several minutes per token. It runs, but it is a tool for offline batch processing, not interactive chat.

The real bottleneck isn't VRAM, it's disk. TLLM adds a layered weight cache that breaks this:

• On the first token, layers are read from disk once (as before).
• Layers that fit are then kept resident in VRAM; the rest spill to pinned host RAM; only what exceeds both is re-read from disk on later tokens.
• So the disk read becomes a one-time cost instead of a per-token cost. Subsequent tokens hit the cache, and a model that fits in VRAM approaches full-VRAM throughput.

The cache is on by default and degrades gracefully: if VRAM or RAM runs low, layers fall back to the original streaming path, so TLLM never uses more memory than uncached streaming on the same hardware. See the cache_layers, vram_cache_reserve_gb, and pinned_cache_gb options in Configurations.

Updates

[2026/08] Disk bottleneck fix: layered VRAM + pinned-RAM weight cache. Layers are read from disk once and reused across tokens instead of re-read on every token. On by default, graceful degradation.

[2026/08] Qwen3.8-27B support: Qwen's new dense VL (Gated DeltaNet + Gated Attention, native vision) runs in 3.33GB of VRAM, measured end to end on one RTX 3090. Needs transformers 5.8+.

[2026/07] Kimi K3 (2.8T) support: the largest open-source model runs on a single card in 3.72GB of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: pip install compressed-tensors flash-attn (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and transformers 4.56.x, as its remote code does not load on 5.x.

[2026/06] v3.0: FP8 model support + the latest models. Run DeepSeek-V3 (671B) on ~12GB and Qwen3-235B on ~3GB, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single AutoModel.

Table of Contents

• Disk bottleneck solved
• Quick start
• Model Compression
• Configurations
• Run on MacOS
• Example notebooks
• Supported Models
• Acknowledgement
• FAQ

Quickstart

1. Install package

First, install the pytllm pip package.

pip install pytllm

2. Inference

Then, initialize the model via AutoModel, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model.

(You can also specify the path to save the splitted layered model through layer_shards_saving_path when init the model.

from pytllm import AutoModel

MAX_LENGTH = 128
# just pass a hugging face repo id — works with almost any popular model:
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3.8-27B")          # 27B dense VL, 3.33GB
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")     # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, runs in ~12GB

# or use a model's local path...
#model = AutoModel.from_pretrained("/home/ubuntu/.cache/huggingface/hub/models--Qwen--Qwen3-32B/snapshots/...")

input_text = [
        'What is the capital of United States?',
        #'I like',
    ]

input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])

print(output)

Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory.

Model Compression - 3x Inference Speed Up!

We support model compression based on block-wise quantization-based model compression. Which can further speed up the inference speed for up to 3x , with almost ignorable accuracy loss! (see more performance evaluation and why we use block-wise quantization in this paper)

How to enable model compression speed up:

• Step 1. make sure you have bitsandbytes installed by pip install -U bitsandbytes
• Step 2. make sure pytllm verion later than 2.0.0: pip install -U pytllm
• Step 3. when initialize the model, passing the argument compression ('4bit' or '8bit'):

model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
                     compression='4bit' # specify '8bit' for 8-bit block-wise quantization
                    )

What are the differences between model compression and quantization?

Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs.

While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy.

Configurations

When initialize the model, we support the following configurations:

• compression: supported options: 4bit, 8bit for 4-bit or 8-bit block-wise quantization, or by default None for no compression
• profiling_mode: supported options: True to output time consumptions or by default False
• layer_shards_saving_path: optionally another path to save the splitted model
• hf_token: huggingface token can be provided here if downloading gated models like: meta-llama/Llama-2-7b-hf
• prefetching: prefetching to overlap the model loading and compute. By default, turned on.
• delete_original: if you don't have too much disk space, you can set delete_original to true to delete the original downloaded hugging face model, only keep the transformed one to save half of the disk space.

Layer cache (disk bottleneck fix)

These control the layered weight cache that stops TLLM from re-reading the model from disk on every token (see Disk bottleneck solved):

• cache_layers: True (default). Keep streamed layers resident in VRAM, spilling to pinned RAM, and only re-read from disk what exceeds both. Set to False to restore uncached per-token streaming behaviour.
• vram_cache_reserve_gb: amount of VRAM to leave free for activations and the KV cache rather than filling it with cached weights. Default 2. Raise it if you see OOMs during generation; lower it to cache more layers.
• pinned_cache_gb: how much page-locked host RAM to use as a second cache tier before falling back to disk. Default uses a fraction of free RAM. Pinned RAM copies to the GPU faster than a disk read, so the spill tier is still much cheaper than re-reading every token.

The cache degrades gracefully: if admitting a layer would exceed the available VRAM or RAM, that layer falls back to the original stream-from-disk path, so peak memory never exceeds uncached streaming on the same hardware.

MacOS

Just install pytllm and run the code the same as on linux. See more in Quick Start.

• make sure you installed mlx and torch
• you probably need to install python native see more here
• only Apple silicon is supported

Example python notebook

Example Python Notebook

Example colabs here:

See the example notebook covering ChatGLM, QWen, Baichuan, Mistral and more.

example of other models (ChatGLM, QWen, Baichuan, Mistral, etc):

from pytllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("THUDM/chatglm3-6b-base")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=True)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=5,
    use_cache= True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])

from pytllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen-7B")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=5,
    use_cache=True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])

from pytllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("baichuan-inc/Baichuan2-7B-Base")
#model = AutoModel.from_pretrained("internlm/internlm-20b")
#model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
input_text = ['What is the capital of China?',]
input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH)
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=5,
    use_cache=True,
    return_dict_in_generate=True)
model.tokenizer.decode(generation_output.sequences[0])

Supported Models

TLLM works out of the box with virtually every popular open LLM — just pass its Hugging Face ID to AutoModel.from_pretrained(...). That covers all the major families:

Llama (2 / 3 / 3.1 / 3.3 / 4) · Qwen (1 / 2 / 2.5 / 3 / 3.5 / 3.8, including MoE, FP8, and native VL) · DeepSeek (V2 / V3 / R1) · Mistral & Mixtral · Phi · Gemma · ChatGLM · Baichuan · InternLM · Yi · Kimi K3 — and most new models the day they're released.

Tiny GPU, huge models

The trick: TLLM only ever keeps one layer on the GPU at a time, so the VRAM you need depends on the model's layer size — not its total size. That's how a 671B model fits on a hobbyist card:

Same one line of code for all of them — no special setup.

Acknowledgement

A lot of the original code is based on SimJeg's great work in the Kaggle exam competition. Big shoutout to SimJeg:

GitHub account @SimJeg, the code on Kaggle, the associated discussion.

FAQ

1. MetadataIncompleteBuffer

safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

If you run into this error, most possible cause is you run out of disk space. The process of splitting model is very disk-consuming. See this. You may need to extend your disk space, clear huggingface .cache and rerun.

2. ValueError: max() arg is an empty sequence

Most likely you are loading QWen or ChatGLM model with Llama2 class. Try the following:

For QWen model:

from pytllm import AutoModel #<----- instead of TLLMLlama2
AutoModel.from_pretrained(...)

For ChatGLM model:

from pytllm import AutoModel #<----- instead of TLLMLlama2
AutoModel.from_pretrained(...)

3. 401 Client Error....Repo model ... is gated.

Some models are gated models, needs huggingface api token. You can provide hf_token:

model = AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", #hf_token='HF_API_TOKEN')

4. ValueError: Asking to pad but the tokenizer does not have a padding token.

Some model's tokenizer doesn't have padding token, so you can set a padding token or simply turn the padding config off:

input_tokens = model.tokenizer(input_text,
   return_tensors="pt",
   return_attention_mask=False,
   truncation=True,
   max_length=MAX_LENGTH,
   padding=False  #<-----------   turn off padding
)

Citing TLLM

If you find TLLM useful in your research and wish to cite it, please use the following BibTex entry:

@software{tllm2026,
  author = {TLLM},
  title = {TLLM: streaming large language model inference with a layered weight cache},
  version = {1.0.0},
  year = {2026},
}

Contribution

Welcomed contributions, ideas and discussions!