thinctx

Inference runtime and OpenAI-compatible server for the ThinCtx models — compact language models whose context memory on the GPU stays small and grows 30 to 40 times more slowly than the original model's KV cache, while the full context is held in system memory.

Install and run

Platforms. Linux x86_64, or Windows 10/11 through WSL2, with an NVIDIA GPU; macOS is not supported. The default PyTorch 2.11 wheel needs a driver with CUDA 13 support (with an older one, see below). In WSL the Windows NVIDIA driver provides CUDA: do not install a Linux driver inside WSL. Triton compiles a small launcher on the first start and needs gcc: sudo apt install -y build-essential (Debian / Ubuntu) if gcc --version fails.

Linux / WSL:

curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env   # only if uv is missing
uv venv --managed-python --python 3.12 ~/thinctx && source ~/thinctx/bin/activate
uv pip install thinctx
thinctx chat thinletter/ThinCtx-Qwen3.5-2B-small

With an older driver, run uv pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128 before installing thinctx. Without uv: python3.12 -m venv ~/thinctx (on Debian / Ubuntu it needs the python3.12-venv package, and the python3.12-dev headers for Triton's launcher compile), then pip install thinctx.

Windows: in an admin PowerShell wsl --install -d Ubuntu-24.04, restart, open Ubuntu, then the Linux steps. Memory sharing across conversations is Linux-native only (WSL runs without it). WSL keeps its file cache in Windows RAM: add autoMemoryReclaim=dropcache under [experimental] in %UserProfile%\.wslconfig.

An OpenAI-compatible endpoint, or the engine from Python:

thinctx serve thinletter/ThinCtx-Qwen3.5-2B-small --port 8000 --auth user:pass

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B-small")   # a hub repo id or a local released folder

First start. The GPU kernels are compiled once, on the first run, and cached in ~/.triton/cache, so the first answer takes longer; later starts reuse the cache.

The server implements the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking); point any OpenAI client at http://host:8000/v1.

1.0.0 shares the memory of a common system prompt across conversations, moves idle conversations to system memory and back, and keeps its public interface stable (below). It runs every ThinCtx model, including the compact -small ones; pip install -U thinctx.

Compressed context

The full-precision releases (ThinCtx-Qwen3.5-0.8B, -2B, -4B) can hold the context in system memory compressed, with the context store of their -small release (1.0.2 and later):

thinctx chat thinletter/ThinCtx-Qwen3.5-4B --compress_context
thinctx serve thinletter/ThinCtx-Qwen3.5-4B --port 8000 --compress_context

engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-4B", compress_context=True)

The system memory for the context gets several times smaller; the GPU memory stays the same, and the weights stay the full-precision ones. thinctx downloads only the small store files of the -small release (a few MB), not its weights, and checks that they belong to the same model. For a local folder, pass the -small release with --compress_context_from REPO_OR_FOLDER (Python: compress_context_from=...); a sibling folder named <folder>-small is found on its own. Quality stays close to the full model; the measured numbers are on the model cards. The -small releases always run with the compressed store.

Memory sharing across conversations

Conversations that start with the same system prompt share its stored context in system memory and its cached part on the GPU. On the 2B, each additional conversation then costs about 180 MB of GPU memory and 30–60 MB of system memory, almost independent of the prompt length (8k–32k tokens); without sharing it costs 320–620 MB of GPU memory and 175–440 MB of system memory. On the 2B-small, an additional conversation costs 100–110 MB of GPU memory and 20–30 MB of system memory. A new conversation whose system prompt is already cached starts generating in 0.15–0.27 s instead of 1.3–6.9 s (8k–32k tokens, RTX 3090). With --warm_prompts, the first conversation starts that fast too. Idle conversations can be moved to system memory and resumed in about the same time as a live turn. Single-conversation decode speed changes by at most ~1.5 %. The system memory a conversation's context takes stays close to the context's size as it grows.

Memory limits (thinctx serve flags, defaults in brackets):

• --prefix_cache_mb (4096) and --prefix_gpu_mb (2048): system and GPU memory the prompt cache may hold; the least recently used prompts are dropped first.
• --prefix_cache_ttl (1800): seconds after which a cached prompt nobody reused is dropped (0: never).
• --max_sessions (the model's profile): conversations kept ready on the GPU; beyond it the least recently used idle one moves to system memory.
• --session_ttl (1800): seconds after which an idle conversation moves to system memory (0: never).
• --session_gpu_mb (0: no limit): GPU memory for all conversations together; beyond it idle ones move to system memory.
• --tenant_cache_mb / --tenant_gpu_mb (0: none): each API key's share of the prompt cache.

Prompt cache and forks

A conversation that starts with a system prompt the server has already seen starts without reprocessing it, and it shares that prompt's memory with every other conversation that starts the same way: the stored context in system memory and the system prompt's part on the GPU are held once. Conversations are named by the request's session field.

• System prompt. Conversations that start with the same system prompt (and tool list) process it once. Every later conversation starts from the cached copy, so its first token comes sooner, and only its own turns add memory. --warm_prompts FILE (a JSON list of {"system": ..., "tools": [...], "namespace": ...}) processes system prompts when the server starts, so even the first request is fast.
• Regenerate, edit, fork (opt-in: --prefix_turns N). The ends of the last N prompts of each conversation are saved too.
  • Sending the same request again regenerates the answer without reprocessing anything.
  • An edited last message resumes from the latest saved point it extends.
  • POST /fork {"session": "new", "from": "old", "turn": 2} starts a new conversation at the start of the second answer of another one. A chat request can do the same with "fork_from": {"session": "old", "turn": 2}.
• Isolation. With --api_keys FILE (one NAME:KEY per line, sent as Authorization: Bearer KEY), each key has its own conversations and its own cached prompts, so one tenant's prompt never speeds up another's. --prefix_share global reuses prompts across keys; --prefix_share off turns the prompt cache off. --tenant_cache_mb / --tenant_gpu_mb give each key its own share of the cache, so one key cannot push out another's prompts. GET /cache lists the cached prompts and POST /cache/clear drops them: a key sees and clears its own, the --auth user and the keys named in --admin_keys every key's.
• Memory. --prefix_cache_mb (default 4096) and --prefix_gpu_mb (default 2048) bound what the prompt cache holds; the least recently used entries are dropped first, and entries nobody reused for --prefix_cache_ttl seconds (default 1800) are dropped. Only prompts of at least 1024 tokens are saved (--prefix_min_tokens).
• Idle conversations. A conversation idle for --session_ttl seconds (default 1800), beyond --max_sessions conversations, or while all conversations together exceed --session_gpu_mb of GPU memory, moves to system memory; its next request continues it exactly where it was. System memory for a conversation's context grows with the context in small steps, so it stays close to the context's actual size.
• Results. A conversation resumed from a cached prompt, or moved to system memory and back, gives exactly the same tokens as one that never was.
  • Compared with a server that has no cache, the end of the system prompt adds one processing boundary. That changes the numbers only within bf16 rounding: in our tests the mean divergence was below 0.001 nats per token, and a greedy token can flip only where two candidates are nearly tied.
  • --prefix_align chunk saves at the processing block before the end instead. The results are then identical to a server without the cache, at the cost of reprocessing up to 2048 tokens of the prompt.

Stability

From 1.0.0 on, thinctx follows semantic versioning for its public interface:

• Python API: thinctx.load(...) and the engine it returns, ChatSession (including fork).
• Command line: thinctx serve and thinctx chat with the flags their --help documents.
• HTTP endpoints: /v1/chat/completions (the OpenAI chat-completions schema), /fork, /cache, /cache/clear, /info and /stats.
• Model formats: every released model format (thinctx-model/1, /2 and /3) stays loadable by later 1.x releases.

Anything else, including module internals and undocumented options, may change in any release. A part of the public interface is only removed after it has printed a deprecation warning for at least one minor release.

Requirements

Linux x86_64 or WSL2, CPython 3.12, an NVIDIA GPU, PyTorch 2.11, gcc (Triton's first start). Installs torch, transformers, fla-core and safetensors. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build first: pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128. causal_conv1d is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence: info@thinletter.io. The models are derivatives of Qwen3.5 (Apache-2.0); each model repo ships Qwen's license and a NOTICE.