thinctx

Inference runtime and OpenAI-compatible server for the ThinCtx models — compact language models whose context memory on the GPU stays small and grows 20 to 30 times more slowly than the original model's KV cache, while the full context is held in system memory.

pip install thinctx

thinctx serve thinletter/ThinCtx-Qwen3.5-2B --port 8000 --auth user:pass   # OpenAI-compatible endpoint
thinctx chat  thinletter/ThinCtx-Qwen3.5-2B                                # terminal REPL

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B")   # a hub repo id or a local released folder

The server implements the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking); point any OpenAI client at http://host:8000/v1.

This release runs the compact -small models (smaller weights and a compressed context store in system memory, for smaller GPUs; they also load faster), adds a prompt cache to the server (below), and needs less GPU memory while reading very long prompts. The -small repos need this version; pip install -U thinctx.

Prompt cache and forks

A conversation that starts with a system prompt the server has already seen starts without reprocessing it. In this version each conversation keeps its own copy of the context in memory; sharing memory across conversations comes in 1.0.0. Conversations are named by the request's session field.

• System prompt. Conversations that start with the same system prompt (and tool list) process it once. Every later conversation starts from the cached copy, so its first token comes sooner.
• Regenerate, edit, fork (opt-in: --prefix_turns N). The ends of the last N prompts of each conversation are saved too. Each save copies the conversation's context into system memory during that turn: for the 2B, about 0.1 / 0.2 / 0.4 s and 0.2 / 0.5 / 1.8 GB at 8k / 32k / 128k tokens.
  • Sending the same request again regenerates the answer without reprocessing anything.
  • An edited last message resumes from the latest saved point it extends.
  • POST /fork {"session": "new", "from": "old", "turn": 2} starts a new conversation at the start of the second answer of another one. A chat request can do the same with "fork_from": {"session": "old", "turn": 2}.
• Isolation. With --api_keys FILE (one NAME:KEY per line, sent as Authorization: Bearer KEY), each key has its own conversations and its own cached prompts, so one tenant's prompt never speeds up another's. --prefix_share global reuses prompts across keys; --prefix_share off turns the prompt cache off.
• Memory. --prefix_cache_mb (default 4096) bounds the system memory the prompt cache uses, and the least recently used entries are dropped first. Only prompts of at least 1024 tokens are saved (--prefix_min_tokens).
• Results. A conversation resumed from a cached prompt gives exactly the same tokens as the conversation that cached it.
  • Compared with a server that has no cache, the end of the system prompt adds one processing boundary. That changes the numbers only within bf16 rounding: in our tests the mean divergence was below 0.001 nats per token, and a greedy token can flip only where two candidates are nearly tied.
  • --prefix_align chunk saves at the processing block before the end instead. The results are then identical to a server without the cache, at the cost of reprocessing up to 2048 tokens of the prompt.

Requirements

Linux x86_64, CPython 3.12, an NVIDIA GPU, PyTorch 2.11. Installs torch, transformers, fla-core and safetensors. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build first: pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128. causal_conv1d is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence: info@thinletter.io. The models are derivatives of Qwen3.5 (Apache-2.0); each model repo ships Qwen's license and a NOTICE.