thinctx

Inference runtime and OpenAI-compatible server for the ThinCtx models — compact language models whose context memory on the GPU stays small and grows 20 to 30 times more slowly than the original model's KV cache, while the full context is held in system memory.

pip install thinctx

thinctx serve thinletter/ThinCtx-Qwen3.5-2B --port 8000 --auth user:pass   # OpenAI-compatible endpoint
thinctx chat  thinletter/ThinCtx-Qwen3.5-2B                                # terminal REPL

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B")   # a hub repo id or a local released folder

The server implements the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking); point any OpenAI client at http://host:8000/v1.

Requirements

Linux x86_64, CPython 3.12, an NVIDIA GPU, PyTorch 2.11. Installs torch, transformers, fla-core and safetensors. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build first: pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128. causal_conv1d is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence: info@thinletter.io. The models are derivatives of Qwen3.5 (Apache-2.0); each model repo ships Qwen's license and a NOTICE.