thinctx

Inference runtime and OpenAI-compatible server for the ThinCtx models — compact language models whose context memory on the GPU stays small and constant while the full context is held in system memory.

pip install thinctx

thinctx serve thinletter/ThinCtx-Qwen3.5-2B --port 8000 --auth user:pass   # OpenAI-compatible endpoint
thinctx chat  thinletter/ThinCtx-Qwen3.5-2B                                # terminal REPL

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B")   # a hub repo id or a local released folder

The server implements the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking); point any OpenAI client at http://host:8000/v1.

Requirements

Linux x86_64, CPython 3.12, an NVIDIA GPU with CUDA 12.8, PyTorch 2.11. Installs transformers, fla-core and safetensors; causal_conv1d is optional (a pure-torch fallback is built in).

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence: info@thinletter.io. The models are derivatives of Qwen3.5 (Apache-2.0); each model repo ships Qwen's license and a NOTICE.