Skip to main content
FreeToken

| Download | Paper | Developer Slack | Community Discord | Community WeChat |

Unlock datacenter-class intelligence on the hardware you already own — Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.

About

FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform. Its core features include:

  • Fast Edge-Native Runtime: Provides efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy), full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.
  • Semantic-Aware Caching: Features semantic anchor checkpoints for recurrent state and KV caches, allowing agentic context edits (e.g., tool calls, thinking blocks) to avoid redundant context recomputation.
  • Elastic Memory Management: Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading.
  • Broad MoE & Ecosystem Support: Supports frontier open-weight MoE models (e.g., DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2) across various parameter scales and quantization formats (e.g., MXFP4, NVFP4, FP8, BF16, GGUF), with Anthropic/OpenAI-compatible APIs for seamless integration with real-world coding and tool-calling agents (e.g., Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness).
  • Diverse Consumer Hardware: Scales across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.

Getting Started

Desktop app

Download FreeToken for Windows or Linux at flashml.ai. It sets the engine up for you and gives you a GUI for running models, chatting, and tuning the engine.

FreeToken Desktop

CLI

Install FreeToken with uv (recommended) or pip:

uv pip install "freetoken[accel]"

Or build from source:

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

For More details:

Citation

If you use FreeToken for your research, please cite our paper:

@article{yang2026freetoken,
  title={FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution},
  author={Yang, Shuo and Fan, Xiaoze and Pan, Melissa and Xi, Haocheng and Wang, Zhe and Sun, Shanlin and Keutzer, Kurt and Han, Song and Zaharia, Matei and Xu, Chenfeng and Stoica, Ion},
  journal={arXiv preprint arXiv:2608.16157},
  year={2026}
}

Acknowledgment

FreeToken was deeply inspired by mini-sglang, and learned the design and reused code from the following projects: SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.

License

Apache License 2.0.

Metadata

Release files for freetoken 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for freetoken 0.2.2
File Actions
freetoken-0.2.2-cp313-cp313-manylinux_2_27_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.2-cp313-cp313-manylinux_2_27_x86_64.whl
freetoken-0.2.2-cp312-cp312-manylinux_2_27_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.2-cp312-cp312-manylinux_2_27_x86_64.whl
freetoken-0.2.2-cp311-cp311-manylinux_2_27_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.2-cp311-cp311-manylinux_2_27_x86_64.whl
freetoken-0.2.2-cp310-cp310-manylinux_2_27_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.2-cp310-cp310-manylinux_2_27_x86_64.whl

Total release size: 19.5 MB

Release files / freetoken-0.2.2-cp313-cp313-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.2-cp313-cp313-manylinux_2_27_x86_64.whl
Size 4.9 MB
Tags CPython 3.13 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
b513c4eded69aafada10aab38dbd279921875fcfde379640b59b866d8eb4162d
BLAKE2b-256 checksum
How to use checksums
68973b99ba4e272d38c459f4ba7672ce86ce82c4a3f204e15605a4923a94e2a1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / freetoken-0.2.2-cp312-cp312-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.2-cp312-cp312-manylinux_2_27_x86_64.whl
Size 4.9 MB
Tags CPython 3.12 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
e8e735cd0929110c131617f5585e61fe609c4c4e081537bd063428c02b1f4130
BLAKE2b-256 checksum
How to use checksums
ef0b47241003ec89fe9ae6d0c025af719685a297fbd7df88f85fae23171c2276
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / freetoken-0.2.2-cp311-cp311-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.2-cp311-cp311-manylinux_2_27_x86_64.whl
Size 4.9 MB
Tags CPython 3.11 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
921ddd2a8fdb386481773a5db9374cab8bfd5724af458b3ca797e6ddf0904593
BLAKE2b-256 checksum
How to use checksums
663305aafdbec8d3d4410e71c7025b3af538561ce7aef03c2c038285dc1b9a3d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / freetoken-0.2.2-cp310-cp310-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.2-cp310-cp310-manylinux_2_27_x86_64.whl
Size 4.9 MB
Tags CPython 3.10 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
a0baa7b884882ed130f0f56f1cebe9144b57c38c6fdb371f3b80da8c1ee5c0c4
BLAKE2b-256 checksum
How to use checksums
c9b0a428b5148822d2647179fce5a4fb97bd8d7836834b6eeb3fe2e81a5d50f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for freetoken 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page