Skip to main content
FreeToken

| Download | Paper | Developer Slack | Community Discord | Community WeChat |

Unlock datacenter-class intelligence on the hardware you already own — Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.

About

FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform. Its core features include:

  • Fast Edge-Native Runtime: Provides efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy), full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.
  • Semantic-Aware Caching: Features semantic anchor checkpoints for recurrent state and KV caches, allowing agentic context edits (e.g., tool calls, thinking blocks) to avoid redundant context recomputation.
  • Elastic Memory Management: Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading.
  • Broad MoE & Ecosystem Support: Supports frontier open-weight MoE models (e.g., DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2) across various parameter scales and quantization formats (e.g., MXFP4, NVFP4, FP8, BF16), with Anthropic/OpenAI-compatible APIs for seamless integration with real-world coding and tool-calling agents (e.g., Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness).
  • Diverse Consumer Hardware: Scales across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.

Getting Started

Desktop app

Download FreeToken for Windows or Linux at flashml.ai. It sets the engine up for you and gives you a GUI for running models, chatting, and tuning the engine.

FreeToken Desktop

CLI

Install FreeToken with uv (recommended) or pip:

uv pip install "freetoken[accel]"

Or build from source:

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

For More details:

Citation

If you use FreeToken for your research, please cite our paper:

@article{yang2026freetoken,
  title={FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution},
  author={Yang, Shuo and Fan, Xiaoze and Pan, Melissa and Xi, Haocheng and Wang, Zhe and Sun, Shanlin and Keutzer, Kurt and Han, Song and Zaharia, Matei and Xu, Chenfeng and Stoica, Ion},
  journal={arXiv preprint arXiv:2608.16157},
  year={2026}
}

Acknowledgment

FreeToken was deeply inspired by mini-sglang, and learned the design and reused code from the following projects: SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.

License

Apache License 2.0.

Metadata

Release files for freetoken 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for freetoken 0.2.1
File Actions
freetoken-0.2.1-cp313-cp313-manylinux_2_27_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.1-cp313-cp313-manylinux_2_27_x86_64.whl
freetoken-0.2.1-cp312-cp312-manylinux_2_27_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.1-cp312-cp312-manylinux_2_27_x86_64.whl
freetoken-0.2.1-cp311-cp311-manylinux_2_27_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.1-cp311-cp311-manylinux_2_27_x86_64.whl
freetoken-0.2.1-cp310-cp310-manylinux_2_27_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.27+ x86-64 Details for freetoken-0.2.1-cp310-cp310-manylinux_2_27_x86_64.whl

Total release size: 19.2 MB

Release files / freetoken-0.2.1-cp313-cp313-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.1-cp313-cp313-manylinux_2_27_x86_64.whl
Size 4.8 MB
Tags CPython 3.13 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
78797543260eaf6b02a34e00a3ebd69bb382d2148de5323c894cd175e169dd38
BLAKE2b-256 checksum
How to use checksums
675ff490751e514cc7282f8e5a351b3fa184fdaa96cf6545a475644f3841462d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / freetoken-0.2.1-cp312-cp312-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.1-cp312-cp312-manylinux_2_27_x86_64.whl
Size 4.8 MB
Tags CPython 3.12 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
b3bb49da1db367f69a6ab7eef0154c57330ed4c7a167ee9dfb8fbd798a3265fd
BLAKE2b-256 checksum
How to use checksums
c6c069969e838ba8cedb83827b5f037cb8f157ac284378e12641b744f72d3e88
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / freetoken-0.2.1-cp311-cp311-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.1-cp311-cp311-manylinux_2_27_x86_64.whl
Size 4.8 MB
Tags CPython 3.11 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
6a837cd00f177a071c5e41091c0143d51ee97d44ee8ca656a8ea79151bf2fa55
BLAKE2b-256 checksum
How to use checksums
23b70f8e36bf11cf6e23f4b89945be9258be0265e93d9bac18d3b2d77d38afe9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / freetoken-0.2.1-cp310-cp310-manylinux_2_27_x86_64.whl

Download URL freetoken-0.2.1-cp310-cp310-manylinux_2_27_x86_64.whl
Size 4.8 MB
Tags CPython 3.10 Linux glibc 2.27+ x86-64
SHA-256 checksum
How to use checksums
f76b4ff156da529bd2658117ae8b74bbba1878d889b8e12b10672cd371c95511
BLAKE2b-256 checksum
How to use checksums
e074cfa32dfdf9f8568f83ccf8fefeac43f2f03f328d485c208b8450751841f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for freetoken 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page