flyto-mlx 0.4.0

Flyto MLX — Apple Silicon LLM server with audio chat, DFlash, and Chinese model presets (based on oMLX)

Flyto MLX

Apple Silicon LLM 服务器 · Audio chat · DFlash 双引擎 · 中文模型预设
Based on oMLX by @jundot.

---

中文 | English

简介

Flyto MLX 是面向中国 Mac 用户与国产模型生态优化的 Apple Silicon 本地 LLM 服务器，基于 @jundot/oMLX fork。在保留 oMLX 全部上游能力（OpenAI 兼容 API、多模型 LRU 调度、KV 分页缓存、Mac menubar GUI）的基础上，加入了上游尚未合并/未支持的功能：

能力	说明
Gemma 4 audio chat	OpenAI input_audio content type 端到端支持，调用 gemma4-e2b / gemma4-e4b 直接听音频回答（不是 ASR 替代，是端到端 audio understanding）
DFlash 双引擎 (Path A)	Qwen / Gemma 4 双 backend，drafter co-loaded 优化
Tahoe 兼容	macOS 26 NSStatusItem occlusion bit 修复
上游已修但未发版的 backport	tokenizer lm_head、TokenBuffer cache hit seed、health-check Session 复用 等 5 处
中文模型预设	Qwen 3.5 MoE/Dense / DeepSeek V4 / Gemma 4 / 等 alias 即装即用
Gitee 镜像 + ModelScope 模型源	国内 access 优化

安装

# pip
pip install flyto-mlx

# 启动 server（CLI 兼容上游 omlx，主名为 fmlx）
fmlx serve --port 8000
# 或
omlx serve --port 8000     # alias，与上游兼容

DMG / brew tap 后续随 release 提供。

快速试 audio chat

# 假设 server 已起在 :8000，API key 设为 mykey
python3 <<'PY'
import base64, requests, json
with open("recording.wav","rb") as f:
    b64 = base64.b64encode(f.read()).decode()
r = requests.post(
    "http://localhost:8000/v1/chat/completions",
    headers={"Authorization": "Bearer mykey"},
    json={
        "model": "gemma4-e2b",
        "max_tokens": 400,
        "temperature": 0.3,
        "messages": [{"role": "user", "content": [
            {"type": "text", "text": "总结这段电话的关键信息"},
            {"type": "input_audio", "input_audio": {"data": b64, "format": "wav"}}
        ]}]
    },
)
print(r.json()["choices"][0]["message"]["content"])
PY

跟上游 oMLX 的关系

Flyto MLX 是 oMLX 的下游 fork，遵循 Apache 2.0。我们定期 cherry-pick 上游 bug fix 与新模型支持，但不再向上游 PR 自家 feature（audio chat、DFlash 等）。如果你只想要纯上游体验，请用 @jundot/oMLX。

详细 attribution 与版权声明见 NOTICE 与 LICENSE。

License

Apache License 2.0. Based on oMLX by @jundot. 详见 LICENSE 与 NOTICE。

---

English

Flyto MLX is a fork of @jundot/oMLX optimized for the Chinese Mac LLM community and sovereign-AI model ecosystem (Qwen, DeepSeek, Gemma 4). It preserves all upstream oMLX capabilities (OpenAI-compatible API, multi-model LRU scheduling, KV paged cache, menubar GUI) and adds:

• Audio chat via OpenAI input_audio — end-to-end Gemma 4 nano audio LLM through /v1/chat/completions
• DFlash Path A double-engine — Qwen and Gemma 4 backends with optimized drafter co-loading
• macOS 26 Tahoe compatibility — NSStatusItem occlusion bit fix
• 5 upstream-fixed-but-unreleased patches backported — tokenizer lm_head, TokenBuffer cache hit seed, health-check session reuse, and more
• Chinese model presets — Qwen 3.5 MoE/Dense, DeepSeek V4, Gemma 4 aliases ready out of the box
• Gitee mirror + ModelScope model registry — for users in mainland China

Install: pip install flyto-mlx. CLI: fmlx serve (or omlx serve alias for upstream compatibility).

We periodically cherry-pick upstream fixes. We do not upstream our own features back. For pure upstream behaviour, please use @jundot/oMLX directly.

License

Apache 2.0. See LICENSE and NOTICE.