Quick start

pip install "hfl[llama,mlx]"        # the MLX part installs only on Apple Silicon

hfl run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M

The model is downloaded the first time and reused from disk after that. To serve it instead:

hfl serve --model hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M

Any OpenAI, Ollama or Anthropic client can now talk to http://localhost:11434.

Why HFL

• The whole Hub, not one file format. GGUF repos run through llama.cpp, MLX builds run natively on Apple Silicon, and safetensors checkpoints are converted and quantized for you on pull. Copy a repo name from the Hub and run it.
• Browse the Hub from your terminal. hfl search pages through the live Hub with sizes, downloads and formats at a glance; filter by GGUF or by size, press a number and the model is pulled, license check included.
• Yours alone. No account, no sign-in, no cloud service behind it. On its own HFL talks to one server, the Hugging Face Hub, to fetch weights; its web-search endpoints reach the web only when a client calls them, and a test pins every host the code can reach. With no network, everything you have already pulled keeps working.
• Plugs into what you already use. OpenAI (chat, completions, embeddings, Responses), Ollama and Anthropic Messages APIs on one port, with structured tool calling for Qwen, Llama 3 and Mistral families and JSON-schema outputs. Existing OLLAMA_* settings such as OLLAMA_HOST are honoured.
• As many models as your memory holds. Before every load HFL estimates what the model will take (weights + KV cache) and keeps the machine under a memory budget you set: models load side by side while they fit, idle ones make room, one in use is never pulled out from under a request, and one that cannot fit is refused with the numbers — before anything is unloaded. GPU-aware on NVIDIA.
• Knows the Hub. Find models that fit your hardware (hfl recommend), pick the best community quant for your machine (hfl pull-smart), size Mixture-of-Experts models by their total parameters, check licenses before downloading and keep a provenance record of every pull.

Install

Use it

Search and pick from the terminal

hfl search qwen3                          # everything matching, most downloaded first
hfl search qwen3 --gguf --max-params 8    # GGUF only, 8B or smaller — searched across the whole Hub
hfl search llama --sort likes             # or: downloads (default), created

Each page lists up to ten models with their size, downloads, likes, format and task. Press 0–9 to pull one (you confirm, and its license is checked first), SPACE for the next page, p for the previous one, q to leave. Sizes are total parameters, so Mixture-of-Experts models are not shown smaller than they are.

Chat

hfl run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M   # from the Hub, pulled on first use
hfl run hf.co/mlx-community/Qwen2.5-0.5B-Instruct-4bit       # an MLX build on Apple Silicon
hfl run llama70b --system "You are a Python expert"          # a local name or alias
hfl run llama70b --session work                              # resume and save a conversation

The hf.co/ prefix is optional. :Q4_K_M picks a quantization, @<ref> pins a branch, tag or commit.

Pull, search and manage

hfl pull meta-llama/Llama-3.3-70B-Instruct                 # Q4_K_M by default
hfl pull meta-llama/Llama-3.3-70B-Instruct --quantize Q5_K_M --alias llama70b
hfl pull meta-llama/Llama-3.3-70B-Instruct@a1b2c3d          # reproducible: pinned revision

hfl list                                # what is on this machine
hfl inspect llama70b                    # details and license
hfl rm llama70b

Find the right model

hfl recommend                           # top models that fit THIS machine's RAM/VRAM
hfl discover --family qwen              # filter the live Hub; marks what you already have
hfl pull-smart Qwen/Qwen3-30B-A3B       # best community variant for your hardware
hfl verify <model>                      # tokenizer, chat template, smoke generation, tools
hfl bench <model>                       # time to first token, tokens/s, p50/p95

See docs/hub-native-features.md for every option.

Several models at once

HFL keeps every model that fits under HFL_MEMORY_BUDGET — the share of total RAM the machine may have in use after a load, other programs included (default 85%). Each load reports the numbers before it happens:

Memory: 65.3 of 128.0 GB in use (51%). qwen3-14b needs ~9.0 GB → after loading, 74.3 GB in use (58%); budget 85%.

• fits → it loads next to the models already resident;
• does not fit → idle models are unloaded, least recently used first;
• the room is held by models answering requests → the load waits for them;
• cannot fit even alone → refused with the numbers (HTTP 507), nothing unloaded.

With an NVIDIA GPU the model must also fit the card (read through nvidia-smi). Idle models unload after keep_alive (default 5m, renewed on every use). hfl ps and GET /api/ps show what is loaded and how much room is left.

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3-32b-q4_k_m",
  "stream": false,
  "messages": [{"role": "user", "content": "Save Hello at topics/hello.md"}],
  "tools": [{"type": "function", "function": {
    "name": "write_wiki", "description": "Create or overwrite a wiki article",
    "parameters": {"type": "object",
      "properties": {"path": {"type": "string"}, "content": {"type": "string"}},
      "required": ["path", "content"]}}}]
}'

{"message": {"role": "assistant", "content": "",
  "tool_calls": [{"function": {"name": "write_wiki",
    "arguments": {"path": "topics/hello.md", "content": "Hello"}}}]},
 "done": true}

pip install "hfl[tts,audio]"
hfl pull suno/bark-small --alias bark
hfl tts bark "Hello, this is a test." -o hello.wav     # write a file (wav, mp3, ogg)
hfl speak bark "Hola mundo" --lang es --speed 0.9      # play it

# OpenAI-compatible
curl http://localhost:11434/v1/audio/speech -H "Content-Type: application/json" \
  -d '{"model": "bark", "input": "Hello world", "voice": "alloy"}' --output speech.wav

# Native: language, speed, sample rate and format (wav, mp3, ogg)
curl http://localhost:11434/api/tts -H "Content-Type: application/json" \
  -d '{"model": "bark", "text": "Hola mundo", "language": "es"}' --output speech.wav

Connect your tools

The server listens on http://localhost:11434 and speaks three APIs.

OpenAI — /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")
reply = client.chat.completions.create(
    model="hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M",
    messages=[{"role": "user", "content": "Explain quantum computing in one paragraph"}],
)
print(reply.choices[0].message.content)

Ollama — /api/chat, /api/generate, /api/embed, /api/tags, /api/ps, /api/pull and more

curl http://localhost:11434/api/chat -d '{"model": "llama70b",
  "messages": [{"role": "user", "content": "Hello!"}]}'

Anthropic — /v1/messages

curl http://localhost:11434/v1/messages -H "Content-Type: application/json" \
  -d '{"model": "llama70b", "max_tokens": 256,
       "messages": [{"role": "user", "content": "Hello!"}]}'

Models can be named by their local name, an alias, or the Hub reference they were pulled from (hf.co/org/repo:QUANT). The server never downloads on its own: hfl pull or hfl serve --model <reference> does.

Reference

hfl pull / run ──▶ Hugging Face Hub ──▶ ~/.hfl/models ──▶ GGUF? ── yes ──▶ llama.cpp
                   (search, download,                  MLX build (Apple Silicon) ──▶ MLX
                    license check)                     safetensors ── convert + quantize ──▶ GGUF

hfl serve ──▶ OpenAI · Ollama · Anthropic APIs ──▶ memory-budgeted model set ──▶ one inference at a time

Documentation

• Hub-native features — discover, recommend, pull-smart, verify, bench and more
• Environment variables — every setting and its default
• Apple Silicon and Docker clients
• Architecture guide
• Changelog

Status: beta — 4,000+ tests at ~90% coverage. Windows builds and installers are published, but Windows is less tested than macOS and Linux.

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md.

git clone https://github.com/ggalancs/hfl && cd hfl
pip install -e ".[dev]"
bash scripts/ci-local.sh        # lint, types and the full test suite, as CI runs them

If HFL saves you a download–convert–quantize afternoon, a ⭐ helps other people find it.

Legal notices

Model licenses. Models keep their own licenses (Llama, Gemma, OpenRAIL, CC-BY-NC, …) and you are responsible for complying with them. HFL shows a model's license before downloading it, stores it with the model and records the pull's provenance — see hfl inspect <model>. Common restrictions include non-commercial use only (CC-BY-NC, MRL), attribution (Llama, Gemma) and usage restrictions (OpenRAIL).

Export compliance. HFL only downloads publicly available open-weight models from the Hugging Face Hub and does not facilitate access to closed-weight or export-controlled weights. Users are responsible for complying with the export regulations of their jurisdiction.

Disclaimer. AI models may generate inaccurate, biased or inappropriate content. Users are solely responsible for evaluating and using model outputs. See DISCLAIMER.md.

Trademarks. "OpenAI" is a trademark of OpenAI, Inc. "Ollama" is a trademark of Ollama, Inc. "Anthropic" is a trademark of Anthropic, PBC. "Hugging Face" and the Hugging Face logo are trademarks of Hugging Face, Inc. These marks are used for identification only. HFL is an independent project, not affiliated with, endorsed by or officially connected to any of these companies. References to their services describe technical interoperability only.

License

HFL is licensed under the Apache License 2.0 — you may use, modify, distribute and sell it, including commercially, as long as you keep the copyright and license notices. See LICENSE and NOTICE.

HFL ships responsible-use safeguards: license checking, AI disclaimers, provenance tracking, privacy protections and respect for gated models. Apache-2.0 does not require you to keep them; as a project norm we ask that redistributions leave them active. See DISCLAIMER.md, PRIVACY.md and NOTICE-EU-AI-ACT.md.

HFL's license covers HFL itself, not the models you download.