llm-huggingface
Access Hugging Face models via the Inference API
Installation
Install this plugin in the same environment as LLM.
llm install llm-huggingface
Configuration
Configure the plugin by setting your Hugging Face API token:
llm keys set huggingface
<paste key here>
You can also set the API key by assigning it to the environment variable HUGGINGFACE_TOKEN.
Usage
The plugin automatically discovers and registers all available text-generation models from Hugging Face. Run a model using the hf/ prefix:
llm -m hf/meta-llama/Llama-3.2-3B-Instruct "Write a haiku about coding"
You can list all available Hugging Face models:
llm models | grep "^hf/"
Set a default model to avoid the -m option:
llm models default hf/mistralai/Mistral-7B-Instruct-v0.3
llm "Explain quantum computing in simple terms"
Features
Streaming Responses
The plugin supports streaming for real-time output:
llm -m hf/meta-llama/Llama-3.2-3B-Instruct "Tell me a story" --stream
JSON Schema Output
Force structured JSON output using schemas:
llm -m hf/meta-llama/Llama-3.2-3B-Instruct \
--schema '{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}' \
"Generate a person's profile"
Or use the simpler DSL format:
llm -m hf/meta-llama/Llama-3.2-3B-Instruct \
--schema 'name: str, age: int, hobbies: list[str]' \
"Generate a person's profile with hobbies"
Note: JSON schema support varies by model. Some models may not fully support structured output or may produce inconsistent results. Models specifically fine-tuned for instruction following and structured output generation tend to perform better with schemas.
Function Calling / Tools
The plugin supports function calling for models that have this capability:
llm -m hf/meta-llama/Llama-3.2-3B-Instruct \
--tool calculate 'def calculate(expression: str) -> float: """Evaluate a mathematical expression"""' \
"What is 15% of 240?"
Generation Parameters
Control generation with various parameters:
-
Temperature (0.0-2.0): Controls randomness
llm -m hf/meta-llama/Llama-3.2-3B-Instruct -o temperature 0.7 "Write creatively"
-
Top-p (0.0-1.0): Nucleus sampling threshold
llm -m hf/meta-llama/Llama-3.2-3B-Instruct -o top_p 0.9 "Generate text"
-
Max tokens: Limit response length
llm -m hf/meta-llama/Llama-3.2-3B-Instruct -o max_tokens 100 "Explain AI"
-
Stop sequences: Halt generation at specific strings
llm -m hf/meta-llama/Llama-3.2-3B-Instruct -o stop '[".", "!"]' "Generate until punctuation"
Interactive Chat
Start an interactive chat session:
llm chat -m hf/meta-llama/Llama-3.2-3B-Instruct
Conversation History
Continue previous conversations:
llm -m hf/meta-llama/Llama-3.2-3B-Instruct "What is Python?" -c
llm -c "What are its main uses?"
Advanced Options
Token Limits
The plugin supports both max_tokens (chat-completions style) and max_new_tokens (text-generation style) parameters:
# Using max_tokens (preferred)
llm -m hf/meta-llama/Llama-3.2-3B-Instruct -o max_tokens 500 "Tell a story"
# Using max_new_tokens (for compatibility)
llm -m hf/meta-llama/Llama-3.2-3B-Instruct -o max_new_tokens 500 "Tell a story"
Note: You cannot set both max_tokens and max_new_tokens simultaneously.
Response Metadata
Access detailed response metadata using the Python API:
import llm
model = llm.get_model("hf/meta-llama/Llama-3.2-3B-Instruct")
response = model.prompt("Hello")
# Access metadata
print(response.response_json)
# {'usage': {...}, 'model': '...', 'finish_reason': 'stop', ...}
Python API
Use the plugin programmatically:
import llm
# Get a model
model = llm.get_model("hf/meta-llama/Llama-3.2-3B-Instruct")
# Simple prompt
response = model.prompt("Explain machine learning")
print(response.text())
# With options
response = model.prompt(
"Write a poem",
temperature=0.9,
max_tokens=200
)
# Streaming
for chunk in model.prompt("Tell me a story", stream=True):
print(chunk, end="", flush=True)
# With system prompt
response = model.prompt(
"Translate to French: Hello",
system="You are a helpful translation assistant."
)
Model Discovery
The plugin automatically discovers models that support the text-generation task from Hugging Face. The discovery is cached to improve performance. Models are registered with the hf/ prefix to distinguish them from other LLM providers.
Limitations
- Attachments: Image, audio, and file attachments are not currently supported
- Model availability: Not all Hugging Face models may be accessible via the Inference API
- Rate limits: Subject to Hugging Face API rate limits based on your account type
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd llm-huggingface
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
License
Apache 2.0
Metadata
Release files for llm-huggingface-plugin 0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_huggingface_plugin-0.1.tar.gz | 16.1 kB | Without provenance | Details for llm_huggingface_plugin-0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_huggingface_plugin-0.1-py3-none-any.whl | Python 3 | none | any | Details for llm_huggingface_plugin-0.1-py3-none-any.whl |
Total release size: 30.9 kB
Release files / llm_huggingface_plugin-0.1.tar.gz
| Download URL | llm_huggingface_plugin-0.1.tar.gz |
|---|---|
| Size | 16.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9fbb7829a900f90812cf8594be096bd3e41d4844b61b7ff173c9f9bde4b6732f
|
|
BLAKE2b-256 checksum How to use checksums |
a216312068714cf337303f86cf27e56e2311fe700c77c00398bdf7f460908198
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.19
|
Release files / llm_huggingface_plugin-0.1-py3-none-any.whl
| Download URL | llm_huggingface_plugin-0.1-py3-none-any.whl |
|---|---|
| Size | 14.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a6e12e8d16d908f457f8c2d7714b4da4cb34e4f6d8f8ee84f59c401f677624f4
|
|
BLAKE2b-256 checksum How to use checksums |
e8c1f9dd9b6343fecb4b91b8ce6bd9021f9130305eeb3ed43ef9b93456dddb93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.19
|
Security for llm-huggingface-plugin 0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.