Generate Text with LLMs and MLX
The easiest way to get started is to install the mlx-lm package:
With pip:
pip install mlx-lm
With conda:
conda install -c conda-forge mlx-lm
The mlx-lm package also has:
Python API
You can use mlx-lm as a module:
from mlx_lm import load, generate
model, tokenizer = load("mistralai/Mistral-7B-Instruct-v0.1")
response = generate(model, tokenizer, prompt="hello", verbose=True)
To see a description of all the arguments you can do:
>>> help(generate)
The mlx-lm package also comes with functionality to quantize and optionally
upload models to the Hugging Face Hub.
You can convert models in the Python API with:
from mlx_lm import convert
upload_repo = "mlx-community/My-Mistral-7B-v0.1-4bit"
convert("mistralai/Mistral-7B-v0.1", quantize=True, upload_repo=upload_repo)
This will generate a 4-bit quantized Mistral-7B and upload it to the
repo mlx-community/My-Mistral-7B-v0.1-4bit. It will also save the
converted model in the path mlx_model by default.
To see a description of all the arguments you can do:
>>> help(convert)
Command Line
You can also use mlx-lm from the command line with:
python -m mlx_lm.generate --model mistralai/Mistral-7B-Instruct-v0.1 --prompt "hello"
This will download a Mistral 7B model from the Hugging Face Hub and generate text using the given prompt.
For a full list of options run:
python -m mlx_lm.generate --help
To quantize a model from the command line run:
python -m mlx_lm.convert --hf-path mistralai/Mistral-7B-Instruct-v0.1 -q
For more options run:
python -m mlx_lm.convert --help
You can upload new models to Hugging Face by specifying --upload-repo to
convert. For example, to upload a quantized Mistral-7B model to the
MLX Hugging Face community you can do:
python -m mlx_lm.convert \
--hf-path mistralai/Mistral-7B-v0.1 \
-q \
--upload-repo mlx-community/my-4bit-mistral
Supported Models
The example supports Hugging Face format Mistral, Llama, and Phi-2 style models. If the model you want to run is not supported, file an issue or better yet, submit a pull request.
Here are a few examples of Hugging Face models that work with this example:
- mistralai/Mistral-7B-v0.1
- meta-llama/Llama-2-7b-hf
- deepseek-ai/deepseek-coder-6.7b-instruct
- 01-ai/Yi-6B-Chat
- microsoft/phi-2
- mistralai/Mixtral-8x7B-Instruct-v0.1
- Qwen/Qwen-7B
- pfnet/plamo-13b
- pfnet/plamo-13b-instruct
- stabilityai/stablelm-2-zephyr-1_6b
Most Mistral, Llama, Phi-2, and Mixtral style models should work out of the box.
For some models (such as Qwen and plamo) the tokenizer requires you to
enable the trust_remote_code option. You can do this by passing
--trust-remote-code in the command line. If you don't specify the flag
explicitly, you will be prompted to trust remote code in the terminal when
running the model.
For Qwen models you must also specify the eos_token. You can do this by
passing --eos-token "<|endoftext|>" in the command
line.
These options can also be set in the Python API. For example:
model, tokenizer = load(
"qwen/Qwen-7B",
tokenizer_config={"eos_token": "<|endoftext|>", "trust_remote_code": True},
)
Release files for mlx-lm 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mlx-lm-0.7.0.tar.gz | 44.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mlx_lm-0.7.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 104.0 kB
Release files / mlx-lm-0.7.0.tar.gz
| Download URL | mlx-lm-0.7.0.tar.gz |
|---|---|
| Size | 44.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
916ce5d54d90fbfa12ea6d5399d54ffe42ccdf7d1d484536f93651277007845a
|
|
BLAKE2b-256 checksum How to use checksums |
d042d570ff796ca0c3992579509293249a0d56a6247adc65492b9c5a1ec9517b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
Release files / mlx_lm-0.7.0-py3-none-any.whl
| Download URL | mlx_lm-0.7.0-py3-none-any.whl |
|---|---|
| Size | 59.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2ac69941bb83e13592646aea143debc2751b9c778498e3adf082189969fe77c2
|
|
BLAKE2b-256 checksum How to use checksums |
102df5e7aa57a89a1a00af23ceedd57c00990cea85db688520f53613586c9ebc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
