mlx-lm

Generate Text with LLMs and MLX

The easiest way to get started is to install the mlx-lm package:

With pip:

pip install mlx-lm

With conda:

conda install -c conda-forge mlx-lm

The mlx-lm package also has:

Python API

You can use mlx-lm as a module:

from mlx_lm import load, generate

model, tokenizer = load("mistralai/Mistral-7B-Instruct-v0.1")

response = generate(model, tokenizer, prompt="hello", verbose=True)

To see a description of all the arguments you can do:

>>> help(generate)

The mlx-lm package also comes with functionality to quantize and optionally upload models to the Hugging Face Hub.

You can convert models in the Python API with:

from mlx_lm import convert

upload_repo = "mlx-community/My-Mistral-7B-v0.1-4bit"

convert("mistralai/Mistral-7B-v0.1", quantize=True, upload_repo=upload_repo)

This will generate a 4-bit quantized Mistral-7B and upload it to the repo mlx-community/My-Mistral-7B-v0.1-4bit. It will also save the converted model in the path mlx_model by default.

To see a description of all the arguments you can do:

>>> help(convert)

Command Line

You can also use mlx-lm from the command line with:

python -m mlx_lm.generate --model mistralai/Mistral-7B-Instruct-v0.1 --prompt "hello"

This will download a Mistral 7B model from the Hugging Face Hub and generate text using the given prompt.

For a full list of options run:

python -m mlx_lm.generate --help

To quantize a model from the command line run:

python -m mlx_lm.convert --hf-path mistralai/Mistral-7B-Instruct-v0.1 -q

For more options run:

python -m mlx_lm.convert --help

You can upload new models to Hugging Face by specifying --upload-repo to convert. For example, to upload a quantized Mistral-7B model to the MLX Hugging Face community you can do:

python -m mlx_lm.convert \
    --hf-path mistralai/Mistral-7B-v0.1 \
    -q \
    --upload-repo mlx-community/my-4bit-mistral

Supported Models

The example supports Hugging Face format Mistral, Llama, and Phi-2 style models. If the model you want to run is not supported, file an issue or better yet, submit a pull request.

Here are a few examples of Hugging Face models that work with this example:

Most Mistral, Llama, Phi-2, and Mixtral style models should work out of the box.

For some models (such as Qwen and plamo) the tokenizer requires you to enable the trust_remote_code option. You can do this by passing --trust-remote-code in the command line. If you don't specify the flag explicitly, you will be prompted to trust remote code in the terminal when running the model.

For Qwen models you must also specify the eos_token. You can do this by passing --eos-token "<|endoftext|>" in the command line.

These options can also be set in the Python API. For example:

model, tokenizer = load(
    "qwen/Qwen-7B",
    tokenizer_config={"eos_token": "<|endoftext|>", "trust_remote_code": True},
)

Metadata

Release files for mlx-lm 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mlx-lm 0.7.0
File	Size	Uploaded	Provenance
mlx-lm-0.7.0.tar.gz	44.5 kB	Apr 5, 2024	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for mlx-lm 0.7.0
File	Interpreter	ABI	Platform	Reset
mlx_lm-0.7.0-py3-none-any.whl 59.5 kB Apr 5, 2024 Without provenance	Python 3	none	any	Details

Total release size: 104.0 kB

Release files / mlx-lm-0.7.0.tar.gz

Download URL	mlx-lm-0.7.0.tar.gz
Size	44.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`916ce5d54d90fbfa12ea6d5399d54ffe42ccdf7d1d484536f93651277007845a`
BLAKE2b-256 checksum How to use checksums	`d042d570ff796ca0c3992579509293249a0d56a6247adc65492b9c5a1ec9517b`
Upload date	Apr 5, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.9.17`

Release files / mlx_lm-0.7.0-py3-none-any.whl

Download URL	mlx_lm-0.7.0-py3-none-any.whl
Size	59.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`2ac69941bb83e13592646aea143debc2751b9c778498e3adf082189969fe77c2`
BLAKE2b-256 checksum How to use checksums	`102df5e7aa57a89a1a00af23ceedd57c00990cea85db688520f53613586c9ebc`
Upload date	Apr 5, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.2 CPython/3.9.17`

Security for mlx-lm 0.7.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.