mistral.rs PyO3 Bindings: mistralrs
mistralrs is a Python package which provides an API for mistral.rs. We build mistralrs with the maturin build manager.
Installation from PyPi
-
Install Rust: https://rustup.rs/
Example on Ubuntu:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env
-
mistralrsdepends on theopenssllibrary.
Example on Ubuntu:
sudo apt install libssl-dev
sudo apt install pkg-config
- Install it!
-
CUDA
pip install mistralrs-cuda -v -
Metal
pip install mistralrs-metal -v -
Apple Accelerate
pip install mistralrs-accelerate -v -
Intel MKL
pip install mistralrs-mkl -v -
Without accelerators
pip install mistralrs -v
All installations will install the mistralrs package. The suffix on the package installed by pip only controls the feature activation.
Installation from source
-
Install required packages
openssl(Example on Ubuntu:sudo apt install libssl-dev)pkg-config(Example on Ubuntu:sudo apt install pkg-config)
-
Install Rust: https://rustup.rs/
Example on Ubuntu:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env
-
Set HF token correctly (skip if already set or your model is not gated, or if you want to use the
token_sourceparameters in Python or the command line.)Example on Ubuntu:
mkdir ~/.cache/huggingface touch ~/.cache/huggingface/token echo <HF_TOKEN_HERE> > ~/.cache/huggingface/token
-
Download the code
git clone https://github.com/EricLBuehler/mistral.rs.git cd mistral.rs
-
cdinto the correct directory for buildingmistralrs:cd mistralrs-pyo3 -
Install
maturin, our Rust + Python build system: Maturin requires a Python virtual environment such asvenvorcondato be active. Themistralrspackage will be installed into that environment.pip install maturin[patchelf] -
Install
mistralrsInstallmistralrsby executing the following in this directory where features such ascudaorflash-attnmay be specified with the--featuresargument just like they would be forcargo run.The base build command is:
maturin develop -r
- To build for CUDA:
maturin develop -r --features cuda
- To build for CUDA with flash attention:
maturin develop -r --features "cuda flash-attn"
- To build for Metal:
maturin develop -r --features metal
- To build for Accelerate:
maturin develop -r --features accelerate
- To build for MKL:
maturin develop -r --features mkl
Please find API docs here and the type stubs here, which are another great form of documentation.
We also provide a cookbook here!
Example
from mistralrs import Runner, Which, ChatCompletionRequest
runner = Runner(
which=Which.GGUF(
tok_model_id="mistralai/Mistral-7B-Instruct-v0.1",
quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
)
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="mistral",
messages=[
{"role": "user", "content": "Tell me a story about the Rust type system."}
],
max_tokens=256,
presence_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res.choices[0].message.content)
print(res.usage)
Metadata
Release files for mistralrs 0.2.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mistralrs-0.2.3.tar.gz | 343.0 kB | Without provenance | Details for mistralrs-0.2.3.tar.gz |
Release files / mistralrs-0.2.3.tar.gz
| Download URL | mistralrs-0.2.3.tar.gz |
|---|---|
| Size | 343.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c7e68b1bbf6f3eb09c6cd3a6337c29c2105e499c14686ac6fd839be8e8368442
|
|
BLAKE2b-256 checksum How to use checksums |
c3e6759dde6e62e1d782a2dc7eedfb1678d71678109c875f35d394fde48f3080
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.8.18
|
Security for mistralrs 0.2.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.