mistral.rs
mistralrs is a Python package which provides an easy to use API for mistral.rs.
Example
More examples can be found here!
from mistralrs import Runner, Which, ChatCompletionRequest
runner = Runner(
which=Which.Plain(
model_id="microsoft/Phi-3.5-mini-instruct",
),
in_situ_quant="Q4K",
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="mistral",
messages=[
{"role": "user", "content": "Tell me a story about the Rust type system."}
],
max_tokens=256,
presence_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res.choices[0].message.content)
print(res.usage)
Multimodal (audio + image) example
mistralrs also supports multimodal vision models that can reason over both
images and audio clips via the same OpenAI-style audio_url / image_url
format. The example below queries the Phi-4-Multimodal model with a single
image and an audio recording – notice how the text prompt references them via
<|audio_1|> and <|image_1|> tokens (indexing starts at 1):
from mistralrs import Runner, Which, ChatCompletionRequest, VisionArchitecture
runner = Runner(
which=Which.VisionPlain(
model_id="microsoft/Phi-4-multimodal-instruct",
arch=VisionArchitecture.Phi4MM,
),
)
IMAGE_URL = "https://www.allaboutbirds.org/guide/assets/og/528129121-1200px.jpg"
AUDIO_URL = "https://upload.wikimedia.org/wikipedia/commons/4/42/Bird_singing.ogg"
response = runner.send_chat_completion_request(
ChatCompletionRequest(
model="phi4mm",
messages=[
{
"role": "user",
"content": [
{"type": "audio_url", "audio_url": {"url": AUDIO_URL}},
{"type": "image_url", "image_url": {"url": IMAGE_URL}},
{
"type": "text",
"text": "<|audio_1|><|image_1|> Describe in detail what is happening, referencing both what you hear and what you see.",
},
],
}
],
max_tokens=256,
temperature=0.2,
top_p=0.9,
)
)
print(response.choices[0].message.content)
See examples/python/phi4mm_audio.py for a ready-to-run version.
Please find API docs here and the type stubs here, which are another great form of documentation.
We also provide a cookbook here!
1 maintainer
Metadata
1 maintainer
Release files for mistralrs-cuda 0.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| mistralrs_cuda-0.6.0-cp312-none-win_amd64.whl | CPython 3.12 | none | Windows x86-64 | Details |
| mistralrs_cuda-0.6.0-cp312-cp312-manylinux_2_39_x86_64.whl | CPython 3.12 | CPython 3.12 | Linux glibc 2.39+ x86-64 | Details |
| mistralrs_cuda-0.6.0-cp311-none-win_amd64.whl | CPython 3.11 | none | Windows x86-64 | Details |
| mistralrs_cuda-0.6.0-cp311-cp311-manylinux_2_39_x86_64.whl | CPython 3.11 | CPython 3.11 | Linux glibc 2.39+ x86-64 | Details |
| mistralrs_cuda-0.6.0-cp310-none-win_amd64.whl | CPython 3.10 | none | Windows x86-64 | Details |
| mistralrs_cuda-0.6.0-cp310-cp310-manylinux_2_39_x86_64.whl | CPython 3.10 | CPython 3.10 | Linux glibc 2.39+ x86-64 | Details |
Total release size: 180.5 MB
Release files / mistralrs_cuda-0.6.0-cp312-none-win_amd64.whl
| Download URL | mistralrs_cuda-0.6.0-cp312-none-win_amd64.whl |
|---|---|
| Size | 27.0 MB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
d4920f91db58900f4e65771cbb94922dc3cdcb4b736cb216176c459997169e55
|
|
BLAKE2b-256 checksum How to use checksums |
9ea930e5be46614a1a0b7ed36edfa323e1dac7b21930b35e1190efc69784815d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.7
|
Release files / mistralrs_cuda-0.6.0-cp312-cp312-manylinux_2_39_x86_64.whl
| Download URL | mistralrs_cuda-0.6.0-cp312-cp312-manylinux_2_39_x86_64.whl |
|---|---|
| Size | 33.2 MB |
| Tags | CPython 3.12 Linux glibc 2.39+ x86-64 |
|
SHA-256 checksum How to use checksums |
b92b00d66a8d490fa89f0ab86f3bdf56f6b2a315dd3fd1d8230207c336b7c4f7
|
|
BLAKE2b-256 checksum How to use checksums |
d3939c57756b2e8d410d0bd56276de71a862506a265f91c7f6a366471fd0affd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.7
|
Release files / mistralrs_cuda-0.6.0-cp311-none-win_amd64.whl
| Download URL | mistralrs_cuda-0.6.0-cp311-none-win_amd64.whl |
|---|---|
| Size | 27.0 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
e243f53463d74a1b63b1b3a2663c606780ff19e960e030555ea92f74cb7a1f3a
|
|
BLAKE2b-256 checksum How to use checksums |
a1474efb6d4e50ff9d42814974b9ee38fe2ea8fca510c2b612e5d17b6c462ef9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.7
|
Release files / mistralrs_cuda-0.6.0-cp311-cp311-manylinux_2_39_x86_64.whl
| Download URL | mistralrs_cuda-0.6.0-cp311-cp311-manylinux_2_39_x86_64.whl |
|---|---|
| Size | 33.2 MB |
| Tags | CPython 3.11 Linux glibc 2.39+ x86-64 |
|
SHA-256 checksum How to use checksums |
b1ce36fe6c160c84faee76e88ddcd5cac27f3a0db360a6dde55466b7b894566b
|
|
BLAKE2b-256 checksum How to use checksums |
f3e0e7a558168e082c35f566588b6ed7927f9c03e4d5f8644f8b9aef1f342cf9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.7
|
Release files / mistralrs_cuda-0.6.0-cp310-none-win_amd64.whl
| Download URL | mistralrs_cuda-0.6.0-cp310-none-win_amd64.whl |
|---|---|
| Size | 27.0 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
b1224d14c4e5b7e102d60768d23c2f5b0b6039c825a5d51347b469e54982a3b4
|
|
BLAKE2b-256 checksum How to use checksums |
86e84ac4866769c1ee06915103eae9e892f1426abf44dde9d196b7358e6b5c80
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.7
|
Release files / mistralrs_cuda-0.6.0-cp310-cp310-manylinux_2_39_x86_64.whl
| Download URL | mistralrs_cuda-0.6.0-cp310-cp310-manylinux_2_39_x86_64.whl |
|---|---|
| Size | 33.2 MB |
| Tags | CPython 3.10 Linux glibc 2.39+ x86-64 |
|
SHA-256 checksum How to use checksums |
0b54b827929064332183cebb3d276cc469fea5ae609c1005555fe09313766b07
|
|
BLAKE2b-256 checksum How to use checksums |
dea0d7a76384ddf1fa1f6c53db41c84b95b19e89cb6fe146246260b67d7ddd04
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.7
|
Security for mistralrs-cuda 0.6.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
