llama-index llms mistral-rs integration
Project description
LlamaIndex Llms Integration: mistral.rs
To use this integration, please install the Python mistralrs
package:
Installation of mistralrs
from PyPi
-
Install Rust: https://rustup.rs/
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env
-
mistralrs
depends on theopenssl
library.
To install it on Ubuntu:
sudo apt install libssl-dev
sudo apt install pkg-config
- Install it!
-
CUDA
pip install mistralrs-cuda
-
Metal
pip install mistralrs-metal
-
Apple Accelerate
pip install mistralrs-accelerate
-
Intel MKL
pip install mistralrs-mkl
-
Without accelerators
pip install mistralrs
All installations will install the mistralrs
package. The suffix on the package installed by pip
only controls the feature activation.
Installation from source
Please follow the instructions here.
Usage
from llama_index.llms.mistral_rs import MistralRS
from mistralrs import Which
llm = MistralRS(
which=Which.GGUF(
tok_model_id="mistralai/Mistral-7B-Instruct-v0.1",
quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
tokenizer_json=None,
repeat_last_n=64,
),
max_new_tokens=4096,
context_window=1024 * 5,
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for llama_index_llms_mistral_rs-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6caf0d6a0a58d76bd6e75d0a927fe5a03c3d6848dc23b222d5082c84cd4a5a41 |
|
MD5 | 536a854b4a90a5c28fb8298e118eb71f |
|
BLAKE2b-256 | a583e994c33bcd885b8ac17bd53be8a96fd8d5aee5fde93bf58b894ea0136d84 |
Close
Hashes for llama_index_llms_mistral_rs-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b05043658d2e36e6f3164b86cd23675050e25b7b8bc64af8487ebe6a1574d9b0 |
|
MD5 | bbbf47fe62596e030c393206ba97f863 |
|
BLAKE2b-256 | 4493905c9ba755df03c434e3a99436fdc3363c9209becbcaa1fdde5106cb64d5 |