Cosmian Python bindings for ggml

This library provides python bindings for the examples models of ggml.

Using this library, you can deploy state-of-the-art LLMs on Intel SGX hardware with Cosmian MSE.

Available models architecture

• gpt-neox

from cosmian_py_ggml import GPTNeoX
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
model = GPTNeoX("ggml_model_path")

res = model(tokenizer.encode(query))
print("Response:", tokenizer.decode(res))

• falcon

from cosmian_py_ggml import Falcon
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-40b")
model = Falcon("ggml_model_path")

res = model(tokenizer.encode(query))
print("Response:", tokenizer.decode(res))

• mpt

from cosmian_py_ggml import MPT
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mosaicml/mpt-30b")
model = MPT("ggml_model_path")

res = model(tokenizer.encode(query))
print("Response:", tokenizer.decode(res))

Build from sources

git clone --recursive git@github.com:Cosmian/cosmian_py_ggml.git
scripts/build.sh