llama-rs-python 0.0.2

Unofficial python bindings for llama-rs. 🐍❤️🦀

llama-rs-python

Unofficial python bindings for llama-rs created with PyO3. 🐍❤️🦀

This package gives access to the basic functionality of the llama-rs project.

GGML converted models can be loaded and executed.

Installation

Simply install it via pip: pip install llama-rs-python

Usage

The package is typehinted for easy usage.

A usage example could look like this:

from llama_rs_python import Model

#load the model
model = Model("path/to/model.bin")

#generate
print(model.generate("The meaning of life is"))

The package also supports callbacks to get each token as it is generated. The callback-function also supports canceling the generation by returning a True value from the pytohn side.

from llama_rs_python import Model
import sys
from typing import Optional

#load the model
model = Model("path/to/model.bin")

#define the callback
def callback(token:str)->Optional[bool]:
    print(token,end="")
    sys.stdout.flush()
    # (return True here to cancel the generation)

#start generation
model.generate("The meaning of life is",callback=callback)

The configuration of the generation is handled by the GenerationConfig class.

from llama_rs_python import Model, GenerationConfig

#load the model
model = Model("path/to/model.bin")

#create a config
config = GenerationConfig(top_p=0.9,seed=1441,max_new_tokens=1024)

#generate
print(model.generate("The meaning of life is",generation_config=config))

To configure model specific settings the SessionConfig class can be used.

from llama_rs_python import Model, SessionConfig

#define the session
session_config = SessionConfig(threads=8,context_length=512)

#load the model
model = Model("path/to/model.bin",session_config=session_config)