allmdev 1.2.2

A simple and efficient python library for fast inference of GGUF Large Language Models.

ALLM

ALLM is a Python library designed for fast inference of GGUF (Generic Global Unsupervised Features) Large Language Models (LLMs) on both CPU and GPU. It provides a convenient interface for loading pre-trained GGUF models and performing inference using them. This library is ideal for applications where quick response times are crucial, such as chatbots, text generation, and more.

Features

• Efficient Inference: ALLM leverages the power of GGUF models to provide fast and accurate inference.
• CPU and GPU Support: The library is optimized for both CPU and GPU, allowing you to choose the best hardware for your application.
• Simple Interface: With a straightforward command line support, you can easily load models and perform inference with just a single command.
• Flexible Configuration: Customize inference settings such as temperature and model path to suit your needs.

Installation

You can install ALLM using pip:

pip install allm

Usage

You can start inference with a simple 'allm-run' command. The command takes name or path, temperature(optional), max new tokens(optional) and additional model kwargs(optional) as arguments.

allm-run --name model_name_or_path

API

You can initiate the inference API by simply using the 'allm-serve' command. This command launches the API server on the default host, 127.0.0.1:5000. If you prefer to run the API server on a different port and host, you have the option to customize the apiconfig.txt file within your model directory.

allm-serve

Supported Model names

Llama2, llama, llama2_chat, Llama_chat, Mistral, Mistral_instruct