A simple library for interfacing with language models.
Project description
langdash
A simple library for interfacing with language models.
Currently in beta!
Features:
- Support for text generation, text classification (through prompting) and vector-based document searching.
- Lightweight, build-it-yourself-style prompt wrappers.
- Token healing and transformers/RNN state reuse for fast inference, like in Microsoft's guidance.
- First-class support for ggml backends.
Documentation: Read on readthedocs.io
Repository: main / Gitlab mirror
Installation
Use pip to install. By default, langdash does not come preinstalled with any additional modules. You will have to specify what you need like in the following command:
pip install --user langdash[embeddings,sentence_transformers]
List of modules:
- core:
- embeddings: required for running searching through embeddings
- backends:
- Generation backends: rwkv_cpp, llama_cpp, ctransformers (alpha), transformers
- Embedding backends: sentence_transformers
Note: If running from source, initialize the git submodules in the langdash/extern
folder to compile foreign backends.
Usage
Examples:
See examples folder for full examples.
Running the Examples
All examples can be ran with the following command:
python examples/instruct.py [model type] [model name or path]
You can specify additional model parameters using the -ae
CLI argument, and passing a valid Python literal. For example, to run the chat example using the WizardLM model with context length of 4096, do:
python examples/chat.py llama_cpp /path/to/ggml-wizardlm.bin -ae n_ctx 4096
Some examples require you to specify the prompt format. Currently, two are implemented: wizardlm
(shortened Alpaca format without the first prompt line and # Instruction:
), and alpaca
(the full format). You will need to specify it for most of the examples:
python examples/instruct.py llama_cpp /path/to/ggml-wizardlm.bin -ae n_ctx 4096 --prompt-format wizardlm
License
Apache 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.