Llama toolchain
Project description
llama-toolchain
This repo contains the API specifications for various components of the Llama Stack as well implementations for some of those APIs like model inference. The Stack consists of toolchain-apis and agentic-apis. This repo contains the toolchain-apis
Installation and Setup
mkdir -p ~/local
cd ~/local
git clone git@github.com:meta-llama/llama-toolchain.git
conda create -n toolchain python=3.10
conda activate toolchain
cd llama-toolchain
pip install -e .
Test with cli
We have built a llama cli to make it easy to configure / run parts of the toolchain
llama --help
usage: llama [-h] {download,inference,model,agentic_system} ...
Welcome to the LLama cli
options:
-h, --help show this help message and exit
subcommands:
{download,inference,model,agentic_system}
There are several subcommands to help get you started
Start inference server that can run the llama models
llama inference configure
llama inference start
Test client
python -m llama_toolchain.inference.client localhost 5000
Initializing client for http://localhost:5000
User>hello world, help me out here
Assistant> Hello! I'd be delighted to help you out. What's on your mind? Do you have a question, a problem, or just need someone to chat with? I'm all ears!
Running FP8
You need fbgemm-gpu
package which requires torch >= 2.4.0 (currently only in nightly, but releasing shortly...).
ENV=fp8_env
conda create -n $ENV python=3.10
conda activate $ENV
pip3 install -r fp8_requirements.txt
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
Close
Hashes for llama_toolchain-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6b90cb369832595b25ff5782b18a448fca7d4439f4ae22fda1fd0655cfa6988 |
|
MD5 | 35251c89450e98a01d7580f13ac0b4ee |
|
BLAKE2b-256 | 370212bfc45731a722d5887306cd2e331e12aa76d7d055922970c77f0b19245b |