llama-api-server 0.1.3

No project description provided

llama-api-server

This project is under active deployment. Breaking changes could be made any time.

Llama as a Service! This project try to build a REST-ful API server compatible to OpenAI API using open source backends like llama.

Get started

Prepare model

llama.cpp

If you you don't have quantize llama, you need to follow instruction to prepare model.

Install

pip install llama-api-server
echo > config.yml << EOF
models:
  completions:
    text-davinci-003:
      type: llama_cpp
      params:
        path: /absolute/path/to/your/7B/ggml-model-q4_0.bin
  embeddings:
    text-embedding-ada-002:
      type: llama_cpp
      params:
        path: /absolute/path/to/your/7B/ggml-model-q4_0.bin
EOF

# start web server
python -m llama_api_server

Call with openai-python

export OPENAI_API_BASE=http://127.0.0.1:5000/v1
openai api completions.create -e text-davinci-003 -p "hello?"

Roadmap

Tested with

• openai-python
  • OPENAI_API_TYPE=default
  • OPENAI_API_TYPE=azure

Supported APIs

• Completions
  • set temperature, top_p, and top_k
  • set max_tokens
  • set stop
  • set stream
  • set n
  • set presence_penalty and frequency_penalty
  • set logit_bias
• Embeddings
  • batch process
• Chat

Supported backed

• llama.cpp via llamacpp-python

Others

• Performance parameters like n_batch and n_thread
• Documents
• Token auth
• Intergration tests
• A tool to download/prepare pretrain model