Skip to main content

No project description provided

Project description

llama-api-server

This project is under active deployment. Breaking changes could be made any time.

Llama as a Service! This project try to build a REST-ful API server compatible to OpenAI API using open source backends like llama.

Tested with

  • openai-python
    • OPENAI_API_TYPE=default
    • OPENAI_API_TYPE=azure

Supported APIs

  • Completions
    • set temperature, top\_p, and top\_k
    • set max\_tokens
    • set stop
    • set stream
    • set n
    • set presence\_penalty and frequency\_penalty
    • set logit\_bias
  • Embeddings
    • batch process
  • Chat

Supported backed

Others

  • Documents
  • Token auth
  • Intergration tests
  • Performance parameters like n_batch and n_thread
  • A tool to download/prepare pretrain model

Get start

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

llama_api_server-0.1.0.tar.gz (5.5 kB view hashes)

Uploaded Source

Built Distribution

llama_api_server-0.1.0-py3-none-any.whl (5.4 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page