LLM Proxy to reduce cost and complexity of using multiple LLMs
Project description
LLM Proxy
A low-code solution to efficiently manage multiple large language models
View Demo
·
Report Bug
·
Request Feature
Table of Contents
What is LLM Proxy?
LLM Proxy is a tool that sits between your application and the different LLM providers. LLM Proxy's goal is to simplify the use of multiple LLMs through a TUI while providing cost and response optimization.
Getting Started
There are 2 ways to get started. You can directly clone the repo into your project or install it as a library.
Prerequisites
- Python 3.11+
Installation
With pip:
pip install proxyllm
With poetry:
poetry add proxyllm
Run the install script for the default configuration file:
config --default-config
If you prefer poetry:
poetry run config --default-config
If the installation script does not work, you can visit the repo to grab a copy manually.
Ensure that you have all of your API keys for each respective provider in the .env file (You can utilize the .env.example for reference)
Note: For Google's models, you will need the path to application credentials, and the project ID inside of the .env
Usage
Currently, the LLM Proxy provides 2 different route types: Cost and Category.
To get started import the LLMProxy client:
from proxyllm import LLMProxy
After the setup is complete, you only need 1 line of code to get started:
llmproxy_client = LLMProxy()
Note: You will need to specify your yaml configuration file if you did not use the default name:
llmproxy_client = LLMProxy(path_to_user_configuration="llmproxy.config.yml")
To use the llmproxy, simply call the route function with your prompt:
output = llmproxy_client.route(prompt=prompt)
The route function will return a CompletionResponse:
print("RESPONSE MODEL: ", output.response_model)
print("RESPONSE: ", output.response)
print("ERRORS: ", output.errors)
response_model:
contains the model used for the requestresponse:
contains the string response from the modelerrors:
contains an array of models that failed to make a request with their respective errors
Important Note: Although parameters changed programmatically, it is best to favor the YAML configuration file. Only use the constructor parameters when you must override the YAML configuration.
Roadmap
- Support for more providers
- Replicate
- Claude
- Support for multimodal Models
- Custom, optimized model for category routing
- Effectiveness Routing
- Context Injection
See the open issues for a full list of proposed features (and known issues).
Contributing
LLM Proxy is open source, so we are open and grateful, for contributions. Open-source communities are what makes software great, so feel fork the repo and create a pull request with the feature tag. Thanks!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
License
Distributed under the MIT License. See LICENSE.txt
for more information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.