proxyllm 0.1.6

LLM Proxy to reduce cost and complexity of using multiple LLMs

LLM Proxy

A low-code solution to efficiently manage multiple large language models
View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License

What is LLM Proxy?

LLM Proxy is a tool that sits between your application and the different LLM providers. LLM Proxy's goal is to simplify the use of multiple LLMs through a TUI while providing cost and response optimization.

(back to top)

Getting Started

There are 2 ways to get started. You can directly clone the repo into your project or install it as a library.

Prerequisites

• Python 3.11+

Installation

With pip:

pip install proxyllm

With poetry:

poetry add proxyllm

Run the install script for the default configuration file:

config --default-config

If you prefer poetry:

poetry run config --default-config

If the installation scripts do not work, you can visit the repo to grab a copy manually.

Note:

• Ensure that you have all of your API keys for each respective provider in the .env file (You can utilize the .env.example for reference)
• For Google's models, you will need the path to application credentials, and the project ID inside of the .env

(back to top)

Usage

Currently, the LLM Proxy provides 2 different route types: Cost and Category.

To get started import the LLMProxy client:

from proxyllm import LLMProxy

After the setup is complete, you only need 1 line of code to get started:

llmproxy_client = LLMProxy()

Note: You will need to specify your yaml configuration file if you did not use the default name:

llmproxy_client = LLMProxy(path_to_user_configuration="llmproxy.config.yml")

To use the llmproxy, simply call the route function with your prompt:

output = llmproxy_client.route(prompt=prompt)

The route function will return a CompletionResponse:

print("RESPONSE MODEL: ", output.response_model)
print("RESPONSE: ", output.response)
print("ERRORS: ", output.errors)

• response_model: contains the model used for the request
• response: contains the string response from the model
• errors: contains an array of models that failed to make a request with their respective errors

Important Note: Although parameters changed programmatically, it is best to favor the YAML configuration file. Only use the constructor parameters when you must override the YAML configuration.

(back to top)

Roadmap

• Support for more providers
  • Replicate
  • Claude
• Support for multimodal Models
• Custom, optimized model for category routing
• Effectiveness Routing
• Context Injection

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

LLM Proxy is open source, so we are open and grateful, for contributions. Open-source communities are what makes software great, so feel fork the repo and create a pull request with the feature tag. Thanks!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)