openaoe 0.0.1

LLM Group Chat Framework: chat with multiple LLMs at the same time

English | 简体中文

Latest Progress 🎉

• [January 2024] Released version v0.0.1, officially open-sourced！

---

Introduction

What is OpenAOE?

AOE, an acronym from DOTA2 for Area of Effect, denotes an ability that can affect a group of targets within a certain area. Here, AOE in AI implies that user can obtain parallel outputs from multiple LLMs with one single prompt.

What problem does OpenAOE want to solve?

Currently, there are many open-source frameworks based on the ChatGPT for chat, but the open-sourced
single-prompt-multiple-LLM-responses framework is still not coming yet.

The emergence of AOE fills this gap: AOE can help large model algorithm researchers, evaluators, engineering developers, and even non-professionals to quickly access the market's well-known commercial and open-source large models, providing both single model serial response and multi-model parallel response modes.

What can you get from OpenAOE?

1. Send a prompt once to one or more large language models and get their return.
2. Provides access to commercial large model APIs, with default support for gpt3.5, gpt4, Google Palm, Minimax, Claude, Spark, etc., and also supports user-defined access to other large model APIs.
3. Provides access to open-source large model APIs; users can use LMDeploy to deploy open-source large models with one click.
4. We also provide backend APIs and a WEB side to meet the needs of different users.

Quick Run

We will provide three different ways to run open-aoe: run by pip， run by docker and run by source code as well.

Run by pip

[!TIP] Require python >= 3.9

Install

pip install -U open-aoe 

Start

open-aoe -f /path/to/your/config-template.yaml

Run by docker

Install

There are two ways to get the OpenAOE docker image by:

1. pull the open-aoe docker image

docker pull open-aoe:latest

2. or build a docker image

git clone https://github.com/internlm/OpenAOE
cd open-aoe
docker build . -f docker/Dockerfile -t open-aoe:latest

Start

docker run -p 10099:10099 -v /path/to/your/config-template.yaml:/app/config-template.yaml --name OpenAOE open-aoe:latest

Run by source code

Install

1. clone this project

git clone https://github.com/internlm/OpenAOE

2. [optional] build the frontend project when the frontend codes are changed

cd open-aoe/openaoe/frontend
npm install
npm run build

Start

cd open-aoe/openaoe
pip install -r backend/requirements.txt
python -m main -f /path/to/your/config-template.yaml

[!TIP] /path/to/your/config.yaml is the configuration file read by OpenAOE at startup, containing the relevant configuration information for the large model, including: API URLs, AKSKs, Tokens, etc., which are required for the startup of OpenAOE. A template file can be found in openaoe/backend/config/config.yaml.

Tech Introduction

You are always welcome to fork this project and contribute your work

If you want to add more LLMs' APIs or features based on OpenAOE, the following info might be helpful.

Tech Stack

The technology stack we use includes:

1. Backend framework based on python + fastapi;
2. Frontend framework based on typescript + Sealion-Client (encapsulated based on React) + Sealion-UI.
3. Build tools:
   1. conda: quickly create a virtual python env to install necessary packages
   2. npm: build the frontend project

[!TIP] The build tools can be installed by sealion-cli by pip install -U sealion-cli

Organization of the Repo

• Frontend code in openaoe/frontend
• Backend code in openaoe/backend
• Project entry-point is openaoe/main.py

How to add a new model

Frontend

• Add new model info like name, avatar, provider, etc in openaoe/frontend/src/config/model-config.ts
• Add a new model basic API request payload configuration in openaoe/frontend/src/config/api-config.ts
• Modify your new model's payload specifically in openaoe/frontend/src/services/fetch.ts, you may need to change the payload structure and handle corner cases according to your model's API definition.