multillm 0.916

VerifAI MultiLLM: A module to invoke multiple LLMs concurrently and rank their results

Multi LLM

VerifAI Implementation of invoking multiple large language models concurrently and ranking results

Quick-Start

For further information look to Running the Multi_LLM Application inside of Useage

Installation

pip3 install multillm

Edit the config file to point to your google and openai API keys.

Basic Example

multillm -c config.json -prompt "write a python function to determine whether a given number is prime"

The above example will run the config.json with the provided prompt.

Output of the above multillm command

multi llm response

1. GPT's implementation of the sort_integers function is cleaner and more readable compared to BARD's implementation. They use the built-in sorted function which returns a new sorted list, while BARD uses the sort method which modifies the original list in-place. GPT's implementation is more intuitive and follows best practices for Python programming.

2. BARD's implementation is functional but could be improved. They use the sort method to sort the list in-place, which can be confusing if someone expects the function to return a new sorted list. However, their implementation is still correct and will sort the list correctly.

Usage

The Multi_LLM application provides a powerful and efficient solution for invoking multiple large language models (LLMs) concurrently and effectively managing their outputs. This section guides you through the steps required to run the application using various command-line options and configuration files.

Prerequisites

Before running the Multi_LLM application, ensure that you have the following prerequisites in place:

• Python >= 3.8 installed on your system.

Command-Line Usage

The Multi_LLM application can be executed from the command line using the following command structure:

multillm -c <config_file> -prompt "<prompt_text>"

• <config_file>: The path to the configuration file containing LLM details.
• <prompt_text>: The prompt text you want to provide to the language models.

Config Files

The configuration file config.json defines the language models and their associated parameters. See Config JSON for more information. It follows the structure outlined below:

Example Config.json file

{
    "Config": {
        "Multi_LLM": {
            "llms": [
                {
                    "file": "bard.py",
                    "class_name": "BARD",
                    "model": "chat-bison@001",
                    "credentials": "/path/to/google/key.json"
                },
                {
                    "file": "GPT.py",
                    "class_name": "GPT",
                    "model": "gpt-3.5-turbo",
                    "credentials": "/path/to/openai/key.json"
                }
            ]
        }
    }
}

Running the Application

To run the Multi_LLM application, follow these steps:

1. Download the config.json file with the desired language model configurations.

2. Open a terminal and navigate to the directory containing the multi_llm.py script.

3. Execute the following command, replacing <config_file> with the actual path to your configuration file and <prompt_text> with the desired prompt text:

   multillm -c <config_file> -prompt "<prompt_text>"
   

4. The application will run the specified language models concurrently, process their responses using the provided prompt, and display the results.

Implementing Your Own BaseLLM

This section will guide you through the process of creating your own BaseLLM implementation by extending the existing class and customizing it to fit the requirements of your specific language model.

Getting Started

To begin, follow these steps:

1. Create a New Python File: Start by creating a new Python file in your project directory, or within the appropriate package, where you'll define your custom BaseLLM implementation.

2. Import BaseLLM: Import the BaseLLM class from the provided code. You'll use this as the parent class for your custom implementation.

from multillm.BaseLLM import BaseLLM

Adding a new LLM interface

Now that you've imported the BaseLLM class, you can add a new LLM. Follow these steps:

1. Create Your Class: Define a new class that inherits from BaseLLM.

class NewLLM(BaseLLM):
    pass

2. Customize Attributes: Customize the attributes of your custom class to match the requirements of your language model. You can add new attributes or modify existing ones as needed.

class NewLLM(BaseLLM):
   
    def __init__(self, **kwargs):
        # Customize initialization as needed
        super().__init__(**kwargs)

3. Implement Methods: Implement the required methods: get_response() and get_content(). The get_response() method should execute your language model with the provided prompt, and the get_content() method should extract relevant content from the response.

class NewLLM(BaseLLM):
    # ... (attributes and __init__ method as before)
    
    def get_response(self, prompt):
        # Implement your language model interaction here
        response =  <"Generated response from new LLM model based on prompt">
        return response
    
    def get_content(self, response):
        # Implement content extraction from the response
        content = "Extracted content from response"
        return content

4. Usage: You can now use your custom NewLLM class in your application code. Instantiate it, call its methods, and integrate it into your application's workflow.

custom_llm = NewLLM(model="custom_model", credentials="your_credentials")
prompt = "Generate something amazing."
response = custom_llm.get_response(prompt)
content = custom_llm.get_content(response)
print(content)

By extending the provided BaseLLM class, you can easily create custom language model implementations tailored to your project's needs. This structured approach ensures consistency and modularity in your codebase, allowing you to focus on the unique aspects of your language model while leveraging the foundational structure provided by BaseLLM.

Example GPT interface

Example Model GPT.py

	
import os,sys
import openai
import json
from multillm.BaseLLM import BaseLLM
from multillm.Prompt import Prompt


# Openai gpt interface
"""
The GPT class extends the BaseModel class,  implements the required methods: get_response() and get_content().
The get_response() method takes a response parameter and returns the content of the first response in the given response object.
"""
class GPT(BaseLLM):

    #implement here
    def __init__ (self, **kwargs):
        # add values here directly or if kwargs are specified they are taken from the config file
        defaults  = {
            "class_name" : "GPT",
            "model" : "gpt-3.5-turbo",
            "credentials" : "key.json"
        }

    # Get Content -- Required Method
    def get_content(self, response):
    
        """ Get the text from the response of an LLM
        e.g.: openai returns the following response, this method should return the 'content'.
        { "choices": [{{"message": 
			{"content": "def binary_sort(arr):"}}}]}
        """
        return response["choices"][0]["message"]["content"]

    # Get Response -- Required Method, Call openai API
    def get_response(self, prompt):
        # setup prompt for API call
        messages=[]
        
        messages.append( {"role": prompt.get_role(), "content" : prompt.get_string()})
        if prompt.context:
            messages.append({"role": prompt.get_role(), "content" : prompt.get_context()})
        
        # Read Credentials file specified in the config.json, setup for openai
        if not os.path.exists(self.credentials):
            print('error (multi_llm): could not find openai_credentials: {0}' .format(self.credentials))
            return 

        # Open the file for reading
        try:
            with open(self.credentials, 'r') as file:
                # Load the JSON data from the file
                data = json.load(file)
                openai.organization = data['organization']
                openai.api_key = data['api_key'] 

        except Exception as e:
            print('(multi_llm) error: could not load credentials {0} : {1}' .format(self.credentials,str(e)))
            return
                    
        # do API call
        response = openai.ChatCompletion.create(
            model = self.model,
            messages=messages
        )
        if response:
            return(self.get_content(response))
        else:
            return response

Config JSON

Using the config.json File

The config.json file offers a convenient way to configure and load multiple language models using the "Multi_LLM" framework. This section will guide you through the process of creating and utilizing a config.json file to load and use specific language models in your application.

Configuration Setup

To get started, follow these steps to configure your config.json file:

1. Create a Configuration File: Create a new file named config.json in your project directory or the desired location.

2. Configure LLMs: Define the language models you want to use within the "llms" array. Each model configuration includes details such as the Python file containing the model implementation, class name, model name, and credentials file path.

{
    "Config": {
        "Multi_LLM": {
            "llms": [
                {
                    "file": "bard.py",
                    "class_name": "BARD",
                    "model": "chat-bison@001",
                    "credentials": "/path/to/google/key.json"
                },
                {
                    "file": "GPT.py",
                    "class_name": "GPT",
                    "model": "gpt-3.5-turbo",
                    "credentials": "/path/to/openai/key.json"
                }
            ]
        }
    }
}

Loading Models Using Multi_LLM

After creating the config.json file, you can use the "Multi_LLM" framework to load and utilize the configured models in your application. Follow these steps:

1. Instantiate Multi_LLM: Create an instance of the Multi_LLM class and provide the path to your config.json file.

from Multi_LLM import Multi_LLM

# Specify the path to your config.json file
config_path = "path/to/config.json"

# Instantiate Multi_LLM
multi_llm = Multi_LLM(config=config_path)

2. Run Models: Use the run method to run the loaded models and process their responses. Provide a prompt and an optional action chain if needed. The responses from each model will be returned in a dictionary.

prompt = "Translate this sentence."
action_chain = None  # You can define an action chain if required

responses = multi_llm.run(prompt, action_chain)
print(responses)

Action Class

This is the interface class which we use to operate on the output while still in parallel. Action class instances are define by the user and can be chained indefinitely with other Actiion instances. There are two methods that are used to interface with the class: apply() and then(). The first, apply() is what is used internally to call the methods we register. Registration is as follows:

# Write an interfacing function, a simple in/out
def capitalize(data):
	data = data.upper()
	return data
	
def lower(data):
	data = data.lower()
	return data
	
# Create the respective objects
action1 = Action(operation=capitalize)
action2 = Action(operation=lower)

# Create the chain of actions
action_chain = action1.then(action2)

What we did was create some actions that are simple I/O operations on data. We can then chain them together using the then() method mentioned above. The order of operations for then is left to right, in this case action1 will go first then action2. These actions can be tailored to your own specification, since the actions are serial, you can specify the information that is going to be passed into the function and what information will be returned.

After you have created the action chain you can pass this into the Multi_LLM.run() method and run.

Action Chain

The Action Chain is a core component of processesing LLM output. In the provided example.py we see the first action defined is for processing the LLM response and extracting the code alone from the response. This could be a first of many steps, what could follow could be saving the code to the file or extracting information through the ast module. The actions are meant to be ran on each of the model's outputs so they should be generalized for use.

Rank Class

The Rank class is identical in functionality to the Action class, though the use is different. While the Action class is used to accept and modify data in each of the models, the Rank class' purpose is to modify the final combined output of the LLMs. This output is stored in a dictionary keyed by the model's names. The Rank class's methods should be written to take this data and operate on it.

#Write an interfacing function, a simple print
def print_llms(dictionary):
	for key,val in dictionary.items():
		print(key, val)

# Create Rank instances
rank_object = Rank(operation=print_llms)

# Running the LLM, assuming an instance of 
# Multi_LLM named mLLM
results = mLLLM.run(prompt, action_chain, rank_object)

In the above code we are doing very similar actions as those seen above. The methodoly is the same. In this case we only have a single object, so instead of creating a rank_chain we simply pass in our single instantiated object.

Architecture

High Level Overview

Multi_LLM Class

This is the highest level, here we can instantiate MultiLLM objects using either config files (see config files) or manually instantiating a custom or hosted LLM. In this class we call multiple LLMs concurrently and then we can operate on the results of each in parallel using the Action Class.

The "Multi_LLM" Python code provides a versatile framework for managing and orchestrating multiple language models (LLMs) within a single application. This code is designed to enhance the efficiency of working with language models by enabling concurrent execution, response processing, and model loading from configuration files. The key features and components of the code include:

• Concurrent Model Execution: The code allows you to run multiple language models concurrently, facilitating efficient utilization of computational resources. This is particularly useful for scenarios that involve processing multiple prompts or tasks simultaneously.

• Action Chain Processing: The framework supports the concept of "action chains," which enable you to preprocess model responses using a sequence of predefined actions. This empowers you to refine and enhance the output generated by the language models.

• Model Loading from Configuration: You can load LLMs from a configuration file. This JSON-based configuration includes essential details about each model, such as the model class, associated credentials, and file paths. The code can dynamically load these models, making it easy to add new models or modify existing ones without altering the core codebase.

• Redis Integration: The code features optional integration with a Redis instance. It checks for a successful Redis connection and adjusts its behavior accordingly. If Redis connectivity is not established, the code gracefully handles the situation by setting a flag that reflects the absence of a Redis connection.

• Simplified Model Registration: The code includes a straightforward method to register models within the framework. This allows for the inclusion of custom LLM implementations while ensuring that the models are appropriately organized and accessible for execution.

By using the "Multi_LLM" Python code, developers can streamline their interactions with multiple language models, seamlessly integrating them into various applications or projects. The code promotes modularity, reusability, and parallelism in working with language models, ultimately enhancing the user experience and productivity.

BaseLLM

BaseLLM is designed to serve as the basis for implementing various language model classes. This BaseLLM class encapsulates essential attributes and methods necessary for interfacing with language models. The code establishes a structured foundation for building specific language model implementations and harmonizes their interaction within a larger application context.

Key features and components of the code include:

• Attributes for Language Models: The BaseLLM class declares a set of attributes, such as model, roles, messages, temp, api_key, max_tokens, and args, that are pertinent to language models. These attributes are meant to be customized and adapted as needed for specific model implementations.

• Customizable Initialization: The __init__() method in the BaseLLM class facilitates flexible instantiation of model instances by allowing the specification of custom values via keyword arguments. It enables the convenient configuration of attributes like name, credentials, model, and class_name.

• Placeholder Methods: The BaseLLM class defines two placeholder methods: get_response() and get_content(). The get_response() method is designed to receive a Prompt object and run the associated language model with the provided prompt. The actual implementation of this method is expected to be customized in derived classes to perform the model-specific interactions. Similarly, the get_content() method is meant to be implemented by deriving classes, providing an interface to extract relevant content from model responses.

• Structured Framework: The code encapsulates a structured framework that abstracts common functionalities of language models. By inheriting from the BaseLLM class, developers can focus on implementing model-specific interactions while leveraging the established structure for attribute handling and method placeholders.

BaseLLM aims to streamline the development of specific language model implementations by providing a consistent structure and standardized attributes. Developers can extend this base class to create custom language model classes that seamlessly integrate into the broader application ecosystem. This modular approach promotes reusability, maintainability, and consistent design patterns when working with various language models.

Contribution

You can contribute by extending the models located in models. See the BaseLLM section for more information on the necessary mechanisms for extending the BaseLLM class. For further information contact ethansaurusrex