ext_llm - a wrapper library for common LLMs (WIP)

Installation

pip install ext_llm

Usage

from concurrent.futures import Future
import ext_llm
from ext_llm.llm.response import Response
from ext_llm.llm.stream import Stream

#read config yaml file
config : str = open("config.yaml").read()

#initialize extllm library
extllm = ext_llm.init(config)

#request a client via user defined presets. In this instance it's "groq-llama"
llm_client = extllm.get_client("groq-llama")

#you can request e concurrent client, based on the same preset. This will allow for non blocking concurrent requests.
llm_concurrent_client = extllm.get_concurrent_client("groq-llama")

#Non blocking call. This will return a future object that will be resolved when the request is completed.
future1: Future [Response | Stream] = llm_concurrent_client.generate_text("You are an helpful assistant", "Recite the first article of the Italian Constitution")

#Blocking call. This will return the result of the request. The result can be either a Response or a Stream object.
#Response or Stream are defined by the user in the config file.
#If the "invocation_method" is "converse" then the result is a Response.
#If the "invocation_method" is "converse_stream" then the result is a Stream.
#A stream has to be handled differently from a response.
result: Response | Stream = llm_client.generate_text("You are an helpful assistant", "Recite the first amendment of the American constitution")


print(result.metadata)
print(result)

print(future1.result().metadata)
print(future1.result())

Config File

Here's an example of a config file:

config:
  logging: true #false
  db_path: logs.db
  presets:
      #User can define multiple presets
      aws-claude-sonnet:
        #available providers: aws, groq
        provider: aws
        #secrets are passed to the library by specifying environment variables names
        aws_access_key_id_variable_name: AWS_ACCESS_KEY_ID
        aws_secret_access_key_variable_name: AWS_SECRET_ACCESS_KEY
        aws_region: eu-west-2
        model_id: anthropic.claude-3-sonnet-20240229-v1:0
        invocation_method: converse_stream
        temperature: 0.5
        max_tokens: 1000

      groq-llama:
        provider: groq
        model_id: deepseek-r1-distill-llama-70b
        groq_api_key_variable_name: GROQ_API_KEY
        invocation_method: converse
        temperature: 0.5
        max_tokens: 500

      groq-llama-streaming:
        provider: groq
        model_id: deepseek-r1-distill-llama-70b
        groq_api_key_variable_name: GROQ_API_KEY
        invocation_method: converse_stream
        temperature: 0.5
        max_tokens: 500

      custom-provider:
        #the library can instantiate custom classes that inherit from the Llm class
        #this can be done by specifying the class name and the module name
        class_name: MyLlm
        module_name: custom_provider_llm
        #these parameters are passed to the custom class, so the user can define custom parameters
        custom_param1: 123
        custom_param2: 456

Defining a custom LLM Client

To define a custom provider that can be used by the ext_llm library, you need to create a class that inherits from the LlmClient class and implements the required methods. Additionally, if your custom provider supports streaming, you should also define a stream class that inherits from the Stream class.

Step-by-Step Guide

1. Create the Custom LLM Client Class:

   • Inherit from the LlmClient class.
   • Implement the generate_text method.
   • Implement the get_config method.

2. Create the Custom Stream Class (if needed):

   • Inherit from the Stream class.
   • Implement the __iter__ method to handle streaming responses.

3. Update the Configuration File:

   • Specify the custom class name and module name in the configuration file.
   • Add any custom parameters required by your custom class.

Example

Custom LLM Client Class

# File: custom_provider_llm.py
from ext_llm import LlmClient

class MyLlmClient(LlmClient):
    def __init__(self, config: dict, preset_name: str):
        super().__init__()
        self.preset_name = preset_name
        self.config = config
        self.custom_param1 = config['custom_param1']
        self.custom_param2 = config['custom_param2']
        

    def generate_text(self, system_prompt: str, prompt: str, max_tokens=None, temperature=None):
        return "Hello, world!"
    
    def get_config(self):
        return self.config

Custom Stream Class (if needed)

# File: custom_provider_llm.py
from ext_llm.llm.stream import Stream
from ext_llm.llm.response import Response

class MyStream(Stream):
    def __init__(self, stream):
        super().__init__()
        self.stream = stream

    def __iter__(self):
        for chunk in self.stream:
            yield Response(chunk, {})

Configuration File

config:
  logging: true
  db_path: logs.db
  presets:
      custom-provider:
        class_name: MyLlmClient
        module_name: custom_provider_llm
        custom_param1: 123
        custom_param2: 456

By following these steps, you can define a custom provider that integrates seamlessly with the ext_llm library.