LLM-Pool

This simple project is to manage multiple LLM(Large Language Model)s in one place. Because there are too many fine-tuned LLMs, and it is hard to evaluate which one is bettern than others, it might be useful to test as many models as possible. Below is the two useful usecases that I had in mind when kicking off this project.

• compare generated text from different models side by side
• complete conversation in collaboration of different models

Usecase

from llmpool import LLModelPool
from llmpool import LocalLoRAModel
from llmpool import RemoteTxtGenIfLLModel

from transformers import AutoModelForCausalLM

model_pool = LLModelPool()
model_pool.add_models(
  # alpaca-lora 13b
  LocalLoRALLModel(
    "alpaca-lora-13b",
    "elinas/llama-13b-hf-transformers-4.29",
    "LLMs/Alpaca-LoRA-EvolInstruct-13B",
    model_cls=AutoModelForCausalLM
  ),
  
  RemoteTxtGenIfLLModel(
    "stable-vicuna-13b",
    "https://...:8080"
  ),
)

for model in model_pool:
  result = model.batch_gen(
    ["hello world"], 
    GenerationConfig(...)
  )
  print(result)
  
  _, stream_result = model.stream_gen(
    "hello world",
    GenerationConfig(...)
  )

  for ret in stream_results:
    if instanceof(model, LocalLoRALLModel) or \
      instanceof(model, LocalLLModel):
      print(ret, end='')
    else:
      print(ret.token.text, end='')

Alternatively, you can organize the model pool with yaml file

from llmpool import instantiate_models

model_pool = instantiate_models('...yaml')

Todo

• Add example notebooks
• Yaml parser to add models to model pool