llm-connect 2.0.0

LLM Connect API

LLMConnect API

Table of Contents

• Introduction
• Features
• Installation
  • Prerequisites
• Commands
• Predefined tasks
  • Command
  • Output
• Command examples
• Author
• License

Introduction

LLMConnect API is a developer-friendly Python-based CLI utility designed to manage and evaluate Language Models including LLMs and SLMs on local servers or clusters. It enables users to run a variety of standard and custom tasks with popular models such as Llama-2, Mistral, Falcon, etc., and also supports the integration of new LLMs.

Features

• Task Versatility: Execute predefined tasks like NER, Sentiment Analysis, Summarisation, or craft your own.
• Model Selection: Choose and add your custom LLMs and SLMs from HuggingFace.
• Adaptable Environments: Operate seamlessly on local servers and extend to local network clusters.
• Hardware Compatibility: Ensure efficient LLM functioning with GPU compatibility checks and memory monitoring.

CLI Interface:

• Navigate tasks, models, and hardware diagnostics with simple, intuitive commands.

Development & Security:

• Developed in Python 3.x, emphasising seamless LLM integration, and detailed documentation.
• Features enhanced input validation for secure, reliable operations.

Deployment:

• Eager to experience the power of Large Language Models through a Python-based Command Line Interface? LLMConnect API is your gateway to harnessing this technology on your local systems!

Installation

Required libraries

• Python 3.10
• click==8.1.7
• setuptools~=68.2.0
• transformers
• torch~=2.1.0
• accelerate
• bitsandbytes
• colorama

Commands

• lc list

List all available tasks or models.

  Usage:
    lc list [OPTIONS] COMMAND [ARGS]

  Options:
    -h, --help  Show this message and exit.

  Commands:
    models  List available models
    tasks   List available tasks

• lc add

Add new Hugging Face Model. 

  Usage:
    lc add [OPTIONS]
    Model format: repo_id/model_id

  Options:
    --model TEXT  [required]
    -h, --help    Show this message and exit.

• lc remove

  Remove an existing HuggingFace Model.

  Usage:
    lc remove [OPTIONS]
    Model format: repo_id/model_id

  Options:
    --model TEXT  [required]
    -h, --help    Show this message and exit.

• lc hardware

Check hardware compatibility for given Hugging Face model.

  Usage: 
    lc hardware [OPTIONS]
    Model format: repo_id/model_id.

  Options:
    --model TEXT  Model name in format: repoID/modelID  [required]
    -h, --help    Show this message and exit.

• lc exec

Execute an input prompt with given model and given task.

  Usage: 
    lc exec [OPTIONS]

  Options:
    --task TEXT   Specify the task name  [required]
    --model TEXT  Specify the model name (repoID/modelID)  [required]
    --input TEXT  Specify input text (optional)
    -h, --help    Show this message and exit.

• lc fetch

Fetch the logs of previous sessions.

  Usage:
    lc fetch [OPTIONS]

  Options:
    -h, --help  Show this message and exit.

Predefined tasks

Command

lc list tasks

Output

Available Tasks:

• NER
• Summary
• AnalyseSentiment
• DetectBias
• TagTopic
• Custom

Command examples

• lc list models
• lc list tasks
• lc add --model ceadar-ie/FinanceConnect-13B
• lc remove --model ceadar-ie/FinanceConnect-13B
• lc hardware --model ceadar-ie/FinanceConnect-13B
• lc exec --task NER --model ceadar-ie/FinanceConnect-13B --input "Hi! I'm LLMConnect API"
• lc fetch

Code Repository

LLM Connect API - GitLab

Author

CeADAR Connect Group

License

APACHE 2.0