locallab 0.2.5

A lightweight AI inference server for running models locally or in Google Colab

🚀 LocalLab

LocalLab is a powerful, lightweight AI inference server designed to deliver cutting-edge language model capabilities on your local machine or through Google Colab. It empowers developers and researchers to run sophisticated AI models on local hardware, optimizing resources with advanced features such as dynamic model loading, memory optimizations, and real-time system monitoring.

What Problem Does LocalLab Solve?

• Local Inference: Run advanced language models without relying on expensive cloud services.
• Optimized Performance: Utilize state-of-the-art techniques like quantization, attention slicing, and CPU offloading for maximum efficiency.
• Seamless Deployment: Easily switch between local deployment and Google Colab, leveraging ngrok for public accessibility.
• Effective Resource Management: Automatically monitor and manage CPU, RAM, and GPU usage to ensure smooth operation.

System Requirements

Minimum Requirements

Component	Local Deployment	Google Colab
RAM	4GB	Free tier (12GB)
CPU	2 cores	2 cores
Python	3.8+	3.8+
Storage	2GB free	-
GPU	Optional	Available in free tier

Recommended Requirements

Component	Local Deployment	Google Colab
RAM	8GB+	Pro tier (24GB)
CPU	4+ cores	Pro tier (4 cores)
Python	3.9+	3.9+
Storage	5GB+ free	-
GPU	CUDA-compatible	Pro tier GPU

Key Features

• Multiple Model Support: Pre-configured models along with the ability to load custom ones on demand.
• Advanced Optimizations: Support for FP16, INT8, and INT4 quantization, Flash Attention, and attention slicing.
• Comprehensive Logging System: Colorized console output with server status tracking, request monitoring, and performance metrics.
• Robust Resource Monitoring: Real-time insights into system performance and resource usage.
• Flexible Client Libraries: Comprehensive clients available for both Python and Node.js.
• Google Colab Friendly: Dedicated workflow for deploying via Google Colab with public URL access.

Unique Visual Overview

Below is a high-level diagram of LocalLab's architecture.

graph TD
    A["User"] --> B["LocalLab Client (Python/Node.js)"]
    B --> C["LocalLab Server"]
    C --> D["Model Manager"]
    D --> E["Hugging Face Models"]
    C --> F["Optimizations"]
    C --> G["Resource Monitoring"]

Google Colab Workflow

sequenceDiagram
    participant U as "User (Colab)"
    participant S as "LocalLab Server"
    participant N as "Ngrok Tunnel"
    U->>S: Run start_server(ngrok=True)
    S->>N: Establish public tunnel
    N->>U: Return public URL
    U->>S: Connect via public URL

Documentation & Usage Guides

For full documentation and detailed guides, please visit our documentation page.

• Getting Started Guide
• Python Client
• Node.js Client
• Client Comparison
• Google Colab Guide
• API Reference

Get Started

1. Installation:

   pip install locallab
   

2. Starting the Server Locally:

   from locallab import start_server
   start_server()
   

3. Starting the Server on Google Colab:

   !pip install locallab
   
   # Set up your ngrok auth token (REQUIRED for public access)
   # Get your free token from: https://dashboard.ngrok.com/get-started/your-authtoken
   import os
   os.environ["NGROK_AUTH_TOKEN"] = "your_token_here"
   
   # Optional: Configure model and optimizations
   os.environ["HUGGINGFACE_MODEL"] = "microsoft/phi-2"  # Choose your preferred model
   os.environ["LOCALLAB_ENABLE_QUANTIZATION"] = "true"  # Enable model optimizations
   
   # Start the server with ngrok for public access
   from locallab import start_server
   start_server(use_ngrok=True)  # Creates a public URL accessible from anywhere
   

4. Connecting your Client:

   from locallab.client import LocalLabClient
   
   # Use the ngrok URL displayed in the output above
   client = LocalLabClient("https://xxxx-xxx-xxx-xxx.ngrok.io")
   
   # Test the connection
   response = client.generate("Hello, how are you?")
   print(response)
   

Join the Community

• Report issues on our GitHub Issues.
• Participate in discussions on our Community Forum.
• Learn how to contribute by reading our Contributing Guidelines.

---

LocalLab is designed to bring the power of advanced language models directly to your workspace—efficiently, flexibly, and affordably. Give it a try and revolutionize your AI projects!