polarisllm 2.0.3

🌟 The Ultimate Multi-Model LLM Runtime Platform - Deploy, manage, and serve 300+ language models with OpenAI-compatible APIs. Built on ms-swift for production-ready performance.

🌟 PolarisLLM - AI Model Orchestration Platform

Deploy and manage 300+ AI models with simple commands

Transform your server into a powerful AI platform. Deploy models in the background, manage them with ease, and access everything through OpenAI-compatible APIs.

---

✨ What You Get

🚀 Background Model Deployment - Models run automatically in the background
🎛️ Simple Management - Start, stop, and monitor with easy commands
📊 Real-time Monitoring - See status, memory usage, and live logs
🔌 OpenAI Compatible - Works with existing OpenAI code
🌐 300+ Models - Qwen, Llama, DeepSeek, Mistral, and more

---

🚀 Get Started

Install

pip install polarisllm --upgrade

Start Server

polarisllm start --daemon

🌟 PolarisLLM Runtime Engine
==================================================
🚀 Starting PolarisLLM server in daemon mode...
   Host: 0.0.0.0
   Port: 7860
   Log File: /home/user/.polarisllm/logs/server.log

✅ Server started successfully!
   PID: 12345
   URL: http://0.0.0.0:7860

💡 Commands:
   polarisllm status              # Check server status
   polarisllm logs --server       # View server logs
   polarisllm stop --server       # Stop server

Deploy Your First Model

polarisllm deploy --model qwen2.5-7b-instruct

🚀 Deploying model: qwen2.5-7b-instruct
📋 Using convenience shortcut for qwen2.5-7b-instruct
   Model Type: qwen2_5
   Model ID: Qwen/Qwen2.5-7B-Instruct

📡 Allocated port: 8000
🔧 Command: swift deploy --model_type qwen2_5 --model Qwen/Qwen2.5-7B-Instruct --port 8000 --host 0.0.0.0
📝 Logs: /home/user/.polarisllm/logs/qwen2.5-7b-instruct.log

🚀 Starting deployment in background...
✅ Started process 12346 for qwen2.5-7b-instruct
✅ Model deployment started successfully!
   Name: qwen2.5-7b-instruct
   PID: 12346
   Port: 8000
   Status: Initializing...

🔍 Monitor with: polarisllm logs qwen2.5-7b-instruct --follow
📊 Check status: polarisllm status
🌐 Access via: http://localhost:7860/v1/chat/completions

Check What's Running

polarisllm list

📋 Deployed Models
========================================================================
NAME                    STATUS      PORT    MEMORY   UPTIME    TYPE
qwen2.5-7b-instruct     🟢 Running  8000    15.2%    2.5h      qwen2_5

📊 Summary:
   Total Models: 1
   Running: 1
   Stopped: 0

💡 Commands:
   polarisllm logs qwen2.5-7b-instruct --follow  # View live logs
   polarisllm stop qwen2.5-7b-instruct           # Stop a model
   polarisllm status                              # Detailed status

---

🎮 Common Commands

Deploy Models

# Popular models (shortcuts available)
polarisllm deploy --model qwen2.5-7b-instruct
polarisllm deploy --model deepseek-coder-6.7b
polarisllm deploy --model mistral-7b-instruct

# Any model with full name
polarisllm deploy --model my-llama \
  --model-type llama3_1 \
  --model-id meta-llama/Meta-Llama-3.1-8B-Instruct

Sample deployment output:

🚀 Deploying model: deepseek-coder-6.7b
📋 Using convenience shortcut for deepseek-coder-6.7b
   Model Type: deepseek
   Model ID: deepseek-ai/deepseek-coder-6.7b-instruct

📡 Allocated port: 8001
🚀 Starting deployment in background...
✅ Model deployed successfully on port 8001!

Manage Your Models

polarisllm list                    # See all models
polarisllm status                  # System overview
polarisllm stop qwen2.5-7b-instruct    # Stop a model
polarisllm undeploy qwen2.5-7b-instruct # Remove completely

Status output:

$ polarisllm status
📊 PolarisLLM System Status
============================================================
🖥️  Server Status:
   Status: 🟢 Running (PID: 12345)
   Memory: 2.1%
   CPU: 0.5%
   API: 🟢 Healthy
   URL: http://localhost:7860

🤖 Models Status:
   Total Models: 2
   Running: 2 🟢
   Stopped: 0 🔴
   Detailed Status:
     qwen2.5-7b-instruct: 🟢 running
       Port: 8000, Memory: 15.2%, Uptime: 2.5h
     deepseek-coder-6.7b: 🟢 running
       Port: 8001, Memory: 12.8%, Uptime: 1.2h

💾 Resource Status:
   Ports: 2/100 used (98 available)
   Range: 8000-8100
   Total Memory: 28.0% (all models combined)

💡 Quick Commands:
   polarisllm deploy --model <name>     # Deploy a model
   polarisllm list                      # List all models
   polarisllm logs <model> --follow     # View live logs
   polarisllm stop <model>              # Stop a model
   polarisllm start --daemon            # Start server in background

Watch Logs

polarisllm logs qwen2.5-7b-instruct --follow    # Live logs
polarisllm logs --server --follow               # Server logs

Sample log output:

📝 Logs for model: qwen2.5-7b-instruct
   Lines: 100
   Follow: Yes
============================================================
🔄 Streaming logs (Press Ctrl+C to stop)...

[INFO:swift] Successfully registered model
[INFO:swift] rank: -1, local_rank: -1, world_size: 1
[INFO:swift] Loading the model using model_dir: /cache/Qwen2___5-7B-Instruct
[INFO:swift] Loading model weights...
[INFO:swift] Model loaded successfully
[INFO:swift] Server started on http://0.0.0.0:8000
[INFO:swift] Waiting for requests...

Server Control

polarisllm start --daemon     # Start in background
polarisllm stop --server      # Stop server
polarisllm restart           # Restart everything

---

🤖 Available Models

Popular Shortcuts:

• qwen2.5-7b-instruct - Great all-around chat model
• qwen2.5-14b-instruct - Larger version for better responses
• deepseek-coder-6.7b - Excellent for programming
• deepseek-vl-7b-chat - Understands images and text
• mistral-7b-instruct - Fast and efficient
• llama3.1-8b-instruct - Meta's latest model

Categories:

• Chat: General conversation and Q&A
• Code: Programming and development help
• Vision: Image understanding and analysis
• Audio: Speech and sound processing

See all 300+ models: python -m swift list-models

---

🔌 Use with Your Code

Python

import openai

client = openai.OpenAI(
    base_url="http://localhost:7860/v1",
    api_key="not-required"
)

response = client.chat.completions.create(
    model="qwen2.5-7b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Sample response:

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "qwen2.5-7b-instruct",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Hello! I'm an AI assistant powered by PolarisLLM. How can I help you today?"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 9,
    "completion_tokens": 20,
    "total_tokens": 29
  }
}

JavaScript

import OpenAI from 'openai';

const client = new OpenAI({
    baseURL: 'http://localhost:7860/v1',
    apiKey: 'not-required'
});

const completion = await client.chat.completions.create({
    model: 'qwen2.5-7b-instruct',
    messages: [{ role: 'user', content: 'Hello!' }]
});

cURL

curl -X POST "http://localhost:7860/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-7b-instruct",
    "messages": [{"role": "user", "content": "Write a Python function to add two numbers"}]
  }'

Sample cURL response:

{
  "id": "chatcmpl-456",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "qwen2.5-7b-instruct",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Here's a simple Python function to add two numbers:\n\n```python\ndef add_numbers(a, b):\n    return a + b\n\n# Example usage\nresult = add_numbers(5, 3)\nprint(result)  # Output: 8\n```"
    },
    "finish_reason": "stop"
  }]
}

---

📊 See What's Running

$ polarisllm list
📋 Deployed Models
========================================================================
NAME                    STATUS      PORT    MEMORY   UPTIME    TYPE
qwen2.5-7b-instruct     🟢 Running  8000    15.2%    2.5h      qwen2_5
deepseek-coder-6.7b     🟢 Running  8001    12.8%    1.2h      deepseek
mistral-7b-instruct     🔴 Stopped  8002    N/A      N/A       mistral

$ polarisllm status
📊 System Status
===========================================
🖥️  Server: 🟢 Running at http://localhost:7860
🤖 Models: 2 running, 1 stopped
💾 Resources: 2/100 ports used, 28% memory

---

🚫 Fix Common Issues

Model won't start?

polarisllm logs <model-name>    # Check what went wrong
polarisllm cleanup              # Clean up any stuck processes

Sample error log:

📝 Logs for model: qwen2.5-7b-instruct
============================================================
[ERROR:swift] CUDA out of memory. Tried to allocate 2.0 GiB
[INFO:swift] Try reducing batch size or using a smaller model
[ERROR:swift] Model loading failed

Server not working?

polarisllm status               # See what's happening
polarisllm restart              # Restart everything

Need to free up space?

polarisllm stop --all           # Stop all models
polarisllm cleanup              # Clean up old processes

Sample cleanup output:

🧹 Cleaning up PolarisLLM...
========================================
🔍 Cleaning up dead processes...
🔍 Cleaning up dead models...
🔍 Cleaning up port allocations...
Cleaned up 2 dead port allocations
🔍 Cleaning up old logs...
Cleaned up 3 old log files

✅ Cleanup completed!
💡 Use 'polarisllm status' to verify system state

---

💡 Pro Tips

• Models run in background automatically - they survive terminal restarts
• Use --follow with logs to watch models start up in real-time
• Each model gets its own port (8000, 8001, 8002...)
• Server remembers your models even after restarts
• Use shortcuts for popular models, full names for everything else

---

🤝 Need Help?

• GitHub Issues: Report bugs or request features
• PyPI Package: Install from here

---

📄 License

MIT License - Free to use for any purpose