rethink-llmhub 2.3.0

LLMHub CLI — manage LLM specs and runtime configs

LLMHub CLI

A powerful command-line tool for managing LLM configurations through human-friendly specs and intelligent runtime generation.

What is LLMHub CLI?

LLMHub CLI is a development tool that separates what you want from your LLMs (preferences, constraints) from how they execute (specific models, parameters). It generates optimized runtime configurations from human-friendly specification files.

The Problem It Solves

Before LLMHub:

# Scattered, hardcoded LLM configs throughout your codebase
from openai import OpenAI

# In file1.py
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4",  # Hardcoded!
    temperature=0.7,  # Duplicated config!
    messages=[...]
)

# In file2.py - using different params for same purpose
response = client.chat.completions.create(
    model="gpt-4",  # Same model, different params
    temperature=0.5,  # Inconsistent!
    max_tokens=1000,
    messages=[...]
)

Problems:

• ❌ Models hardcoded across multiple files
• ❌ Inconsistent parameters for same use case
• ❌ Hard to swap providers (OpenAI → Anthropic)
• ❌ Different configs for dev/staging/prod environments
• ❌ No central config management

With LLMHub:

# llmhub.spec.yaml - Single source of truth
roles:
  llm.inference:
    kind: chat
    description: Main reasoning engine
    preferences:
      quality: high
      cost: medium

# Your application - clean and maintainable
from llmhub_runtime import LLMHub

hub = LLMHub(config_path="llmhub.yaml")
response = hub.completion(role="llm.inference", messages=[...])

Benefits:

• ✅ One config file, consistent behavior
• ✅ Swap models by editing YAML (no code changes)
• ✅ Environment-specific configs (dev/prod)
• ✅ Version controlled LLM decisions
• ✅ Easy testing and validation

Installation

Prerequisites

• Python 3.10 or higher
• pip or poetry

Install from PyPI

pip install rethink-llmhub

This automatically installs the required dependencies:

• llmhub-runtime - Runtime execution library
• typer - CLI framework
• rich - Beautiful terminal output
• pydantic - Data validation
• python-dotenv - Environment management

Install for Development

# Clone the repository
git clone https://github.com/your-org/llm-hub.git
cd llm-hub/packages/llmhub

# Install in editable mode
pip install -e .

Verify Installation

llmhub --version
llmhub --help

Quick Start

1. Initialize Your Project

cd your-project
llmhub init

This creates:

• llmhub.spec.yaml - Your LLM specification
• .env.example - Environment variable template

Output:

✓ Minimal spec created at llmhub.spec.yaml
✓ Environment example created at .env.example

Next steps:
  1. Edit llmhub.spec.yaml to add more roles
  2. Set OPENAI_API_KEY environment variable
  3. Run: llmhub generate

2. Configure Environment

# Copy and edit .env
cp .env.example .env

# Add your API keys
echo "OPENAI_API_KEY=sk-..." >> .env
echo "ANTHROPIC_API_KEY=sk-ant-..." >> .env

3. Generate Runtime Config

llmhub generate

This analyzes your spec and generates llmhub.yaml with:

• Optimal model selections based on preferences
• Appropriate parameters for each role
• Provider configurations

4. Test Your Setup

# Run health check
llmhub doctor

# Test a specific role
llmhub test --role llm.inference --prompt "Hello, world!"

5. Use in Your Application

from llmhub_runtime import LLMHub

# Initialize hub with generated config
hub = LLMHub(config_path="llmhub.yaml")

# Call by role name
response = hub.completion(
    role="llm.inference",
    messages=[{"role": "user", "content": "Explain AI"}]
)

print(response)

Core Concepts

Spec vs Runtime

Spec (llmhub.spec.yaml) - What you want:

roles:
  llm.summarize:
    kind: chat
    description: Summarize long documents
    preferences:
      cost: low        # Prefer cheaper models
      latency: low     # Prefer faster models
      quality: medium  # Good enough quality

Runtime (llmhub.yaml) - How it runs:

roles:
  llm.summarize:
    provider: openai
    model: gpt-4o-mini  # Selected based on preferences
    mode: chat
    params:
      temperature: 0.3
      max_tokens: 1024

Role-Based Design

Instead of calling specific models, you call logical roles:

# ❌ Tightly coupled
response = openai.chat(model="gpt-4", ...)

# ✅ Loosely coupled
response = hub.completion(role="llm.inference", ...)

Benefits:

• Swap models without code changes
• Consistent behavior for same purpose
• Environment-specific configurations
• Easier testing and mocking

Preference-Based Selection

Define what you need, not which model:

llm.analytics:
  kind: chat
  preferences:
    quality: high    # Prioritize accuracy
    cost: low        # But keep costs down
    latency: medium  # Response time is okay
    providers: [openai, anthropic]  # Allowed providers

The generator selects the best model matching your criteria.

Configuration Files

llmhub.spec.yaml

Human-friendly specification of your LLM needs:

project: my-app
env: production

providers:
  openai:
    enabled: true
    env_key: OPENAI_API_KEY
  anthropic:
    enabled: true
    env_key: ANTHROPIC_API_KEY

roles:
  llm.preprocess:
    kind: chat
    description: Clean and normalize user input
    preferences:
      cost: low
      latency: low
      quality: medium
  
  llm.inference:
    kind: chat
    description: Main reasoning and response generation
    preferences:
      quality: high
      cost: medium
      providers: [anthropic, openai]
  
  llm.embedding:
    kind: embedding
    description: Generate embeddings for search
    preferences:
      cost: low
      quality: medium

defaults:
  providers: [openai]

llmhub.yaml

Machine-optimized runtime configuration:

project: my-app
env: production

providers:
  openai:
    env_key: OPENAI_API_KEY
  anthropic:
    env_key: ANTHROPIC_API_KEY

roles:
  llm.preprocess:
    provider: openai
    model: gpt-4o-mini
    mode: chat
    params:
      temperature: 0.2
      max_tokens: 512
  
  llm.inference:
    provider: anthropic
    model: claude-3-5-sonnet-20241022
    mode: chat
    params:
      temperature: 0.7
      max_tokens: 2048
  
  llm.embedding:
    provider: openai
    model: text-embedding-3-small
    mode: embedding
    params: {}

CLI Commands

Project Setup

# Quick initialization with defaults
llmhub init

# Interactive setup with guided questions
llmhub setup

# Check project status
llmhub status

# Show resolved file paths
llmhub path

Spec Management

# View current spec
llmhub spec show

# Validate spec file
llmhub spec validate

# List all roles
llmhub roles

# Add a new role
llmhub add-role llm.translate

# Edit existing role
llmhub edit-role llm.inference

# Remove a role
llmhub rm-role llm.old-role

Runtime Generation

# Generate runtime from spec
llmhub generate

# Dry run (preview without writing)
llmhub generate --dry-run

# Force overwrite existing runtime
llmhub generate --force

# Show model selection explanations
llmhub generate --explain

# View generated runtime
llmhub runtime show

# Compare spec vs runtime
llmhub runtime diff

Environment Management

# Sync .env.example from spec
llmhub env sync

# Check for missing environment variables
llmhub env check

# Check with custom .env file
llmhub env check --env-file .env.production

Testing & Validation

# Test a role interactively
llmhub test

# Test specific role with prompt
llmhub test --role llm.inference --prompt "Hello!"

# Output raw JSON response
llmhub test --role llm.inference --json

# Run comprehensive health check
llmhub doctor

# Health check without network calls
llmhub doctor --no-network

Advanced Usage

Multiple Environments

Maintain separate configs for different environments:

# Development
llmhub generate --dry-run > llmhub.dev.yaml

# Production
llmhub generate --force > llmhub.prod.yaml

In your application:

import os
env = os.getenv("ENV", "dev")
hub = LLMHub(config_path=f"llmhub.{env}.yaml")

Forcing Specific Models

Override generator with explicit model choices:

roles:
  llm.critical:
    kind: chat
    description: Critical production workload
    force_provider: anthropic
    force_model: claude-3-5-sonnet-20241022
    preferences:
      quality: high

Custom Parameters

Pass model-specific parameters:

roles:
  llm.creative:
    kind: chat
    description: Creative writing assistant
    mode_params:
      temperature: 0.9
      top_p: 0.95
      presence_penalty: 0.6
      frequency_penalty: 0.3

Integration with CI/CD

# .github/workflows/validate-llm-config.yml
name: Validate LLM Config

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: '3.10'
      - run: pip install rethink-llmhub
      - run: llmhub spec validate
      - run: llmhub env check

Workflow Examples

Example 1: Adding a New Feature

You need to add translation functionality:

# 1. Add role to spec
llmhub add-role llm.translate
# Select: kind=chat, cost=low, quality=high

# 2. Regenerate runtime
llmhub generate

# 3. Test it
llmhub test --role llm.translate --prompt "Translate 'Hello' to Spanish"

# 4. Use in code

response = hub.completion(
    role="llm.translate",
    messages=[{
        "role": "user",
        "content": "Translate 'Hello World' to French"
    }]
)

Example 2: Swapping Providers

Switch from OpenAI to Anthropic for main inference:

# 1. Edit spec (just change preferences)
llmhub edit-role llm.inference
# Update: providers=[anthropic]

# 2. Regenerate
llmhub generate

# 3. Verify
llmhub runtime show

# Application code remains unchanged!

Example 3: Cost Optimization

Reduce costs by using cheaper models where quality isn't critical:

# 1. Edit roles in spec
# Change: llm.preprocess → cost: low
# Change: llm.summarize → cost: low

# 2. Regenerate
llmhub generate --explain

# 3. Review changes
llmhub runtime diff

# 4. Test to ensure quality is acceptable
llmhub test --role llm.preprocess

Troubleshooting

Common Issues

Issue: "Spec file not found"

# Initialize project first
llmhub init

Issue: "Missing environment variable"

# Check what's missing
llmhub env check

# Add to .env
echo "OPENAI_API_KEY=sk-..." >> .env

Issue: "Runtime file not found"

# Generate runtime from spec
llmhub generate

Issue: "Unknown role error"

# List available roles
llmhub roles

# Add missing role
llmhub add-role your-role-name

Debug Mode

For verbose output:

# Check file paths
llmhub path

# Validate configurations
llmhub spec validate
llmhub doctor

# Test with dry-run
llmhub generate --dry-run --explain

Best Practices

1. Version Control

# Track these files
git add llmhub.spec.yaml
git add llmhub.yaml
git add .env.example

# Don't track these
echo ".env" >> .gitignore

2. Environment Variables

• Use .env.example for documentation
• Never commit actual .env files
• Use different keys for dev/prod

3. Role Naming

# ✅ Good - descriptive, hierarchical
llm.user.summarize
llm.admin.analytics
llm.public.search

# ❌ Bad - vague, flat
summarizer
model1
gpt

4. Regular Validation

# Add to pre-commit hook
llmhub spec validate
llmhub env check --env-file .env.example

5. Testing

# Test critical roles before deployment
llmhub test --role llm.inference
llmhub test --role llm.embedding
llmhub doctor --no-network  # CI/CD

Architecture

LLMHub CLI is built with modularity in mind:

llmhub/
├── context.py          # Project context resolution
├── spec_models.py      # Spec schema & validation
├── runtime_io.py       # Runtime config I/O
├── env_manager.py      # Environment management
├── ux.py              # CLI output & prompts
├── generator_hook.py   # Spec → Runtime generation
├── commands/          # Command implementations
│   ├── setup_cmd.py
│   ├── spec_cmd.py
│   ├── runtime_cmd.py
│   ├── env_cmd.py
│   └── test_cmd.py
└── cli.py             # Main CLI entry point

Contributing

We welcome contributions! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes with tests
4. Run tests: pytest tests/
5. Submit a pull request

Roadmap

• Async API support
• Streaming interface
• Model performance analytics
• Cost tracking and budgets
• Multi-project management
• Web UI for configuration
• Integration with popular frameworks

Support

• Documentation: GitHub Wiki
• Issues: GitHub Issues
• Discussions: GitHub Discussions

License

MIT License - see LICENSE file for details.

---

Made with ❤️ for developers building with LLMs