rethink-llmhub-runtime 2.0.0

A lightweight runtime for LLMHub that delegates to any-llm

llmhub_runtime

llmhub_runtime is a small Python library that lets you call LLMs by role using a simple YAML config (llmhub.yaml), while delegating all provider-specific logic to any-llm.

It is designed to be:

• Runtime-light – minimal dependencies, no discovery logic.
• Provider-agnostic – supports any provider that any-llm supports.
• Role-centric – your application code never handles provider/model strings directly.

llmhub_runtime is intended for:

• Application backends (e.g. memory systems, agents, tools).
• The future llmhub CLI/Web tool, which will generate llmhub.yaml and then use this runtime internally.

Installation

pip install llmhub-runtime any-llm-sdk

(Exact package name to be confirmed when publishing.)

Runtime Config: llmhub.yaml

llmhub_runtime reads a generated config file, typically named llmhub.yaml:

project: memory
env: dev

providers:
  openai:
    env_key: OPENAI_API_KEY
  anthropic:
    env_key: ANTHROPIC_API_KEY

roles:
  llm.preprocess:
    provider: openai
    model: gpt-4o-mini
    mode: chat
    params:
      temperature: 0.2
      max_tokens: 512

  llm.inference:
    provider: anthropic
    model: claude-3-5-sonnet-20241022
    mode: chat
    params:
      temperature: 0.7
      max_tokens: 2048

You typically do not edit this by hand; it is generated by higher-level tools (e.g. llmhub CLI/Web).

Basic Usage

from llmhub_runtime.hub import LLMHub

hub = LLMHub(config_path="llmhub.yaml")

response = hub.completion(
    role="llm.inference",
    messages=[{"role": "user", "content": "Hello"}],
)

print(response)

Embeddings:

embedding = hub.embedding(
    role="llm.embedding",
    input="Hello world",
)

To override parameters per call:

response = hub.completion(
    role="llm.inference",
    messages=[...],
    params_override={"temperature": 0.1},
)

Architecture Overview

llmhub_runtime is intentionally small and has three main layers:

1. Config layer

   • models.py – Pydantic models for RuntimeConfig, ProviderConfig, RoleConfig, ResolvedCall.
   • config_loader.py – loads and validates llmhub.yaml.

2. Resolution layer

   • resolver.py – maps a logical role name to {provider, model, mode, params}, with optional fallback from defaults.

3. Execution layer

   • hub.py – exposes the LLMHub class:
     • Resolves roles.
     • Calls any-llm (completion / embedding) with the resolved settings.
     • Optional hooks for logging/metrics.

All domain-specific errors live in errors.py.

Design Principles

• No provider logic – llmhub_runtime never talks to provider SDKs directly; it only calls any-llm.
• No discovery or scoring – it assumes llmhub.yaml already contains concrete provider/model choices.
• Role-first – application code only sees role names; you can swap models by editing/generating llmhub.yaml without changing app code.

Roadmap

• Async APIs (acompletion, aembedding).
• Streaming interfaces.
• More modes (image, audio, tool).
• Tight integration with the llmhub CLI/Web for config generation.