multi-temp-lm 0.1.0

Intelligent temperature routing for LLM queries using local embeddings

multi-temp-lm

Intelligent temperature routing for LLM queries using local embeddings.

multi-temp-lm provides a callable MultiTempRouter object that automatically classifies query intent and selects the optimal temperature value for any LLM SDK. Zero configuration, zero API calls, and universal SDK compatibility.

Quick Start

from multi_temp_lm import MultiTempRouter
import litellm

router = MultiTempRouter()

response = litellm.completion(
    model="gpt-4",
    messages=[{"role": "user", "content": "Write a poem about stars"}],
    temperature=router,  # Auto-selects optimal temperature
)

Installation

pip install multi-temp-lm

Dependencies:

• sentence-transformers (for embeddings; model downloads on first use ~22MB with ONNX)
• numpy
• onnxruntime (optional, for quantized inference)

Features

• Intent-aware temperature routing — Automatically classifies query type and selects optimal temperature
• Local embeddings — Uses sentence-transformers with ONNX quantization (~22MB), no API calls
• Universal SDK compatibility — Works with LiteLLM, OpenAI, Anthropic, Google, Cohere, Ollama, and more
• Async support — Non-blocking with asyncio.to_thread()
• LRU cache — Repeated queries return instantly
• Customizable — Override temperatures, add categories, or plug in your own classifier
• Type annotated — Ships with py.typed marker for full mypy support

How It Works

multi-temp-lm uses a small, local sentence-transformer model (all-MiniLM-L6-v2) to classify the intent of every user query. Based on the detected category, it returns a temperature value optimized for that task:

• Factual / analytical queries → low temperature (deterministic, precise)
• Creative / coding queries → moderate temperature (balanced coherence and diversity)
• Open-ended chat → default temperature

The router runs entirely on your machine. It does not send any data to external APIs for classification.

Default Temperatures

Category	Temperature	Rationale
factual	0.2	High determinism for facts and recall
analytical	0.2	Low variance for structured analysis
creative	0.9	High diversity for poems, stories, and brainstorming
coding	0.3	Focused output for code generation and debugging
reasoning	0.4	Step-by-step logic with controlled creativity
summarization	0.3	Consistent, concise summaries
conversational	0.7	Natural, balanced dialogue
default	0.7	Fallback for unrecognized queries

Override any of these with the temperature_map constructor argument.

Usage Examples

LiteLLM (sync)

import litellm
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()

response = litellm.completion(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain quantum mechanics."}],
    temperature=router,
)

LiteLLM (async)

response = await litellm.acompletion(
    model="gpt-4",
    messages=messages,
    temperature=await router.aroute(messages),
)

OpenAI SDK (sync)

from openai import OpenAI
from multi_temp_lm import MultiTempRouter

client = OpenAI()
router = MultiTempRouter()

response = client.chat.completions.create(
    model="gpt-4",
    messages=messages,
    temperature=router,
)

OpenAI SDK (async)

from openai import AsyncOpenAI

client = AsyncOpenAI()

response = await client.chat.completions.create(
    model="gpt-4",
    messages=messages,
    temperature=await router.aroute(messages),
)

Anthropic

The Anthropic SDK does not accept callable temperature values. Use router.for_messages(messages) for explicit pre-computation:

import anthropic
from multi_temp_lm import MultiTempRouter

client = anthropic.Anthropic()
router = MultiTempRouter()

response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=messages,
    temperature=router.for_messages(messages),
)

Google Vertex AI

import vertexai
from vertexai.generative_models import GenerativeModel
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()
model = GenerativeModel("gemini-1.5-flash")

response = model.generate_content(
    messages,
    generation_config={"temperature": router.for_messages(messages)},
)

Cohere

import cohere
from multi_temp_lm import MultiTempRouter

client = cohere.Client()
router = MultiTempRouter()

response = client.chat(
    model="command-r",
    message="Explain quantum mechanics.",
    temperature=router.for_messages(messages),
)

Ollama

import ollama
from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()

response = ollama.chat(
    model="llama3",
    messages=messages,
    options={"temperature": router.for_messages(messages)},
)

Generic for_messages() Pattern

If your SDK does not support callable temperature, use the explicit pre-computation pattern. This works with any SDK that accepts a numeric temperature:

from multi_temp_lm import MultiTempRouter

router = MultiTempRouter()

temperature = router.for_messages(messages)
# Pass `temperature` to any SDK call

API Reference

MultiTempRouter.__init__

MultiTempRouter(
    model_name: str = "all-MiniLM-L6-v2",
    temperature_map: Optional[Dict[str, float]] = None,
    references: Optional[List[ReferenceExample]] = None,
    classifier: Optional[Callable[[str], str]] = None,
    default_temperature: float = 0.7,
    override_temperature: Optional[float] = None,
    debug: bool = False,
    cache_size: int = 128,
)

Parameter	Type	Default	Description
model_name	str	"all-MiniLM-L6-v2"	Sentence-transformers model to use for embeddings
temperature_map	Dict[str, float]	None	Override default category-to-temperature mapping
references	List[ReferenceExample]	None	Custom labeled reference examples for classification
classifier	Callable[[str], str]	None	Custom classifier callable replacing embeddings logic
default_temperature	float	0.7	Fallback temperature for unknown categories and __float__
override_temperature	float	None	If set, bypasses classification and always returns this value
debug	bool	False	If True, stores debug info on each call
cache_size	int	128	Maximum size of the LRU query cache

__call__(messages)

router(messages: List[Message]) -> float

Classifies the query in messages and returns the optimal temperature. This is the primary entry point for SDKs that accept callable temperature (e.g., LiteLLM, OpenAI).

for_messages(messages)

router.for_messages(messages: List[Message]) -> float

Explicit pre-computation. Returns a plain float that any SDK will accept. This is the universal fallback for SDKs that do not support callable temperature.

aroute(messages)

await router.aroute(messages: List[Message]) -> float

Async route using asyncio.to_thread(). Does not block the event loop under concurrent load. Use this for async SDK patterns.

__float__()

float(router) -> float

Returns default_temperature. Used as a last-resort fallback when an SDK only accepts numeric values and does not support callables.

classify(query_text)

router.classify(query_text: str) -> str

Classifies raw query text into an intent category (e.g., "factual", "creative", "coding"). Uses the embedding-based classifier if available, otherwise falls back to a keyword heuristic.

get_debug_info()

router.get_debug_info() -> Optional[DebugInfo]

Returns a DebugInfo dataclass with the last classification result if debug=True was passed to the constructor. Returns None otherwise.

DebugInfo fields:

• query_text: str — The extracted user query
• detected_category: str — The classified intent category
• category_scores: Dict[str, float] — Average cosine similarity per category
• chosen_temperature: float — The temperature that was returned
• override_active: bool — Whether an override was active

set_override(temperature)

router.set_override(temperature: Optional[float]) -> None

Sets or clears the temperature override. When set, the router bypasses classification and always returns the given temperature. Pass None to clear.

Advanced Customization

Override Temperature Map

from multi_temp_lm import MultiTempRouter

router = MultiTempRouter(
    temperature_map={
        "factual": 0.1,
        "creative": 1.0,
        "coding": 0.2,
    }
)

Add Custom Categories and References

from multi_temp_lm import MultiTempRouter, ReferenceExample

router = MultiTempRouter(
    references=[
        ReferenceExample("What is the capital of France?", "factual"),
        ReferenceExample("Write a haiku about rain.", "creative"),
        ReferenceExample("Refactor this Java class.", "coding"),
        ReferenceExample("My custom domain query.", "custom_category"),
    ],
    temperature_map={
        "custom_category": 0.5,
    },
)

Use a Custom Classifier

def my_classifier(query: str) -> str:
    if "def " in query or "class " in query:
        return "coding"
    return "default"

router = MultiTempRouter(classifier=my_classifier)

When a custom classifier is provided, the embedding-based classifier is completely bypassed.

Adjust Cache Size

router = MultiTempRouter(cache_size=256)

Set cache_size=0 to disable caching entirely.

Override Mode

Force a fixed temperature regardless of classification:

router = MultiTempRouter()
router.set_override(0.5)

# Always returns 0.5 until override is cleared
router.set_override(None)

Debug Mode

Enable debug mode to inspect classification decisions:

router = MultiTempRouter(debug=True)

temperature = router.for_messages([{"role": "user", "content": "Write a poem about the ocean."}])

debug = router.get_debug_info()
print(debug.detected_category)   # "creative"
print(debug.chosen_temperature)  # 0.9
print(debug.category_scores)     # {"factual": 0.12, "creative": 0.89, ...}

Performance

Classification latency depends on hardware and the embedding backend:

• ONNX runtime (default): typically 5–30 ms per query on a modern CPU
• PyTorch fallback: typically 20–80 ms per query on a modern CPU
• Cache hit: sub-millisecond for repeated queries

The first call downloads and caches the all-MiniLM-L6-v2 model (~22 MB with ONNX). Subsequent calls use the cached model.

Architecture Overview

User Code
    |
    | temperature=router
    v
Any LLM SDK (LiteLLM / OpenAI / Anthropic / Google / Cohere / Ollama / ...)
    |
    | router(messages)
    v
MultiTempRouter
    |
    +-- MessageExtractor → query_text
    |
    +-- QueryCache (LRU) → cache hit? return float
    |
    +-- EmbeddingClassifier → embedding + cosine similarity
    |       +-- ReferenceStore (pre-computed reference vectors)
    |
    +-- TemperatureMapper → category → float
    |
    v
float temperature

1. Message Extractor pulls the user query text from the message list.
2. Query Cache checks for an exact string match; if found, returns instantly.
3. Embedding Classifier computes a local embedding for the query and compares it against pre-computed reference embeddings via cosine similarity.
4. Temperature Mapper maps the best-matching category to a temperature value.
5. The result is cached and returned to the SDK.

All classification is performed locally. No external API calls are made for routing.

Contributing

Contributions are welcome! Please open an issue or pull request on GitHub.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/my-feature)
3. Install dev dependencies: pip install -e ".[dev]"
4. Run tests: pytest
5. Run linting: ruff check src tests and mypy src
6. Submit a pull request

License

This project is licensed under the MIT License. See LICENSE for details.