kolmo 0.0.3

Ultra-low-latency LLM router using a compression-distance heuristic inspired by Kolmogorov complexity.

⚡ Kolmo v0.0.3

The Information-Theoretic LLM Router Zero VRAM. Microsecond latency. Dictionary-based compression distance heuristic.

---

🧬 What is Kolmo?

Kolmo is a high-performance LLM router that leverages a dictionary-based compression distance heuristic inspired by Normalized Compression Distance (NCD) to classify and route prompts to the most appropriate model. No neural networks. No GPU memory. Pure computational elegance.

from kolmo import KolmoRouter

# Initialize with your fallback model
router = KolmoRouter(default_model="gpt-4o-mini")

# Train domains from HuggingFace datasets or local samples
router.add_domain(
    name="code",
    source="huggingface:codeparrot/github-code",
    target_model="deepseek-coder"
)
router.add_domain(
    name="science",
    samples=[b"Photosynthesis converts light energy into...", b"Quantum mechanics states that..."],
    target_model="claude-3-opus"
)

# Route in microseconds ⚡
result = router.route("Implement a Red-Black tree in Python")
print(result.winner)           # "code"
print(result.target_model)     # "deepseek-coder"
print(f"{result.latency_ms:.3f}ms")  # ~0.042ms

---

🚀 Why Kolmo?

Feature	Traditional Routers	Kolmo
Latency	Milliseconds	Microseconds
VRAM Usage	2-20GB	Zero 🎯
Explainability	Black-box embeddings	Compression ratios
Language Support	Requires training data	Universal (byte-level)
Deploy Cost	GPU required	CPU-only
Cold Start	Model loading time	Instant

---

📦 Installation

pip install kolmo

Requires Python 3.10+.

---

🎮 Quick Start

Python API

from kolmo import KolmoRouter

router = KolmoRouter()

# Add domains from various sources
router.add_domain(
    name="medical",
    source="huggingface:medalpaca/medical_meadow_medical_flashcards",
    target_model="meditron-7b",
    sample_count=5000
)

# Or train from raw samples
router.add_domain(
    name="legal",
    samples=[b"Contract clause 1...", b"In accordance with Section 2..."],
    target_model="lawgpt"
)

# Route with confidence
result = router.route("Patient presents with tachycardia and chest pain...")
print(f"Domain: {result.winner}")              # "medical"
print(f"Model: {result.target_model}")         # "meditron-7b"
print(f"Confidence: {result.confidence:.2%}")  # 0.94
print(f"Latency: {result.latency_ms:.3f}ms")   # 0.038ms

# Save as portable bundle
router.save_bundle("medical_router.kolmo")

CLI Interface

Kolmo ships with a powerful CLI for training and routing without writing code:

# Create config.json
{
  "default_model": "gpt-4o-mini",
  "dict_size": 112640,
  "domains": [
    {
      "name": "code",
      "source": "huggingface:codeparrot/github-code",
      "target_model": "deepseek-coder",
      "sample_count": 1000
    },
    {
      "name": "general",
      "samples": ["Hello world", "How are you?"],
      "target_model": "gpt-4o-mini"
    }
  ]
}

# Train your bundle
kolmo train config.json router.kolmo

# Route a prompt
kolmo route router.kolmo "Write a Python function to reverse a string"

# Get bundle info
kolmo info router.kolmo

---

🧠 The Science: Compression Distance Heuristic

Kolmo is inspired by Kolmogorov Complexity theory and uses a dictionary-based compression distance heuristic (not strict NCD). For context, the strict NCD definition is below.

The Foundation

Kolmogorov Complexity K(x) is the theoretical minimum information needed to represent a string x. While K(x) is uncomputable, compression provides an excellent upper bound.

Strict NCD Formula

$$ \mathrm{NCD}(x, y) = \frac{C(xy) - \min(C(x), C(y))}{\max(C(x), C(y))} $$

Where:

• C(x) = compressed size of x
• C(xy) = compressed size of x concatenated with y
• When x and y share structure → C(xy) < C(x) + C(y)

How Kolmo Uses a Compression Heuristic

Kolmo approximates the NCD idea with zstd dictionaries by treating each dictionary as a compact "reference string":

score(prompt, dict) = (C(dict) + C(prompt | dict) - min(C(prompt), C(dict))) / max(C(prompt), C(dict))

This is a dictionary-based compression distance heuristic, not strict NCD, because it uses dictionary-conditional compression to approximate (C(xy)).

The domain with the LOWEST score wins — the prompt compresses best with that domain's dictionary, indicating highest similarity.

---

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Kolmo v0.0.3 • Router                   │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│   Input Prompt                                               │
│        │                                                     │
│        ▼                                                     │
│   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐  │
│   │ zstd + Dict  │    │ zstd + Dict  │    │ zstd + Dict  │  │
│   │   (Code)     │    │  (Medical)   │    │   (Legal)    │  │
│   └──────┬───────┘    └──────┬───────┘    └──────┬───────┘  │
│          │                    │                    │          │
│          ▼                    ▼                    ▼          │
│      NCD: 0.12           NCD: 0.34           NCD: 0.89       │
│          │                    │                    │          │
│          └────────────────────┼────────────────────┘          │
│                               │                              │
│                    Winner: Code (lowest NCD)                 │
│                               │                              │
│                    Route → deepseek-coder                    │
└─────────────────────────────────────────────────────────────┘

---

🔥 v0.0.3 • Features

⚡ Batch Routing

Process thousands of prompts efficiently:

prompts = [
    "Write a Python function",
    "Explain quantum entanglement",
    "Draft a legal contract clause"
]

batch = router.route_many(prompts)
print(f"Throughput: {batch.throughput_rps:.0f} req/s")
print(f"Avg latency: {batch.avg_latency_ms:.3f}ms")

🎯 Thresholded Routing with Ambiguity Detection

Handle uncertain decisions gracefully:

result = router.route_thresholded(
    prompt,
    min_confidence=0.3,      # Minimum confidence to accept
    ambiguity_gap=0.05,      # Gap between top candidates
    max_candidates=3         # Return top-N candidates
)

if result.ambiguous:
    print(f"Uncertain ({result.reason}): {result.candidates}")
    # Fallback to general-purpose model
else:
    model = result.target_model

📊 Pluggable Metrics Hooks

Integrate with your monitoring stack:

def prometheus_hook(metrics: RoutingMetrics):
    latency.observe(metrics.latency_ms)
    requests.labels(domain=metrics.winner).inc()

router = KolmoRouter(metrics_hooks=[prometheus_hook])

🛠️ Fine-Grained Controls

Tune dictionary size per domain:

router.add_domain(
    name="code",
    samples=code_samples,
    dict_size=50000,           # Custom dictionary size
)

---

📈 Performance

Run the reproducible benchmark script to generate the table for your hardware:

python examples/quickstart_benchmark.py --iterations 500 --domains 5

It logs CPU details, domain count, iteration count, and prompt lengths, then prints a README-ready table.

Prompt Length	Domains	Avg Latency (ms)	p50 (ms)	p95 (ms)	Throughput (req/s)	Accuracy
(run script)	5	(run script)	(run)	(run)	(run script)	(run)

Memory footprint: ~50KB per domain dictionary

---

🌍 Language Agnostic

Kolmo operates at the byte level using statistical compression:

• ✅ English, Spanish, Mandarin, Japanese...
• ✅ Python, Rust, Go, C++, JavaScript...
• ✅ Log files, JSON, XML, Protocol Buffers...
• ✅ Mixed-domain prompts

No tokenization. No language-specific training.

---

📚 Advanced Examples

Coding vs. Reasoning Router

router = KolmoRouter()

router.add_domain(
    "coding",
    source="huggingface:codeparrot/github-code",
    target_model="deepseek-coder"
)
router.add_domain(
    "reasoning",
    source="huggingface:sci_q",
    target_model="deepseek-r1"
)

# Coding prompt → deepseek-coder
res = router.route("Write a FastAPI endpoint for user login")

# Reasoning prompt → deepseek-r1
res = router.route("If 3x + 5 = 20, what is x?")

Language Detection

router.add_domain("en", samples=english_samples, target_model="gpt-4o")
router.add_domain("es", samples=spanish_samples, target_model="gpt-4o-es")

res = router.route("Hola, ¿cómo estás?")
print(res.winner)  # "es"

Using Pre-trained Dictionaries

# Load a pre-trained zstd dictionary
dict_bytes = open("medical.dict", "rb").read()

router.add_domain(
    name="medical",
    dict_bytes=dict_bytes,
    target_model="meditron-7b"
)

---

🧪 API Reference

KolmoRouter

Method	Description
add_domain(name, ...)	Add a routing domain
route(prompt)	Route a single prompt
route_thresholded(prompt, ...)	Route with ambiguity detection
route_many(prompts)	Batch routing with aggregate stats
save_bundle(path)	Save router to .kolmo file
load_bundle(path)	Load router from .kolmo file

CLI Commands

kolmo train <config.json> <output.kolmo>  # Train a bundle
kolmo route <bundle.kolmo> <prompt>        # Route a prompt
kolmo info <bundle.kolmo>                  # Show bundle info

---

🗺️ Roadmap

✅ Implemented (Current)

• Batch routing with throughput stats
• Thresholded routing with ambiguity detection
• CLI tooling (train, route, info)
• Pluggable metrics hooks
• Fine-grained dictionary sizing controls

🚀 Next Steps (High Priority)

• Dictionary optimization and pruning tools
• Config-driven domain registry (YAML/JSON)
• Comprehensive benchmark suite
• Caching layer for repeated prompts
• Improved dataset ingestion with progress tracking

🗺️ Future Horizons (Backlog)

• Hierarchical multi-stage routing
• Adaptive dictionary refresh
• REST/gRPC service layer
• SIMD-optimized compression kernels

---

🤝 Contributing

We welcome contributions! Kolmo is in active development.

git clone https://github.com/charlybgai/kolmo
cd kolmo
uv sync --frozen
uv tool install pre-commit
pre-commit install

---

📄 License

MIT License — See LICENSE for details.

---

🙏 Acknowledgments

Kolmo is inspired by research in:

• Normalized Compression Distance (Li et al.)
• The Similarity Metric (Li et al., 2004)
• Modern zstd dictionary training techniques (Facebook)

---

Built by Carlos Bustamante & KODEX Information theory meets production inference. ⚡