leastgen 0.1.0

LeastGen: High-performance local inference optimizer and context governance gateway for AI developer agents.

⚡ LeastGen

Zero-cost caching proxy for LLM agent traffic. Sits between your AI coding agent and the API — intercepts repetitive prompt patterns, serves them from a local cache. After warm-up, ~70%+ of requests resolve at $0 cost and ~0ms latency.

Agent ──► LeastGen (:8766) ──► OpenRouter / API
                │
        ┌───────┴────────┐
        │  Seen ≥3×?      │
        │  YES ── cache   │
        │  NO  ── forward │
        └────────────────┘

---

Quick Start

curl -sLS https://leastgen.com/install.sh | bash

This installs the package, creates a default config, and starts the proxy as a systemd user service on port 8766.

What you get:

• A transparent proxy on localhost:8766
• Dashboard at http://localhost:8766/dashboard
• Metrics at http://localhost:8766/metrics
• Persistent SQLite cache at ~/.leastgen/data/

Prerequisites

• Python 3.10+ — the proxy runs entirely in stdlib (no heavy ML deps)
• An API key — set OPENROUTER_API_KEY env var or put it in ~/.leastgen/config.yaml
• Ollama (optional) — for local pipeline inference when the cache misses. Install from ollama.com
• Hermes Agent (optional) — LeastGen integrates natively via model.base_url

---

Manual Installation

1. Install the package

git clone https://github.com/KhalidAlnujaidi/leastgen.git
cd leastgen
pip install -e .

2. Configure

Edit ~/.leastgen/config.yaml:

port: 8766
ttl_days: 7                          # Evict cache entries after 7 days of inactivity
learn_threshold: 3                   # Cache after N occurrences of the same pattern
upstream_url: "https://openrouter.ai/api/v1"
upstream_model: "deepseek/deepseek-chat"
local_planner_model: "vibethinker-3b"   # Ollama model for planning step
local_executor_model: "qwen3:4b"        # Ollama model for code generation step
api_key: "sk-or-..."                    # Optional — or set OPENROUTER_API_KEY

3. Set your API key

LeastGen looks for an API key in this order:

1. api_key field in ~/.leastgen/config.yaml
2. OPENROUTER_API_KEY environment variable
3. ~/.config/free-claude-code/.env

export OPENROUTER_API_KEY="sk-or-..."

4. Start the proxy

leastgen start

Or directly:

python3 -m leastgen.cli start --port 8766

5. Verify it's running

curl -s http://localhost:8766/metrics | python3 -m json.tool

Expected output:

{
  "total": 0,
  "hits": 0,
  "hit_rate_pct": 0.0,
  "templates": 0,
  "uptime_seconds": 2,
  "cost_saved": 0.0,
  "cost_passed": 0.0
}

---

Connecting Your Agent

Hermes Agent

hermes config set model.base_url http://localhost:8766/v1

All LLM requests from Hermes now route through LeastGen.

Any OpenAI-compatible client

Point your base_url to http://localhost:8766/v1. Everything OpenAI-compatible works — streaming, tool calls, reasoning tokens.

---

GPU & VRAM Guidance

LeastGen's caching layer needs zero GPU — it's a lightweight Python proxy using SQLite. You can run it on any machine, even a Raspberry Pi.

The local inference pipeline (optional — the VibeThinker-3B planner + Qwen3:4b executor) does need VRAM if you want fully offline fallback. Here's what you need for each configuration:

Configuration A: Caching Only (0 GB VRAM — any machine)

No GPU required. The proxy just intercepts, caches, and forwards to the API when it misses. Run this on a laptop, a $5 VPS, or alongside your agent.

Agent ──► LeastGen ──► cache hit (instant, $0)
                └────► cache miss ──► API ($)

Set in ~/.leastgen/config.yaml:

# No local models — pure caching proxy
# Every miss goes to the upstream API

Configuration B: Lightweight Local Fallback (~4 GB VRAM)

Run smaller Ollama models for offline inference when cache misses. Works on most consumer GPUs (GTX 1060 6GB, RTX 2060, RTX 3050, RTX 3060, etc.).

# Install lightweight models via Ollama
ollama pull qwen2.5-coder:1.5b     # ~1.1 GB
ollama pull qwen2.5-coder:0.5b     # ~0.5 GB (ultra-light, for planning)

# ~/.leastgen/config.yaml
local_planner_model: "qwen2.5-coder:0.5b"
local_executor_model: "qwen2.5-coder:1.5b"

VRAM estimate: ~2–3 GB — fits on any GPU with 4 GB or more.

Configuration C: Standard Pipeline (~6 GB VRAM)

The default configuration with VibeThinker-3B + Qwen3:4b. Requires ~6 GB VRAM. Runs on RTX 2060 Super (8 GB), RTX 3060 (12 GB), RTX 4060, M-series Macs with 16 GB unified memory, etc.

ollama pull vibethinker-3b     # ~2 GB
ollama pull qwen3:4b           # ~4 GB

Configuration D: High-Performance Pipeline (~10–12 GB VRAM)

Larger models for better code generation quality. Needs RTX 3090/4090, A4000+, or similar.

ollama pull deepseek-r1:8b        # R1 reasoning for planning
ollama pull qwen2.5-coder:7b      # Strong code executor

local_planner_model: "deepseek-r1:8b"
local_executor_model: "qwen2.5-coder:7b"

Quick VRAM Reference

Config	Models	VRAM	GPU Examples
A — Cache only	None	0 GB	Any machine
B — Lightweight	Qwen 0.5B + 1.5B	~2–3 GB	GTX 1060, 3050, 2060
C — Standard	VibeThinker-3B + Qwen3:4b	~6 GB	RTX 2060S, 3060, 4060
D — High perf	R1:8b + Qwen2.5-Coder:7b	~10–12 GB	RTX 3090, 4090, A4000

Tokens/Second Benchmarks (RTX A4500)

Model	Tokens/sec	vs. Cloud
VibeThinker-3B (Q4_K_M)	201 tok/s	~2-4× faster
Qwen3:4b	161 tok/s	~2-3× faster
Claude 5 Fable (cloud API)	~50–100 tok/s	+200–500ms network latency

Fair-Comparison Note

Local proxy models may generate ~3× more tokens than a frontier model to complete the same task — the reduced capability means more iterations to converge on a correct answer. However, these extra tokens are generated entirely free on local GPU at the speeds above. The "tokens saved" figures on the site are not a 1:1 apples-to-apples comparison with cloud tokens — they represent tokens that would have cost money if sent to the API. The local pipeline may do more total generation, but every token is instant and costs $0.00.

CPU-Only Mode

Ollama models run on CPU by default if no GPU is detected. Performance is slower (seconds instead of milliseconds) but works on any machine. For CPU-only:

# Ollama automatically falls back to CPU if no NVIDIA GPU
ollama pull qwen2.5-coder:1.5b     # ~1.1 GB RAM on CPU

Note: on CPU, responses take 5–30 seconds instead of <1 second on GPU. The caching layer is still instant — only the fallback path is slower.

---

Dashboard & Monitoring

Live Dashboard

Open http://localhost:8766/dashboard in your browser.

Shows in real time:

• Cache Hit Rate — % of requests served locally
• Tokens Saved / Passed — cumulative counts
• Cost Saved / Cost Incurred — estimated $
• Templates — distinct patterns learned
• Hit rate chart — live time-series
• Recent Activity — last 20 cache/learn/forward events

Metrics Endpoint

curl -s http://localhost:8766/metrics | python3 -m json.tool

Returns: total, hits, template_hits, hit_rate_pct, templates, cached_responses, learned_patterns, uptime_seconds, prompt_tokens_saved, completion_tokens_saved, tokens_saved, tokens_passed, cost_saved, cost_passed, recent

Run a Savings Audit

leastgen audit

Scans your Hermes session history and estimates how much LeastGen would save on your actual traffic patterns.

---

Systemd Service (Auto-Start)

The install script sets this up automatically. To do it manually:

# Create the service file
mkdir -p ~/.config/systemd/user
cat > ~/.config/systemd/user/leastgen.service << 'EOF'
[Unit]
Description=LeastGen Local Inference Optimizer Caching Gateway
After=network.target

[Service]
Type=simple
ExecStart=%h/.local/bin/leastgen start
Restart=always
RestartSec=5

[Install]
WantedBy=default.target
EOF

# Enable and start
systemctl --user daemon-reload
systemctl --user enable leastgen.service
systemctl --user start leastgen.service

# Check status
systemctl --user status leastgen

---

Enterprise Value Projection

Based on Anthropic Claude 5 Fable pricing ($10/1M input, $50/1M output) at 20M prompt tokens per developer per day with a ~70:30 prompt-to-completion ratio:

Team Size	Annual Tokens Saved	Annual API Cost Avoided
1 Developer	7.1 Billion	$155,000
10 Developers	71.5 Billion	$1,575,000
50 Developers	357.5 Billion	$7,875,000
100 Developers	715 Billion	$15,750,000

(See leastgen.com for the live projection calculator and benchmarks.)

---

Architecture

┌──────────────────────────────────────────────────────┐
│                    Your Agent                         │
│  base_url = http://localhost:8766/v1                  │
└────────────────────────┬─────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────┐
│              LeastGen (port 8766)                      │
│                                                        │
│  Incoming request ──► template_normalize()             │
│                         │                              │
│                    ┌────┴────┐                         │
│                    │         │                         │
│               exact hit  template hit                  │
│               (hash)    (pattern)                      │
│                    │         │                         │
│               serve from   serve from                  │
│               cache        cache                       │
│                    │         │                         │
│                    └────┬────┘                         │
│                         │                              │
│               match?    │                              │
│                    ┌────┴────┐                         │
│                    │  seen   │                         │
│                    │  ≥ 3×?  │                         │
│                    │  YES ──► learn & cache            │
│                    │  NO  ──► forward to API           │
│                    └─────────┘                         │
└────────────────────────────────────────────────────────┘
                         │
                         ▼
┌──────────────────────────────────────────────────────┐
│              Upstream API (OpenRouter, etc.)           │
│  https://openrouter.ai/api/v1/chat/completions         │
└──────────────────────────────────────────────────────┘

---

Troubleshooting

Proxy won't start

# Check if port is in use
ss -tlnp | grep 8766
# Kill any hanging process
pkill -9 -f leastgen

"No API key" error

export OPENROUTER_API_KEY="sk-or-..."
# Or add to config:
#   api_key: "sk-or-..."  # in ~/.leastgen/config.yaml

Dashboard shows 0% hit rate

Normal on a fresh start — the gate needs ≥3 occurrences of a pattern before caching. Work normally and the hit rate climbs as patterns repeat.

High cache miss rate

Template normalization only uses the last user message's first ~10 significant words. Long, unique prompts may not match previously cached templates. The exact hash cache still catches identical repeats.

Reset all state (start fresh)

systemctl --user stop leastgen
rm -rf ~/.leastgen/data/cache.db
systemctl --user start leastgen

---

Links

• Website: https://www.leastgen.com
• GitHub: https://github.com/KhalidAlnujaidi/leastgen
• Author: Khalid Alnujaidi — [e-mail]