Hermes-Lite ⚡

Cloud-first AI agent via NVIDIA NIM Free API — with local fallback for offline use.

Hermes-Lite is an agent framework that defaults to cloud LLMs (NVIDIA NIM Free API) for quality, with a local mode for offline/privacy use. Rate limiting, API key rotation, and a smart fallback chain keep it resilient — all for free.

---

Why Hermes-Lite?

• ☁️ Cloud-first, free tier: NVIDIA NIM Free API gives 40 RPM of production-grade LLMs (z-ai/glm-5.2, Kimi K2.6, Qwen 3.5, DeepSeek V4) at zero cost
• 🔄 Resilient by default: Token-bucket rate limiter, exponential backoff on 429s, API key rotation from a pool
• 🏠 Local fallback: Switch to a local 7B Qwen model for offline/privacy — same codebase, different prefix
• 🛡️ Smart routing: Complexity-based tier selection. Simple queries → fast model. Complex reasoning → heavy model. Escalation on repeated failures.
• 🧩 6 built-in tools: read_file, search_files, terminal, memory, web_search, web_fetch
• 🔌 Sub-agent delegation: Spawn isolated subagents for parallel work

---

Quick Start

Cloud Mode (default, no local model needed)

# 1. Set your NVIDIA NIM API key (free at https://build.nvidia.com/)
export HERMES_LITE_NVIDIA_API_KEY="nvapi-..."

# Optional: add multiple keys for rotation
# export HERMES_LITE_NVIDIA_API_KEYS="key1,key2,key3"

# 2. Install hermes-lite
git clone https://github.com/ahmedhabibo/hermes-lite.git
cd hermes-lite
pip install -e ".[test]"

# 3. Run the CLI — starts with cloud models
python -m hermes_lite

Local Mode (offline / privacy)

# 1. Install llama.cpp
brew install llama.cpp

# 2. Download the model (Qwen 2.5 Coder 7B Instruct, IQ3_XS — 3.1 GB)
hf download bartowski/Qwen2.5-Coder-7B-Instruct-IQ3_XS-GGUF \
    Qwen2.5-Coder-7B-Instruct-IQ3_XS.gguf \
    --local-dir ~/.hermes_lite/models/

# 3. Start the server
llama-server \
    -m ~/.hermes_lite/models/Qwen2.5-Coder-7B-Instruct-IQ3_XS.gguf \
    --port 8080 --temp 0.3 --repeat-penalty 1.1 \
    -ngl 28 -c 65536 --batch-size 512 --cache-type-k q8_0 --cache-type-v q8_0

# 4. Set a local-first fallback chain (or use local: prefix in chat)
export LITE_FALLBACK_CHAIN="local:Qwen2.5-Coder-7B-Instruct-IQ3_XS.gguf,z-ai/glm-5.2,minimaxai/minimax-m3,moonshotai/kimi-k2.6,qwen/qwen3.5-397b-a17b,deepseek-ai/deepseek-v4-flash"

pip install -e ".[test]"
python -m hermes_lite

---

How It Works

┌─────────────────────────────────────────────────┐
│              Router (cloud-first)                │
│  LiteRouter: complexity → pick NIM model in chain│
│  Escalation: failure → next model in chain       │
└──────────────┬──────────────────────────────────┘
               │
┌──────────────▼──────────────────────────────────┐
│                  LLM Layer                       │
│  NVIDIA NIM (cloud)   │  Qwen 7B (local)       │
│  40 RPM rate limit    │  Text-parsed tool calls │
│  Key rotation + retry │  No tools/grammar sent  │
└──────────────┬──────────────────────────────────┘
               │
┌──────────────▼──────────────────────────────────┐
│               Tool Loop                          │
│  2-tier: LLM → tool → result → LLM → response   │
│  Max 4 iterations, repeated-error, malformed-JSON│
└──────────────┬──────────────────────────────────┘
               │
┌──────────────▼──────────────────────────────────┐
│           Tool Layer (PluginRegistry)            │
│ read_file │ search_files │ terminal │ memory │  │
│ web_search │ web_fetch │ subagent               │
└──────────────┬──────────────────────────────────┘
               │
┌──────────────▼──────────────────────────────────┐
│              Sandbox / Backend                   │
│  sandbox-exec   │  Hermes MCP   │  curl          │
└──────────────────────────────────────────────────┘

---

Rate Limiting & Key Rotation

Hermes-Lite v0.4+ handles NVIDIA NIM Free API limits automatically:

|| Feature | Detail | ||---------|--------| || Rate limiter | Token-bucket, 40 RPM (configurable via HERMES_LITE_RPM) | || Exponential backoff | 429 errors: 1s → 2s → 4s → 8s (max 16s) | || Key rotation | Comma-separated pool in HERMES_LITE_NVIDIA_API_KEYS. On 401/403/429, rotates to next key | || Key cooldown | Failed keys cool down for 60s before reuse | || Max retries | 4 attempts (configurable via HERMES_LITE_MAX_RETRIES) |

---

NIM Fallback Chain

The default chain (cloud-first):

z-ai/glm-5.2           ← preferred (general-purpose)
  → minimaxai/minimax-m3 ← strong reasoning
    → moonshotai/kimi-k2.6 ← MoE, efficient
      → qwen/qwen3.5-397b-a17b ← fast fallback
        → deepseek-ai/deepseek-v4-flash ← ultra-fast

Override with LITE_FALLBACK_CHAIN env var (comma-separated).

For local-first: local:Qwen2.5-Coder-7B-Instruct-IQ3_XS.gguf,z-ai/glm-5.2,minimaxai/minimax-m3

---

Features

|| Area | What it does | ||------|-------------| || Tool registry | 6 built-in essentials: read_file, search_files, terminal, memory, web_search, web_fetch. Pydantic-validated dispatch. Extensible via ToolDefinition. | || LLM layer | OpenAI-compatible chat API. Default: cloud NIM (z-ai/glm-5.2). Supports local fallback (Qwen 2.5 Coder 7B via llama.cpp). Rate limiting + key rotation + exponential backoff. | || Tool loop | Two-tier loop: LLM calls tools → results fed back → LLM responds. Max 4 iterations, repeated-error and malformed-JSON guards. | || Router | LiteRouter classifies prompts by complexity. Cloud-first chain: light queries → fast model. Complex reasoning → heavier model. Consecutive-failure escalation walks the chain. | || Sandbox | terminal tool runs commands in a macOS sandbox (sandbox-exec). Timeout-safe with process lifecycle management. Sandbox security: command allowlist/blocklist, secret env scrubbing, audit log redaction | || Memory | SQLite bridge for cross-session facts. add, replace, remove, list — unique-match semantics. Loaded into every prompt (800 char cap). | || Sub-agent | Spawn isolated subagents for parallel work (default: cloud NIM flash model). Nested orchestration (max 2 levels). Subagent isolation: env sanitization | || MoA | Mixture-of-Agents: run 3–5 diverse LLMs in parallel, then aggregate into a single superior answer. 5 built-in presets (council, speed, verification, coding, creative). CLI: /moa <preset>. | || Observability | Per-turn JSONL logging, rotation at 10 MB, python -m hermes_lite.stats for session summary. | || CLI | prompt_toolkit + Rich terminal. Ctrl+C/D, !tool {args} direct invocation, /tools, /history, /help. |

---

Configuration

|| Variable | Default | Purpose | ||----------|---------|---------| || HERMES_LITE_CLOUD_URL | https://integrate.api.nvidia.com/v1 | Cloud endpoint | || HERMES_LITE_CLOUD_MODEL | z-ai/glm-5.2 | Default cloud model | || HERMES_LITE_NVIDIA_API_KEY | — | Single NVIDIA NIM API key | || HERMES_LITE_NVIDIA_API_KEYS | — | Comma-separated key pool for rotation | || HERMES_LITE_RPM | 40 | Requests per minute (token bucket) | || HERMES_LITE_MAX_RETRIES | 4 | Max retry attempts on 429/server errors | || HERMES_LITE_LOCAL_URL | http://127.0.0.1:8080/v1 | Local llama.cpp endpoint | || HERMES_LITE_LOCAL_MODEL | Qwen2.5-Coder-7B-Instruct-IQ3_XS.gguf | Local model file | || HERMES_LITE_LOCAL_TOOLS | unset | Set 1 to send tools/tool_choice to local endpoint | || LITE_LOCAL_MAX_COMPLEXITY | 0.3 | Max complexity score for using lighter model | || HERMES_LITE_MOA_PRESET | — | Auto-activate MoA preset on startup (e.g. council) | || HERMES_LITE_MOA_TIMEOUT | 30 | Seconds before a reference model is skipped | || LITE_FALLBACK_CHAIN | z-ai/glm-5.2,minimaxai/minimax-m3,moonshotai/kimi-k2.6,qwen/qwen3.5-397b-a17b,deepseek-ai/deepseek-v4-flash | Model fallback chain | || HERMES_LITE_SUBAGENT_MODEL | deepseek-ai/deepseek-v4-flash | Subagent default model |

---

Mixture-of-Agents (MoA)

Run multiple diverse LLMs on the same prompt in parallel, then let an aggregator model synthesize their outputs into a single, superior answer.

Quick Start

❯ /moa council      # Activate the "council" preset (3 diverse refs + 1 aggregator)
❯ Tell me about Python GIL
[MoA] 🔀 3 refs → aggregator → final

Built-in Presets

|| Preset | References | Aggregator | Use case | ||--------|-----------|------------|----------| || council | z-ai/glm-5.2, minimax-m3, kimi-k2.6 | deepseek-v4-pro | General reasoning — maximum diversity | || speed | z-ai/glm-5.2, deepseek-v4-flash | deepseek-v4-flash | Fast answers — lighter models | || verification | kimi-k2.6, qwen3.5-397b, deepseek-v4-pro | z-ai/glm-5.2 | Fact-checking — cross-verify claims | || coding | deepseek-v4-pro, qwen3.5-397b | deepseek-v4-pro | Code generation — precision-focused | || creative | z-ai/glm-5.2, kimi-k2.6, qwen3.5-122b | z-ai/glm-5.2 | Creative writing — divergent styles |

Commands

|| Command | Action | ||---------|--------| || /moa | Show status + available presets | || /moa council | Activate a preset | || /moa off | Deactivate (back to normal ToolLoop) |

Architecture

User prompt
    │
    ├─→ Reference A (model-1) ──┐
    ├─→ Reference B (model-2) ──┼─→ Aggregator ─→ Final answer
    └─→ Reference C (model-3) ──┘

• Parallel mode (default): asyncio.gather runs all references concurrently, then feeds collected outputs to the aggregator
• Sequential mode: Each reference sees the previous reference's output as context — useful for iterative refinement
• Graceful degradation: If some references fail/timeout, the aggregator works with whatever succeeded. If all fail, falls back to a direct single-model call.

Environment Variable Override

HERMES_LITE_MOA_PRESET=council  # Auto-activate on startup

---

Test Suite

cd hermes-lite
pip install -e ".[test]"
python -m pytest tests/ -v

Tests cover: registry (48), memory (47), orchestrator (31), tool loop (15), tools-essentials (55), LLM (5), router (37), sandbox (60), sub-agent (19), memory bridge (10), observability (6), e2e smoke (5), moa 15, api_key_exhaustion 5, sanitize ~20, cli_commands ~10, streaming 4, conftest 1. 467 total.

---

Contributing

See CONTRIBUTING.md — PRs welcome, TDD preferred.

---

License

MIT — use it, fork it, ship it.

---

CHANGELOG

0.6.0 — Security Hardening + Streaming + Docker

2026-07-03

• Security (Phase 1, 7 items): API key exhaustion handling, Auth/Authorization framework, Input sanitization pipeline, Rate-limit hardening with jitter, Sandbox tightening (command allowlist/blocklist, secret scrubbing, audit log redaction), Secret redaction in logs, Subagent isolation (env sanitization)
• Streaming: chat_stream() async generator — token-by-token streaming for both cloud and local endpoints
• Docker: Dockerfile + docker-compose.yml for containerized deployment
• Changed: Default model to z-ai/glm-5.2, local model to Qwen2.5-Coder-7B-Instruct-IQ3_XS (3.1GB, Bartowski quant)
• CI: GitHub Actions — Python 3.9/3.11/3.12 matrix, pytest + coverage + ruff lint
• Tests: 432 → 467

0.5.0 — Toolchain Expansion + Observability

2026-06-15

• Added: web_fetch tool, memory bridge observability, sandbox-exec timeout propagation, subagent nesting limit (2 levels)
• Changed: Tool loop max iterations increased to 4, MoA preset verification added, local model quantized to IQ3_XS
• Tests: 342 → 432

0.4.0 — Cloud-first NIM pivot + rate limiting

• Cloud-first default: NIM Free API as primary LLM (MiniMax M3, Kimi K2.6, Qwen 3.5, DeepSeek V4 Flash)
• Rate limiting: Token-bucket at 40 RPM (NIM Free API limit)
• API key rotation: Comma-separated pool (HERMES_LITE_NVIDIA_API_KEYS), auto-rotate on 401/403/429
• Exponential backoff: 1s → 2s → 4s → 8s (cap 16s) on 429/server errors
• Router v2: Cloud-first fallback chain, consecutive-cloud-failure escalation walks the chain
• Subagent cloud default: deepseek-ai/deepseek-v4-flash (set HERMES_LITE_SUBAGENT_MODEL for local)
• Local still supported: local: prefix or local-first fallback chain for offline/privacy use
• Added stepfun-ai/ to cloud prefix detection
• Bare model IDs containing / now route to cloud automatically

0.3.0 — Qwen 2.5 7B Instruct + text-based tool-call parser

• Upgraded local model: Qwen 2.5 3B → Qwen 2.5 7B Instruct Q4_K_M (4.4 GB)
• Text-based tool-call parser: 4 regex patterns (Qwen blank-line JSON, fenced tool_call, fenced JSON, bare JSON)
• Skip tools/tool_choice for local endpoint — avoids PEG grammar 500 errors on small models
• Cloud fallback via NVIDIA NIM (configurable model chain)
• MIT license + CONTRIBUTING guide added

0.2.0 — Local Qwen 3B + 6 essential tools + router + sandbox

• Tool registry with 6 essentials (read_file, search_files, terminal, memory, web_search, web_fetch)
• LiteRouter: prompt complexity classifier for local/cloud tier routing
• ToolLoop: two-tier tool-calling loop with termination guards
• Sandboxed terminal execution (macOS sandbox-exec)
• Memory Bridge: cross-session persistent facts (SQLite)
• Subagent: parallel tool-spawning with isolated context
• Observability: per-turn JSONL logging + stats CLI