LLM-Rivotril
A lightweight Python framework to reduce LLM hallucinations, enforce guardrails, manage stateful memory, and monitor performance through a local web dashboard.
Why LLM-Rivotril?
LLM-Rivotril is an orchestration and safety layer around an LLM; it is not a model and does not make a small model inherently smarter. It lets the same application move between local and hosted models while keeping a consistent interface for guardrails, memory, caching, retries, streaming, tool calling, telemetry, and framework integrations.
Use Ollama when privacy, offline operation, and predictable local costs matter; use a hosted provider when you need a larger or more capable model. In both cases, LLM-Rivotril keeps the application-level controls in one place.
Features
- Guardrails — Block disallowed keywords, enforce allowed topics, limit output size, and validate JSON schemas on outputs.
- Semantic Guardrails (optional) — Match prompts against allowed topics using dense embeddings instead of exact keywords.
- Moderation Guardrail — Flag adversarial/unsafe content via OpenAI's moderation endpoint.
- Stateful Memory — Sliding-window conversation store to prevent context drift, with optional automatic disk persistence.
- Anti-Hallucination Verifier — Pluggable grounding checks, including keyword overlap, citation markers, and embedding-based faithfulness.
- Resilience — Built-in rate limiting, retry with backoff, and circuit breaker for LLM calls.
- Token-Saving Controls (optional) — Semantic (similarity-based) response cache, token-budget-aware memory trimming, and LLM-summarized history compaction.
- Telemetry & Dashboard — Built-in FastAPI dashboard with live request logs, token usage, latency, and success rate; optional per-token auth.
- Benchmark / Red-Team Evaluator — Labeled suite to measure guardrail and verifier accuracy without API costs.
- CLI — Launch the dashboard with a single command.
- Type-Safe Responses — Optional Pydantic response models via
instructor, including streamed structured output. - Async API —
run_async()for non-blocking execution. - Multi-Provider — OpenAI-compatible servers (Ollama, vLLM, ...), plus native Anthropic, Cohere, Gemini, Azure OpenAI, and AWS Bedrock adapters.
- Vector-Store Retrievers (optional) — pgvector, Qdrant, Weaviate, and Pinecone adapters for RAG beyond in-memory scale.
- Framework Integrations (optional) — Drop-in adapters for CrewAI, AG2/AutoGen, LangChain/LangGraph, and Google ADK, so guardrails/PII/RAG/observability apply inside those frameworks too.
Installation
pip install llmrivotril
For semantic (embedding-based) guardrails and verifiers:
pip install llmrivotril[semantic]
For local development:
git clone https://github.com/ailake-io/llmrivotril.git
cd llmrivotril
pip install -e ".[dev,semantic]"
This installs test, lint, type-check, and packaging tools (pytest, ruff, mypy, build, twine) plus every optional runtime dependency (all providers, RAG loaders, vector stores, Redis, framework integrations).
Quick Start
import os
from llmrivotril import RivotrilAgent, Guardrail
from llmrivotril.verifier import KeywordOverlapVerifier
agent = RivotrilAgent(
model="gpt-4o-mini",
api_key=os.getenv("OPENAI_API_KEY"),
guardrails=[
Guardrail(
name="safe-content",
allowed_topics=["AI safety", "machine learning"],
disallowed_keywords=["password", "secret"],
max_tokens=500,
)
],
verifier=KeywordOverlapVerifier(threshold=0.1),
)
response = agent.run("Explain what a guardrail is in AI safety.")
print(response)
Local Ollama
With Ollama running locally, use its OpenAI-compatible endpoint. The API key is only a placeholder required by the OpenAI client and is not sent to a remote provider:
ollama serve
ollama pull qwen2.5:3b
from llmrivotril import RivotrilAgent
agent = RivotrilAgent(
model="qwen2.5:3b",
base_url="http://127.0.0.1:11434/v1",
api_key="ollama",
)
print(agent.run("Explain what a guardrail is in AI safety."))
The model quality, latency, and hardware requirements depend on the Ollama model selected. Keep Ollama bound to localhost when prompts must stay on the machine.
Documentation
- Providers — OpenAI-compatible servers, Anthropic, Cohere, Gemini, Azure OpenAI, AWS Bedrock.
- Guardrails & Safety — Semantic guardrails, moderation, PII redaction, token budget.
- RAG — Local pipeline, vector-store retrievers (pgvector/Qdrant/Weaviate/Pinecone), grounding verification.
- Framework Integrations — CrewAI, AG2/AutoGen, LangChain/LangGraph, Google ADK adapters, and multi-agent setup notes.
- Async, Streaming & Function Calling
- Reliability — Resilience, response caching (including semantic cache), memory token budget & summarization, schema-repair fallback.
- Observability — Metrics persistence, local dashboard, benchmark, cost tracking.
- Configuration — Config files, environment variables, plugins, project scaffolding.
- Releasing — Build validation, TestPyPI, and the PyPI release workflow.
Interactive Demo
Run a complete walkthrough with mock LLM responses (no API key, no cost):
python examples/demo.py --mock --dashboard
Then open http://127.0.0.1:8767 to watch the dashboard update live.
Comparison: With vs. Without llmrivotril
See the same scenarios side-by-side:
python examples/comparison.py --mock
The comparison highlights how plain LLM calls return harmful or hallucinated
answers that are only caught manually afterwards, while llmrivotril blocks
them at runtime and records structured telemetry.
Project Structure
llmrivotril/
├── src/llmrivotril/
│ ├── agent.py # RivotrilAgent orchestrator (sync + async)
│ ├── config.py # Environment-variable and file configuration loader
│ ├── guardrails.py # Input/output guardrails
│ ├── moderation.py # OpenAI moderation-endpoint guardrail
│ ├── memory.py # Conversation memory store
│ ├── verifier.py # Hallucination / grounding checks
│ ├── providers.py # OpenAI / Azure / Anthropic / Cohere / Gemini / Bedrock adapters
│ ├── rag/ # RAG loaders, chunkers, retrievers, vector stores, and pipeline
│ ├── integrations/ # CrewAI / AG2 / LangChain / Google ADK adapters
│ ├── resilience.py # Rate limiter, retry, and circuit breaker
│ ├── semantic.py # Optional embedding-based guardrails/verifiers
│ ├── metrics.py # Telemetry collector
│ ├── server.py # FastAPI dashboard
│ ├── cli.py # Click CLI
│ ├── exceptions.py # Custom exceptions
│ ├── templates/
│ │ └── dashboard.html # Dashboard UI template
│ └── static/
│ └── tailwind.min.js # Bundled Tailwind CSS for offline dashboard
├── tests/
├── examples/
└── docs/
Development
Run lint, type checks, and tests:
pip install -e ".[dev,semantic]"
ruff check src tests examples scripts
ruff format --check src tests examples scripts
mypy src
pytest -v
Slow integration tests (e.g. loading sentence-transformers models) are skipped by
default. Run them with:
pytest -v --run-slow
For a reproducible provider/integration environment, apply the versions validated locally before installing the desired extras:
python -m pip install -c .github/constraints-runtime.txt \
-e ".[integrations,providers,semantic,qdrant]"
CI/CD
The GitHub Actions workflow runs linting, type checking, tests, and package builds on Python 3.10–3.13.
License
MIT License — see LICENSE.
Metadata
Release files for llmrivotril 0.1.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llmrivotril-0.1.6.tar.gz | 2.2 MB | Without provenance | Details for llmrivotril-0.1.6.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llmrivotril-0.1.6-py3-none-any.whl | Python 3 | none | any | Details for llmrivotril-0.1.6-py3-none-any.whl |
Total release size: 2.4 MB
Release files / llmrivotril-0.1.6.tar.gz
| Download URL | llmrivotril-0.1.6.tar.gz |
|---|---|
| Size | 2.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
78af6882600185de4fa43a0de40df4096644ee675eb0ea81ac57a5f796fe84f0
|
|
BLAKE2b-256 checksum How to use checksums |
fb0fec88471d17b2d209a59951a74150bd9901a01d8fb02c68f7901c15da76c3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Release files / llmrivotril-0.1.6-py3-none-any.whl
| Download URL | llmrivotril-0.1.6-py3-none-any.whl |
|---|---|
| Size | 216.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fab67dd1114d04715c38154a9a7cc4aee00228cedfeb6d049e35f57a16098b2b
|
|
BLAKE2b-256 checksum How to use checksums |
a6768f364dc7d2a70f0ba85e68c13b0e5acdcc596a27f3fa3ad7c613ce6e0300
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Security for llmrivotril 0.1.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.