Skip to main content

LLM-Rivotril

LLM-Rivotril logo

PyPI version Python versions

Português

A lightweight Python framework to reduce LLM hallucinations, enforce guardrails, manage stateful memory, and monitor performance through a local web dashboard.

Why LLM-Rivotril?

LLM-Rivotril is an orchestration and safety layer around an LLM; it is not a model and does not make a small model inherently smarter. It lets the same application move between local and hosted models while keeping a consistent interface for guardrails, memory, caching, retries, streaming, tool calling, telemetry, and framework integrations.

Use Ollama when privacy, offline operation, and predictable local costs matter; use a hosted provider when you need a larger or more capable model. In both cases, LLM-Rivotril keeps the application-level controls in one place.

Features

  • Guardrails — Block disallowed keywords, enforce allowed topics, limit output size, and validate JSON schemas on outputs.
  • Semantic Guardrails (optional) — Match prompts against allowed topics using dense embeddings instead of exact keywords.
  • Moderation Guardrail — Flag adversarial/unsafe content via OpenAI's moderation endpoint.
  • Stateful Memory — Sliding-window conversation store to prevent context drift, with optional automatic disk persistence.
  • Anti-Hallucination Verifier — Pluggable grounding checks, including keyword overlap, citation markers, and embedding-based faithfulness.
  • Resilience — Built-in rate limiting, retry with backoff, and circuit breaker for LLM calls.
  • Token-Saving Controls (optional) — Semantic (similarity-based) response cache, token-budget-aware memory trimming, and LLM-summarized history compaction.
  • Telemetry & Dashboard — Built-in FastAPI dashboard with live request logs, token usage, latency, and success rate; optional per-token auth.
  • Benchmark / Red-Team Evaluator — Labeled suite to measure guardrail and verifier accuracy without API costs.
  • CLI — Launch the dashboard with a single command.
  • Type-Safe Responses — Optional Pydantic response models via instructor, including streamed structured output.
  • Async API — run_async() for non-blocking execution.
  • Multi-Provider — OpenAI-compatible servers (Ollama, vLLM, ...), plus native Anthropic, Cohere, Gemini, Azure OpenAI, and AWS Bedrock adapters.
  • Vector-Store Retrievers (optional) — pgvector, Qdrant, Weaviate, and Pinecone adapters for RAG beyond in-memory scale.
  • Framework Integrations (optional) — Drop-in adapters for CrewAI, AG2/AutoGen, LangChain/LangGraph, and Google ADK, so guardrails/PII/RAG/observability apply inside those frameworks too.

Installation

pip install llmrivotril

For semantic (embedding-based) guardrails and verifiers:

pip install llmrivotril[semantic]

For local development:

git clone https://github.com/ailake-io/llmrivotril.git
cd llmrivotril
pip install -e ".[dev,semantic]"

This installs test, lint, type-check, and packaging tools (pytest, ruff, mypy, build, twine) plus every optional runtime dependency (all providers, RAG loaders, vector stores, Redis, framework integrations).

Quick Start

import os
from llmrivotril import RivotrilAgent, Guardrail
from llmrivotril.verifier import KeywordOverlapVerifier

agent = RivotrilAgent(
    model="gpt-4o-mini",
    api_key=os.getenv("OPENAI_API_KEY"),
    guardrails=[
        Guardrail(
            name="safe-content",
            allowed_topics=["AI safety", "machine learning"],
            disallowed_keywords=["password", "secret"],
            max_tokens=500,
        )
    ],
    verifier=KeywordOverlapVerifier(threshold=0.1),
)

response = agent.run("Explain what a guardrail is in AI safety.")
print(response)

Local Ollama

With Ollama running locally, use its OpenAI-compatible endpoint. The API key is only a placeholder required by the OpenAI client and is not sent to a remote provider:

ollama serve
ollama pull qwen2.5:3b
from llmrivotril import RivotrilAgent

agent = RivotrilAgent(
    model="qwen2.5:3b",
    base_url="http://127.0.0.1:11434/v1",
    api_key="ollama",
)
print(agent.run("Explain what a guardrail is in AI safety."))

The model quality, latency, and hardware requirements depend on the Ollama model selected. Keep Ollama bound to localhost when prompts must stay on the machine.

Documentation

  • Providers — OpenAI-compatible servers, Anthropic, Cohere, Gemini, Azure OpenAI, AWS Bedrock.
  • Guardrails & Safety — Semantic guardrails, moderation, PII redaction, token budget.
  • RAG — Local pipeline, vector-store retrievers (pgvector/Qdrant/Weaviate/Pinecone), grounding verification.
  • Framework Integrations — CrewAI, AG2/AutoGen, LangChain/LangGraph, Google ADK adapters, and multi-agent setup notes.
  • Async, Streaming & Function Calling
  • Reliability — Resilience, response caching (including semantic cache), memory token budget & summarization, schema-repair fallback.
  • Observability — Metrics persistence, local dashboard, benchmark, cost tracking.
  • Configuration — Config files, environment variables, plugins, project scaffolding.
  • Releasing — Build validation, TestPyPI, and the PyPI release workflow.

Interactive Demo

Run a complete walkthrough with mock LLM responses (no API key, no cost):

python examples/demo.py --mock --dashboard

Then open http://127.0.0.1:8767 to watch the dashboard update live.

Comparison: With vs. Without llmrivotril

See the same scenarios side-by-side:

python examples/comparison.py --mock

The comparison highlights how plain LLM calls return harmful or hallucinated answers that are only caught manually afterwards, while llmrivotril blocks them at runtime and records structured telemetry.

Project Structure

llmrivotril/
├── src/llmrivotril/
│   ├── agent.py              # RivotrilAgent orchestrator (sync + async)
│   ├── config.py             # Environment-variable and file configuration loader
│   ├── guardrails.py         # Input/output guardrails
│   ├── moderation.py         # OpenAI moderation-endpoint guardrail
│   ├── memory.py             # Conversation memory store
│   ├── verifier.py           # Hallucination / grounding checks
│   ├── providers.py          # OpenAI / Azure / Anthropic / Cohere / Gemini / Bedrock adapters
│   ├── rag/                  # RAG loaders, chunkers, retrievers, vector stores, and pipeline
│   ├── integrations/         # CrewAI / AG2 / LangChain / Google ADK adapters
│   ├── resilience.py         # Rate limiter, retry, and circuit breaker
│   ├── semantic.py           # Optional embedding-based guardrails/verifiers
│   ├── metrics.py            # Telemetry collector
│   ├── server.py             # FastAPI dashboard
│   ├── cli.py                # Click CLI
│   ├── exceptions.py         # Custom exceptions
│   ├── templates/
│   │   └── dashboard.html    # Dashboard UI template
│   └── static/
│       └── tailwind.min.js   # Bundled Tailwind CSS for offline dashboard
├── tests/
├── examples/
└── docs/

Development

Run lint, type checks, and tests:

pip install -e ".[dev,semantic]"
ruff check src tests examples scripts
ruff format --check src tests examples scripts
mypy src
pytest -v

Slow integration tests (e.g. loading sentence-transformers models) are skipped by default. Run them with:

pytest -v --run-slow

For a reproducible provider/integration environment, apply the versions validated locally before installing the desired extras:

python -m pip install -c .github/constraints-runtime.txt \
  -e ".[integrations,providers,semantic,qdrant]"

CI/CD

CI

The GitHub Actions workflow runs linting, type checking, tests, and package builds on Python 3.10–3.15.

Set max_output_tokens for a native per-generation output budget, including follow-ups and summaries. This is not a cumulative spending cap. For supported text-only OpenAI/Azure sessions, max_session_cost_usd plus explicit input/output prices enables an estimated session cost limit using atomic local reservations. See reservation limits and accounting; provider billing can still differ from local estimates. provider_usage tracks additional calls. AG2 supports function-tool events, structured validation and text streaming (AG2 1.3+, Python 3.11+). See integrations. Run local comparisons without a model account. Optional Jev checks and context selection are available through llmrivotril[jev]. They use TypeSafe by default or a local System One-compatible server such as local-jev with base_url=; local inference has no per-call provider charge. Ollama 0.35+ also supports this API; see the local setup. Input guardrails can run in a bounded pool with parallel_guardrails=True. Set max_concurrent_calls to bound provider concurrency per agent. Explicitly close streams when stopping early; see reliability.

Offline SDK contracts: pytest -q tests/test_http_contracts.py uses in-memory HTTP with dummy credentials. For telemetry without request or error content, pass MetricsCollector(include_content=False) to the agent. See observability.

Optional size controls: max_context_chars, max_attachment_bytes, and max_response_chars accept positive integers (default None). Response checks run locally and do not cap provider billing. See configuration for streaming and attachment semantics.

License

MIT License — see LICENSE.

Metadata

Release files for llmrivotril 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llmrivotril 0.2.1
File Size Uploaded Provenance Actions
llmrivotril-0.2.1.tar.gz 2.2 MB Without provenance Details for llmrivotril-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llmrivotril 0.2.1
File Interpreter ABI Platform Actions
llmrivotril-0.2.1-py3-none-any.whl Python 3 none any Details for llmrivotril-0.2.1-py3-none-any.whl

Total release size: 2.4 MB

Release files / llmrivotril-0.2.1.tar.gz

Download URL llmrivotril-0.2.1.tar.gz
Size 2.2 MB
Tags Source
SHA-256 checksum
How to use checksums
72d2d0f6ebd105f5c8cfa09b55102a48236be8d3d12ea7a65ece145497c5289b
BLAKE2b-256 checksum
How to use checksums
d52f0856867a2b4f7811ef178ff6aba19ccaa6998c3f99e0c8acf17db90e260c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / llmrivotril-0.2.1-py3-none-any.whl

Download URL llmrivotril-0.2.1-py3-none-any.whl
Size 226.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
133900162806ee588c06b25847e7fad1fc1549a61e693437da0b48ad48e896e0
BLAKE2b-256 checksum
How to use checksums
07f362ee309957ce246aa9f14714aaf14106a23155728e04c40e872770404d63
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for llmrivotril 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page