lorme

LORME — Local Offline Retrieval & MCP Engine

An MCP server for semantic search over your local documents. Preprocesses, indexes, and serves any format — PDF, HTML, DOCX, markdown, and more. No cloud, no external database — just a single .db file on your workstation.

What it does

• Preprocesses source documents in any format (PDF, HTML, DOCX, PPTX, XLSX, EPUB, images, audio, video, CSV, JSON, XML, markdown) into clean markdown
• Indexes with vector embeddings and full-text search (hybrid FTS5 + vector with RRF fusion)
• Serves three MCP tools (search_docs, list_indexed_sources, list_collections) for any MCP client
• Evaluates retrieval quality with built-in RAG evaluation and parameter optimization
• Runs locally with automatic GPU/API/CPU fallback for embedding generation

Quickstart

1. Install

git clone https://github.com/romainsc/lorme.git
cd lorme
python -m venv .venv
source .venv/bin/activate
pip install -e ".[parse]"

Install extras by need:

• pip install -e . — core (markdown only)
• pip install -e ".[html]" — add HTML support (trafilatura)
• pip install -e ".[pdf]" — add PDF/DOCX/PPTX support (Docling)
• pip install -e ".[video]" — add video/audio download from YouTube/platforms (yt-dlp)
• pip install -e ".[parse]" — all format support
• pip install -e ".[eval]" — RAG evaluation (RAGAS)

2. Create a config file

# config.yaml
database:
  path: ./lore.db

embedding:
  model: nomic-ai/nomic-embed-text-v2-moe
  mode: builtin    # builtin, builtin:gpu, builtin:cpu, api

chunking:
  chunk_size: 1024
  chunk_overlap: 128

See docs/configuration.md for all options.

3. Create a manifest

# manifest.yaml
collection: my-docs
level: libre

sources:
  - title: Architecture Guide
    license: Apache-2.0
    orig: architecture.pdf

  - title: API Reference
    orig: api-ref.html

  - url: https://example.com/guide.md

The manifest declares sources abstractly. orig is the source file in its native format. path (output filename) is generated automatically.

4. Preprocess and build

# Preprocess: convert + clean sources
lorme preprocess manifest.yaml \
  --config config.yaml \
  --docs-base-dir /path/to/corpus/ \
  --orig-dir orig \
  --prep-dir prep

# Build: index preprocessed sources
lorme build manifest-prep.yaml \
  --config config.yaml \
  --docs-dir /path/to/corpus/prep/ \
  --output-dir /path/to/output/ \
  --skip-optimize

Or combine both in one step:

lorme build manifest.yaml \
  --config config.yaml \
  --docs-dir /path/to/corpus/orig/ \
  --output-dir /path/to/output/ \
  --preprocess --skip-optimize

5. Start the MCP server

Option A: HTTP server (recommended)

lorme --config config.yaml --transport sse

{
  "mcpServers": {
    "lore": {
      "url": "http://localhost:8000/sse"
    }
  }
}

Option B: subprocess (stdio)

{
  "mcpServers": {
    "lore": {
      "command": "/path/to/.venv/bin/lorme",
      "args": ["--config", "/path/to/config.yaml"]
    }
  }
}

6. Use from your MCP client

search_docs("how to configure authentication")
search_docs("deployment", top_k=10, collection="docs-libre")
list_indexed_sources()
list_collections()

Search uses hybrid retrieval (vector + FTS5 full-text) with Reciprocal Rank Fusion for better keyword matching alongside semantic similarity.

CLI commands

Command	Purpose
lorme	Start MCP server
lorme preprocess	Convert and clean sources
lorme build	Index sources into .db
lorme lint	Analyze source quality
lorme eval	Evaluate retrieval quality
lorme optimize	Auto-optimize parameters
lorme enrich	LLM enrichment (context, Q&A)
lorme state	Manage pipeline state (--list, --purge)

All commands accept --config config.yaml.

Architecture

Uses nomic-ai/nomic-embed-text-v2-moe for embeddings (768 dimensions, multilingual, Apache 2.0) and sqlite-vec for vector storage. Hybrid search combines vector KNN with FTS5 full-text via RRF fusion.

Preprocessing uses trafilatura (HTML, Apache 2.0), Docling (PDF/DOCX, MIT), and markitdown (CSV/JSON/XML, MIT).

See docs/architecture.md for the full design.

Roadmap

Done

• SQLite + sqlite-vec storage with model validation
• Embedding with GPU/API/CPU fallback
• MCP server (search_docs, list_indexed_sources, list_collections)
• Multi-collection support with license classification
• Preprocessing tool (multi-format parsing, clean, dedup, PII detection, quality gate)
• LLM enrichment (contextual retrieval, Q&A mode)
• Hybrid search (FTS5 + vector + RRF fusion)
• RAG evaluation and parameter optimization
• Build workflow (manifest + config → optimized .db + metadata)
• Unified config.yaml (no env vars)
• MarkdownTextSplitter (structure-aware chunking)
• Reranking (cross-encoder, granite reranker)
• Adjacent-chunk / parent-child retrieval
• End-to-end parameter optimization
• CI/CD with GitHub Actions
• Docker image (Containerfile)
• Audio ingestion (STT via API)
• Video ingestion (STT + frame extraction, 4 strategies: scene/interval/hybrid/OCR)
• Video download from YouTube/platforms (yt-dlp, caption download → skip STT)
• Resumable pipeline with per-phase hash (config change → auto re-run)
• Configurable enrichment prompts (prompts.yaml, custom overrides)
• Smart frame captioning filter (OCR before VLM)
• Graceful shutdown (Ctrl+C → cleanup)
• 554+ tests (TDD)

Next

• pip install lorme (PyPI)
• STT post-correction (LLM fix of transcription errors)
• Detailed eval report (markdown)

AI-assisted development

This project is developed with AI assistance (Claude, Anthropic). All AI-assisted content is marked with Assisted-by and Co-Authored-By trailers in commits. Every contribution is reviewed, tested, and validated by a human.

See docs/ai-guidelines.md.

License

AGPL-3.0-or-later — see docs/adr/001-license-gpl-v3.md for the rationale.

Copyright (C) 2026 Romain Chantereau