lore-llm 0.3.0

Local-first LLM instance framework: deploy, specialize, and chat with grounded, cited model instances.

Lore

Local-first LLM instance framework — deploy, manage, and specialize LLM instances without sending private data to commercial AI providers or paying per token.

Working name; see CLAUDE.md for the full design document that governs this project, and docs/ROADMAP.md for current sequencing.

What it does

• One-command local setup: lore setup installs Ollama (with your confirmation), starts it, and pulls models sized to your hardware. lore doctor diagnoses anything broken with the exact command to fix it.
• A vetted, free model catalog: every supported model is open-weight and free to run. lore models catalog shows what fits your hardware; lore models guide maps use cases to the best model for this machine. A weekly CI job re-verifies the catalog against the Ollama registry.
• Named, specialized instances: each instance = model + private knowledge base(s) + persistent memory + behavior config, all defined in YAML.
• Grounded, cited answers: responses are grounded in the instance's knowledge bases (PDFs, markdown, text, and source code chunked per function/class) with page/section/line citations.
• A configurable grounding gate: weak retrieval refuses before generation; after generation, a judge model verifies each claim against the retrieved sources and refuses, hedges, or warns below your threshold — always saying why and showing the nearest chunks. Optional self-consistency sampling flags unstable answers.
• Per-instance memory across sessions — pinned facts and rolling summaries, fully inspectable, deletable with a wipe that VACUUMs the database so the data is actually gone.
• A generic analysis task engine: define batch workflows in YAML (enumerate items → grounded verdict per item → cited report). AOU gap analysis, requirements coverage, and test-gap templates ship built in; your own workflows are a YAML file away.
• CLI-first with web parity by construction: the CLI is a thin client of a local FastAPI server; the future web UI consumes the same API.

Deployment model: set Lore up independently on each of your machines — natively (lore setup) or via Docker (deploy/). Each machine hosts its own instances, CLI, and (coming next) its own web GUI served by lore-server. Machines are independent by design; cloud provisioning is tabled (see roadmap).

An honest note on "accuracy"

No system can guarantee a numeric accuracy percentage, and Lore does not claim to. What it implements is a grounding gate (see CLAUDE.md §4.3): claims in a draft answer are verified against retrieved chunks, and answers below the configured grounding score are refused, hedged, or flagged — structurally preventing confident assertion of unsupported claims. Accuracy stays measured (via the grounding eval harness), never asserted.

Status

Phase	Scope	Status
0	Scaffold, config schemas, SQLite store, server + CLI lifecycle	✅ done
1	Provider adapters (Ollama, OpenAI-compat), instances, streaming chat	✅ done
2	RAG & knowledge bases (ingest, embed, retrieve, cite; PDF/markdown/code)	✅ done
3	Grounding gate, memory, analysis task engine, self-consistency	✅ done
4	Docker packaging + token auth (cloud provisioning tabled)	✅ done
5	Per-machine web dashboard served by lore-server at /	✅ done

Docker (one command per machine)

cd deploy
export LORE_API_TOKEN=$(openssl rand -hex 24)
docker compose up -d --build     # lore-server + Ollama + persistent volumes
docker compose exec lore lore models pull qwen2.5:7b

See deploy/README.md for KB bind-mounts, GPU passthrough, upgrades, and talking to a Lore machine over the network (the API requires the bearer token whenever it's reachable beyond loopback).

Installing

Lore is on PyPI as lore-llm — the command is lore:

pipx install lore-llm        # or: uv tool install lore-llm
lore setup                   # install + start Ollama, pull models
lore up                      # start the server (and the dashboard)

Working from a checkout instead? uv runs the CLI inside the project venv with pinned dependencies: uv run lore … (the Quickstart below uses this form; installed users just type lore …).

Web dashboard

lore up also serves a full web dashboard at the server root (default http://127.0.0.1:5673) — the same API the CLI uses, so the two stay in feature parity by construction:

• Chat with streaming answers, inline citation chips, an evidence drawer showing every claim and retrieved chunk, and refusal cards that explain why the gate held an answer back.
• Instances: create and edit instance config with a form ↔ YAML toggle (everything is YAML underneath), including the grounding gate.
• Knowledge bases: create, ingest with live progress, browse snapshots.
• Memory: inspect pinned facts and summaries, pin, forget, or wipe (type-the-name confirmation; rows deleted and the database vacuumed).
• Models: installed models, the hardware-fit catalog, and pull-from-GUI with live download progress.
• Settings: global config with the same YAML toggle, plus lore doctor in the browser.

Two themes, four accent colors, self-hosted fonts — no external requests, consistent with local-first. When the server binds beyond loopback (e.g. Docker), set the bearer token under Settings → API token.

Quickstart

Requires Python 3.11+ and uv.

uv sync                 # create venv, install lore + dev deps
uv run lore setup       # install + start Ollama, pull hardware-sized models
uv run lore doctor      # diagnose: provider, models, GPU, disk, server

uv run lore models guide     # which model for which job, sized to THIS machine
uv run lore models catalog   # all supported models: memory needs, fit, installed

# chat with a general instance
uv run lore instance create helper --model qwen2.5:7b \
    --system-prompt "You are a concise engineering assistant."
uv run lore chat helper                    # interactive, streaming
uv run lore ask helper "what can you do?"  # one-shot; --json for scripts

lore setup detects your GPU/RAM and recommends models accordingly (e.g. 24 GB VRAM → qwen2.5:32b-instruct, 8 GB → qwen2.5:7b, CPU-only laptop → qwen2.5:1.5b) — recommendations, not requirements. All catalog models are open-weight and free to run (no per-token costs — that's the point); licenses vary per family, so check the notes for commercial use. Already have Ollama? Setup detects it and fills in only what's missing. No Ollama at all? Point providers.openai_compat.base_url at any OpenAI-compatible endpoint (llama.cpp server, vLLM, LM Studio) and use --provider openai_compat.

Ground an instance in your own material

# ingest documents (pdf, md, txt, rst) and/or code (py, c, h, cpp — per-function chunks)
uv run lore kb create sa8775-docs --source ~/datasheets/sa8775/

cat > expert.yaml <<'EOF'
name: sa8775-expert
model: qwen2.5:7b
system_prompt: Answer strictly from the mounted knowledge base.
knowledge_bases:
  - name: sa8775-docs      # pin a snapshot hash instead of latest for reproducibility
accuracy:
  min_grounding_score: 0.85
  on_below_threshold: refuse
EOF
uv run lore instance create sa8775-expert --from expert.yaml

uv run lore ask sa8775-expert "what is the max junction temperature?"
# → cited answer:  ... 105 C [1]
#   [1] trm.pdf p.87 · Thermal > Limits (score 0.83)   ● grounded 0.94
# → or an explicit refusal with the nearest chunks when the KB doesn't cover it

KBs are versioned: every ingest is an immutable snapshot (lore kb update, lore kb snapshots <name>); instances mount latest or a pinned hash.

Memory

uv run lore memory pin sa8775-expert "our target is SA8797 rev B"
uv run lore memory show sa8775-expert        # pinned facts, session summaries
uv run lore memory wipe sa8775-expert --all  # deleted AND vacuumed — unrecoverable

Batch analysis (generic task engine)

uv run lore analyze list                     # built-ins + your ~/.lore/tasks/*.yaml
uv run lore analyze run aou-gap --instance code-auditor \
    --items aou.md --md gap-report.md
# per item: retrieval → gate → grounded verdict with citations;
# items the KB can't support are marked insufficient-evidence, never guessed

Any workflow of the shape enumerate → grounded verdict → report is a YAML template — labels, prompt, and item splitting are all yours to define.

Everything is configured via ~/.lore/config.yaml (override the home dir with LORE_HOME). Every YAML file validates against a Pydantic schema with precise, user-facing errors.

Repository layout

src/lore/
├── cli/          # Typer app; thin HTTP client of the server
├── server/
│   ├── api/      # FastAPI routers: health, models, instances, kb, chat,
│   │             #   memory, analyze, system (doctor/config)
│   ├── core/     # instance manager, chat orchestration, KB mounting,
│   │             #   analysis engine, task templates
│   ├── rag/      # parsing (PDF/markdown/code), chunking, ingestion,
│   │             #   hybrid retrieval (dense + BM25)
│   ├── verify/   # grounding judge, self-consistency
│   ├── memory/   # budgeted injection, rolling summaries, wipe
│   └── providers/# Ollama + OpenAI-compat adapters behind the Provider ABC
├── config/       # Pydantic schemas for all YAML files + Lore home paths
├── setup/        # hardware detection, Ollama install/start, doctor checks
├── store/        # SQLite (WAL, migrations) + ChromaDB vector store
└── tasks/        # built-in analysis templates (aou-gap, …)

Development

See CONTRIBUTING.md for setup, checks, and project conventions.