Reddit Stash Insights
Companion to reddit-stash, adds semantic search and AI chat to your Reddit archive.
How It Works
reddit-stash saves your Reddit content as markdown files. reddit-stash-insights indexes those files and lets you search and chat with them:
reddit-stash saves --> rsi indexes --> search & chat
(markdown) (BGE-M3 + LanceDB) (RAG + LLM)
Features
- Hybrid search: combines semantic (BGE-M3, 1024-dim) and keyword (BM25 via Tantivy) search with RRF reranking
- RAG chat: ask questions about your archive using local or cloud LLMs with streaming responses
- Bounded evidence: packs complete passages within the chat model budget and cites only included sources
- Multiple LLM providers: llama-cpp (local GGUF), Ollama, OpenAI-compatible (vLLM, LM Studio)
- Local fallback: switches between local providers; cloud fallback requires explicit consent
- MCP server: expose search and chat as tools for Claude Code
- Streamlit UI: web-based chat interface with source citations
- S3 support: fetch reddit-stash data directly from S3 buckets
- Repeatable evaluation: packaged questions, reference answers, relevance labels, and retrieval metrics
- CLI: scan, index, search, chat, UI, evaluation, health checks, and MCP service
Quick Start
pip install "reddit-stash-insights[search]"
rsi index ~/reddit # Build search index (first run downloads ~2GB model)
rsi search "best python libraries" # Search your archive
To add AI chat:
pip install "reddit-stash-insights[search,chat-ollama]"
# Install Ollama and pull a model first; see the provider guide below.
rsi chat --provider ollama --model qwen2.5:7b "What did I save about Python async programming?"
Getting Your Archive
reddit-stash saves to Dropbox, to S3 (under the reddit/ prefix) or to a local folder. rsi needs a local copy:
rsi index ~/reddit # Local reddit-stash output
rsi index ~/reddit --s3-bucket <bucket> # Fetch s3://<bucket>/reddit/ first, then index
aws s3 sync s3://<bucket>/reddit/ ./reddit --exclude '.reddit-stash/*' # Or sync it yourself
Dropbox users can sync or download the reddit-stash folder and point rsi at it.
Notes:
- Every reddit-stash file type is indexed: your own posts and comments (
POST_,COMMENT_), saved (SAVED_), upvoted (UPVOTE_) and GDPR exports (GDPR_). An item that is both saved and upvoted is indexed once. - The
.reddit-stash/folder holds reddit-stash's control files and is skipped. file_log.jsonmay contain failure entries without atype; these are ignored.rsi indexupdates the existing index in place (rows are upserted, so re-running never duplicates them). After upgrading from a version that indexed onlyPOST_/COMMENT_files, delete the index (~/.rsi/index/vectors.lanceby default) and runrsi indexagain so saved and upvoted items appear.
Installation
pip install reddit-stash-insights # Core only (scan)
Source installation
git clone https://github.com/rhnfzl/reddit-stash-insights.git
cd reddit-stash-insights
pip install -e ".[dev]"
Install extras for additional features:
| Extra | What it adds | Key dependencies |
|---|---|---|
search |
Semantic + keyword search | sentence-transformers, LanceDB, FlagEmbedding |
index |
Vector storage without local embedding models | LanceDB |
embed-openai |
OpenAI-compatible embeddings API | openai, tiktoken |
chat |
Local LLM inference | llama-cpp-python |
chat-ollama |
Ollama LLM provider | requests |
chat-openai |
OpenAI-compatible provider | openai |
mcp |
MCP server for Claude Code | mcp |
ui |
Streamlit chat interface and index access | streamlit, LanceDB |
all |
Everything above | All listed dependencies |
# Common combinations
pip install "reddit-stash-insights[search,chat-ollama,ui]" # Local search and Ollama chat UI
pip install "reddit-stash-insights[all]" # Includes compiled llama-cpp support
pip install "reddit-stash-insights[search,chat-openai]" # Search + OpenAI/vLLM
Usage
Scan
Inspect a reddit-stash directory without indexing:
rsi scan ~/reddit
# Found 1523 posts and 847 comments
Index
Build or update the vector search index:
rsi index ~/reddit
rsi index ~/reddit --s3-bucket my-bucket # Fetch from S3 first, then index
Search
Query your indexed content with three search modes:
rsi search "machine learning papers" # Hybrid (default)
rsi search "transformers" --mode semantic # Semantic only
rsi search "python tutorial" --mode keyword # BM25 keyword only
rsi search "rust" --subreddit programming -n 5 # Filter by subreddit
Chat
Ask questions using RAG with your preferred LLM:
# Single question (streams response by default)
rsi chat "summarize my saved cooking recipes"
# Interactive REPL with conversation history
rsi chat
# you> what did I save about sourdough starters?
# you> /sources (show source documents)
# you> /clear (reset history)
# you> /quit
# Override provider for a session
rsi chat --provider ollama --model qwen2.5:7b
rsi chat --provider openai --model gpt-4o
Streamlit UI
Launch the web-based chat interface:
pip install "reddit-stash-insights[search,chat-ollama,ui]"
rsi ui
The UI opens on 127.0.0.1:8501. Use rsi ui --port 8502 if that port is busy.
Configure a working LLM provider before asking chat questions.
Evaluation
Run the packaged synthetic check without downloading a model or providing private data:
pip install "reddit-stash-insights[index]"
rsi eval --fixture --mode keyword --split all --output /tmp/rsi-fixture.json
Reports include recall@10, MRR@10, nDCG@10, per-query rankings, errors, latency, and input hashes. The 50 answerable questions and 10 absent controls have source-checked answers and independent agent review. Synthetic scores check the runner; they do not establish real archive quality or superiority over another search system. See evaluation instructions for frozen-corpus evaluation and judgment limits.
MCP Server
Expose your Reddit archive as tools for Claude. Works with both the Claude desktop app and Claude Code.
pip install "reddit-stash-insights[search,chat-ollama,mcp]"
This exposes two tools:
| Tool | Description |
|---|---|
reddit_search |
Search your archive (no LLM needed for results) |
reddit_chat |
Ask questions with RAG, retrieves context and generates answers |
See MCP setup for Claude desktop and Claude Code examples.
Configuration
Settings live in ~/.rsi/config.toml. Minimal example:
[llm]
provider = "llama-cpp"
model = "~/.rsi/models/qwen2.5-7b-instruct-q4_k_m.gguf"
allow_cloud_fallback = false
[chat]
context_docs = 5
search_mode = "hybrid"
Environment variables override config file values (e.g., RSI_LLM_PROVIDER, RSI_S3_BUCKET).
See configuration for the full reference.
LLM Providers
| Provider | Type | Best for |
|---|---|---|
llama-cpp |
Local GGUF | Privacy, offline use, no API costs |
ollama |
Local HTTP | Easy model management, multiple models |
openai |
Cloud/API | OpenAI-compatible endpoints, including vLLM and LM Studio |
The system checks the selected provider, then tries local alternatives (llama-cpp, ollama).
OpenAI-compatible fallback requires [llm].allow_cloud_fallback = true or RSI_ALLOW_CLOUD_FALLBACK=true.
Selecting openai directly also permits API calls. Retrieved context and conversation history go to that endpoint.
CLI and UI show the selected destination before generation. MCP advertises configured destinations during tool discovery.
Embedding privacy is configured separately: embedding.provider = "openai" sends indexed text and search queries to its API.
See provider setup for model installation instructions.
Feedback
Use Discussions for questions and feature requests. Report a missed result with the search that failed form. Share sanitised examples only. Keep private archives, queries, usernames, paths, and credentials out of public reports.
Development
pip install -e ".[all,dev]"
# Tests (unittest, not pytest)
python -m unittest tests.test_models tests.test_parser tests.test_scanner tests.test_cli tests.test_config -v # Core
python -m unittest tests.test_chat_providers tests.test_chat_engine tests.test_cli_chat -v # Chat (mocked)
python -m unittest tests.test_embedder tests.test_vector_store tests.test_search tests.test_cli_search -v # Search (slow, loads model)
# Lint
ruff check src/ tests/
# Multi-version testing
nox # All sessions (3.11-3.14)
nox -s core # Core tests only
Architecture
The system is organized in layers: Core (parsing, scanning) -> Indexer (embedding, vector store, search) -> Chat (RAG engine, LLM providers) -> UI/MCP (Streamlit, MCP server). Each layer depends only on the ones below it.
See architecture for the full system design.
License
Existing index migration
Passage indexing retains evidence from long threads and returns each source thread once. Existing whole-document indexes remain readable. Build passages in a separate directory:
rsi index ~/reddit --db-path ~/.rsi/index/passages.lance
RSI_DB_PATH=~/.rsi/index/passages.lance rsi health
Point RSI_DB_PATH at the new directory after validation. Keep the old index for rollback.
The writer refuses legacy schemas and changed chunk settings. An explicit --rebuild discards the selected table.
See index settings and migration for identity fields and retry behavior.
Metadata
Release files for reddit-stash-insights 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| reddit_stash_insights-0.1.0.tar.gz | 208.2 kB | With provenance | Details for reddit_stash_insights-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| reddit_stash_insights-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for reddit_stash_insights-0.1.0-py3-none-any.whl |
Total release size: 307.7 kB
Release files / reddit_stash_insights-0.1.0.tar.gz
| Download URL | reddit_stash_insights-0.1.0.tar.gz |
|---|---|
| Size | 208.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
40a408d095092f45d63a4425fc270dc9f5ee6c69a69105b139686ccfdda870ae
|
|
BLAKE2b-256 checksum How to use checksums |
649ef886946436ef4881bb9e4c4ba6075483103bdc87606703be524f6d90f9ae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logRelease files / reddit_stash_insights-0.1.0-py3-none-any.whl
| Download URL | reddit_stash_insights-0.1.0-py3-none-any.whl |
|---|---|
| Size | 99.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0ec69744d011f5313aa53fddd7683d6960ac6176bb59b34b56550f3ba913f8d0
|
|
BLAKE2b-256 checksum How to use checksums |
1ca1d431b0e5d454c51e71f4d1d34013215c2761baecc9753f662c9399a2697b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logSecurity for reddit-stash-insights 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for reddit_stash_insights-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.