Skip to main content

Reddit Stash Insights

Python CI License: MIT

Companion to reddit-stash, adds semantic search and AI chat to your Reddit archive.

Scan, keyword search, and evaluation on a synthetic archive

How It Works

reddit-stash saves your Reddit content as markdown files. reddit-stash-insights indexes those files and lets you search and chat with them:

reddit-stash saves  -->  rsi indexes  -->  search & chat
   (markdown)          (BGE-M3 + LanceDB)    (RAG + LLM)

Features

  • Hybrid search: combines semantic (BGE-M3, 1024-dim) and keyword (BM25 via Tantivy) search with RRF reranking
  • RAG chat: ask questions about your archive using local or cloud LLMs with streaming responses
  • Bounded evidence: packs complete passages within the chat model budget and cites only included sources
  • Multiple LLM providers: llama-cpp (local GGUF), Ollama, OpenAI-compatible (vLLM, LM Studio)
  • Local fallback: switches between local providers; cloud fallback requires explicit consent
  • MCP server: expose search and chat as tools for Claude Code
  • Streamlit UI: web-based chat interface with source citations
  • S3 support: fetch reddit-stash data directly from S3 buckets
  • Repeatable evaluation: packaged questions, reference answers, relevance labels, and retrieval metrics
  • CLI: scan, index, search, chat, UI, evaluation, health checks, and MCP service

Quick Start

pip install "reddit-stash-insights[search]"
rsi index ~/reddit                   # Build search index (first run downloads ~2GB model)
rsi search "best python libraries"   # Search your archive

To add AI chat:

pip install "reddit-stash-insights[search,chat-ollama]"
# Install Ollama and pull a model first; see the provider guide below.
rsi chat --provider ollama --model qwen2.5:7b "What did I save about Python async programming?"

Getting Your Archive

reddit-stash saves to Dropbox, to S3 (under the reddit/ prefix) or to a local folder. rsi needs a local copy:

rsi index ~/reddit                              # Local reddit-stash output
rsi index ~/reddit --s3-bucket <bucket>         # Fetch s3://<bucket>/reddit/ first, then index
aws s3 sync s3://<bucket>/reddit/ ./reddit --exclude '.reddit-stash/*'   # Or sync it yourself

Dropbox users can sync or download the reddit-stash folder and point rsi at it.

Notes:

  • Every reddit-stash file type is indexed: your own posts and comments (POST_, COMMENT_), saved (SAVED_), upvoted (UPVOTE_) and GDPR exports (GDPR_). An item that is both saved and upvoted is indexed once.
  • The .reddit-stash/ folder holds reddit-stash's control files and is skipped.
  • file_log.json may contain failure entries without a type; these are ignored.
  • rsi index updates the existing index in place (rows are upserted, so re-running never duplicates them). After upgrading from a version that indexed only POST_/COMMENT_ files, delete the index (~/.rsi/index/vectors.lance by default) and run rsi index again so saved and upvoted items appear.

Installation

pip install reddit-stash-insights           # Core only (scan)

Source installation

git clone https://github.com/rhnfzl/reddit-stash-insights.git
cd reddit-stash-insights
pip install -e ".[dev]"

Install extras for additional features:

Extra What it adds Key dependencies
search Semantic + keyword search sentence-transformers, LanceDB, FlagEmbedding
index Vector storage without local embedding models LanceDB
embed-openai OpenAI-compatible embeddings API openai, tiktoken
chat Local LLM inference llama-cpp-python
chat-ollama Ollama LLM provider requests
chat-openai OpenAI-compatible provider openai
mcp MCP server for Claude Code mcp
ui Streamlit chat interface and index access streamlit, LanceDB
all Everything above All listed dependencies
# Common combinations
pip install "reddit-stash-insights[search,chat-ollama,ui]" # Local search and Ollama chat UI
pip install "reddit-stash-insights[all]"                   # Includes compiled llama-cpp support
pip install "reddit-stash-insights[search,chat-openai]"    # Search + OpenAI/vLLM

Usage

Scan

Inspect a reddit-stash directory without indexing:

rsi scan ~/reddit
# Found 1523 posts and 847 comments

Index

Build or update the vector search index:

rsi index ~/reddit
rsi index ~/reddit --s3-bucket my-bucket  # Fetch from S3 first, then index

Query your indexed content with three search modes:

rsi search "machine learning papers"              # Hybrid (default)
rsi search "transformers" --mode semantic          # Semantic only
rsi search "python tutorial" --mode keyword        # BM25 keyword only
rsi search "rust" --subreddit programming -n 5     # Filter by subreddit

Chat

Ask questions using RAG with your preferred LLM:

# Single question (streams response by default)
rsi chat "summarize my saved cooking recipes"

# Interactive REPL with conversation history
rsi chat
# you> what did I save about sourdough starters?
# you> /sources    (show source documents)
# you> /clear      (reset history)
# you> /quit

# Override provider for a session
rsi chat --provider ollama --model qwen2.5:7b
rsi chat --provider openai --model gpt-4o

Streamlit UI

Launch the web-based chat interface:

pip install "reddit-stash-insights[search,chat-ollama,ui]"
rsi ui

The UI opens on 127.0.0.1:8501. Use rsi ui --port 8502 if that port is busy. Configure a working LLM provider before asking chat questions.

Evaluation

Run the packaged synthetic check without downloading a model or providing private data:

pip install "reddit-stash-insights[index]"
rsi eval --fixture --mode keyword --split all --output /tmp/rsi-fixture.json

Reports include recall@10, MRR@10, nDCG@10, per-query rankings, errors, latency, and input hashes. The 50 answerable questions and 10 absent controls have source-checked answers and independent agent review. Synthetic scores check the runner; they do not establish real archive quality or superiority over another search system. See evaluation instructions for frozen-corpus evaluation and judgment limits.

MCP Server

Expose your Reddit archive as tools for Claude. Works with both the Claude desktop app and Claude Code.

pip install "reddit-stash-insights[search,chat-ollama,mcp]"

This exposes two tools:

Tool Description
reddit_search Search your archive (no LLM needed for results)
reddit_chat Ask questions with RAG, retrieves context and generates answers

See MCP setup for Claude desktop and Claude Code examples.

Configuration

Settings live in ~/.rsi/config.toml. Minimal example:

[llm]
provider = "llama-cpp"
model = "~/.rsi/models/qwen2.5-7b-instruct-q4_k_m.gguf"
allow_cloud_fallback = false

[chat]
context_docs = 5
search_mode = "hybrid"

Environment variables override config file values (e.g., RSI_LLM_PROVIDER, RSI_S3_BUCKET).

See configuration for the full reference.

LLM Providers

Provider Type Best for
llama-cpp Local GGUF Privacy, offline use, no API costs
ollama Local HTTP Easy model management, multiple models
openai Cloud/API OpenAI-compatible endpoints, including vLLM and LM Studio

The system checks the selected provider, then tries local alternatives (llama-cpp, ollama). OpenAI-compatible fallback requires [llm].allow_cloud_fallback = true or RSI_ALLOW_CLOUD_FALLBACK=true. Selecting openai directly also permits API calls. Retrieved context and conversation history go to that endpoint. CLI and UI show the selected destination before generation. MCP advertises configured destinations during tool discovery. Embedding privacy is configured separately: embedding.provider = "openai" sends indexed text and search queries to its API.

See provider setup for model installation instructions.

Feedback

Use Discussions for questions and feature requests. Report a missed result with the search that failed form. Share sanitised examples only. Keep private archives, queries, usernames, paths, and credentials out of public reports.

Development

pip install -e ".[all,dev]"

# Tests (unittest, not pytest)
python -m unittest tests.test_models tests.test_parser tests.test_scanner tests.test_cli tests.test_config -v  # Core
python -m unittest tests.test_chat_providers tests.test_chat_engine tests.test_cli_chat -v                     # Chat (mocked)
python -m unittest tests.test_embedder tests.test_vector_store tests.test_search tests.test_cli_search -v      # Search (slow, loads model)

# Lint
ruff check src/ tests/

# Multi-version testing
nox              # All sessions (3.11-3.14)
nox -s core      # Core tests only

Architecture

The system is organized in layers: Core (parsing, scanning) -> Indexer (embedding, vector store, search) -> Chat (RAG engine, LLM providers) -> UI/MCP (Streamlit, MCP server). Each layer depends only on the ones below it.

See architecture for the full system design.

License

MIT

Existing index migration

Passage indexing retains evidence from long threads and returns each source thread once. Existing whole-document indexes remain readable. Build passages in a separate directory:

rsi index ~/reddit --db-path ~/.rsi/index/passages.lance
RSI_DB_PATH=~/.rsi/index/passages.lance rsi health

Point RSI_DB_PATH at the new directory after validation. Keep the old index for rollback. The writer refuses legacy schemas and changed chunk settings. An explicit --rebuild discards the selected table. See index settings and migration for identity fields and retry behavior.

Metadata

Release files for reddit-stash-insights 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for reddit-stash-insights 0.1.0
File Size Uploaded Provenance Actions
reddit_stash_insights-0.1.0.tar.gz 208.2 kB With provenance Details for reddit_stash_insights-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for reddit-stash-insights 0.1.0
File Interpreter ABI Platform Actions
reddit_stash_insights-0.1.0-py3-none-any.whl Python 3 none any Details for reddit_stash_insights-0.1.0-py3-none-any.whl

Total release size: 307.7 kB

Release files / reddit_stash_insights-0.1.0.tar.gz

Download URL reddit_stash_insights-0.1.0.tar.gz
Size 208.2 kB
Tags Source
SHA-256 checksum
How to use checksums
40a408d095092f45d63a4425fc270dc9f5ee6c69a69105b139686ccfdda870ae
BLAKE2b-256 checksum
How to use checksums
649ef886946436ef4881bb9e4c4ba6075483103bdc87606703be524f6d90f9ae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.

Transparency log

Release files / reddit_stash_insights-0.1.0-py3-none-any.whl

Download URL reddit_stash_insights-0.1.0-py3-none-any.whl
Size 99.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0ec69744d011f5313aa53fddd7683d6960ac6176bb59b34b56550f3ba913f8d0
BLAKE2b-256 checksum
How to use checksums
1ca1d431b0e5d454c51e71f4d1d34013215c2761baecc9753f662c9399a2697b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.

Transparency log

Security for reddit-stash-insights 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for reddit_stash_insights-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 8, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page