local-semantic-rag

A local-first Retrieval-Augmented Generation (RAG) library for Python.

Build semantic search and RAG applications using local embeddings, vector stores, and local LLMs — without sending your documents to external APIs.

Features

• 🔍 Semantic Search – Search documents using vector embeddings.
• 🧠 RAG Pipeline – Retrieve relevant context and generate grounded answers.
• 📄 Multiple Document Formats – TXT, PDF, Markdown, HTML, DOCX, CSV, XML, JSON, JSONL, and Excel.
• ✂️ Flexible Chunking – Fixed-size, sentence-based, and recursive chunking.
• 🔢 Local Embeddings – Powered by Sentence Transformers.
• 🗄️ Vector Stores – In-memory and optional FAISS support.
• 🔄 Reranking – Optional cross-encoder reranking.
• 🤖 Local LLMs – Ollama and Hugging Face Transformers.
• 📊 Evaluation – Precision@K, Recall@K, MRR, and Hit Rate.
• 🧩 Extensible – Replace or implement any core component.
• 🔒 Privacy First – Documents remain on your machine unless you explicitly use a remote service.
• 🐍 Library First – Designed to be imported into your Python applications.
• 💻 CLI Included – Index, search, and ask questions directly from the terminal.

Architecture

Documents
    │
    ▼
Document Loaders
    │
    ▼
Chunking
    │
    ▼
Embeddings
    │
    ▼
Vector Store
    │
    ▼
Retriever
    │
    ├── Metadata Filtering
    │
    └── Optional Reranking
    │
    ▼
RAG Pipeline
    │
    ▼
Local LLM
    │
    ▼
Answer + Sources

Installation

Core Package

pip install local-semantic-rag

All Optional Dependencies

pip install local-semantic-rag[all]

Individual Extras

pip install local-semantic-rag[pdf]
pip install local-semantic-rag[docx]
pip install local-semantic-rag[html]
pip install local-semantic-rag[markdown]
pip install local-semantic-rag[faiss]
pip install local-semantic-rag[excel]
pip install local-semantic-rag[transformers]
pip install local-semantic-rag[ollama]

Quick Start

1. Create a Knowledge Base

from local_semantic_rag import (
    Document,
    KnowledgeBase,
    SentenceTransformerEmbedding,
)

kb = KnowledgeBase(
    embedding_model=SentenceTransformerEmbedding()
)

2. Add Documents

kb.add_documents([
    Document(
        id="doc1",
        content="Laravel is a PHP framework."
    ),
    Document(
        id="doc2",
        content="Django is a Python framework."
    ),
])

3. Search

results = kb.search("PHP framework", top_k=5)

for result in results:
    print(result.document.content)
    print(f"Score: {result.score:.4f}")

4. Save the Index

kb.save("./my_index")

5. Load and Search Later

kb = KnowledgeBase.load("./my_index")

results = kb.search("PHP framework")

for result in results:
    print(result.document.content)

RAG with Ollama

local-semantic-rag can use Ollama for completely local RAG generation.

Install Ollama Support

pip install local-semantic-rag[ollama]

Install Ollama and pull a model:

ollama pull llama3.2

Create a RAG Pipeline

from local_semantic_rag import (
    KnowledgeBase,
    RAGPipeline,
    OllamaLLM,
)

kb = KnowledgeBase.load("./my_index")

llm = OllamaLLM(
    model="llama3.2"
)

rag = RAGPipeline(
    retriever=kb,
    llm=llm,
)

response = rag.ask("What is Laravel?")

print(response.answer)

for source in response.sources:
    print(
        f"- {source.document.id} "
        f"(score: {source.score:.4f})"
    )

Command Line Interface

local-semantic-rag also provides a simple CLI.

Index Documents

local-semantic-rag index ./documents --output ./my_index

Semantic Search

local-semantic-rag search ./my_index "PHP framework"

Ask a Question

local-semantic-rag ask ./my_index "What is Laravel?" --llm llama3.2

Supported Document Formats

local-semantic-rag supports loading documents from multiple formats:

Format	Support
TXT	✅
PDF	✅
Markdown	✅
HTML	✅
DOCX	✅
CSV	✅
XML	✅
JSON	✅
JSONL	✅
Excel	✅

Embeddings

The default embedding model is:

all-MiniLM-L6-v2

It provides:

• 384-dimensional embeddings
• Small model size
• Fast local inference
• Good general-purpose semantic search

Custom Embedding Model

from local_semantic_rag import SentenceTransformerEmbedding

embedder = SentenceTransformerEmbedding(
    model_name="all-MiniLM-L12-v2",
    device="cpu",
)

GPU can be enabled with:

embedder = SentenceTransformerEmbedding(
    model_name="all-MiniLM-L12-v2",
    device="cuda",
)

Models are lazy-loaded and downloaded only when first used.

Vector Stores

In-Memory

Good for development, testing, and small datasets.

from local_semantic_rag import InMemoryVectorStore

store = InMemoryVectorStore()

FAISS

For larger datasets and high-performance similarity search:

pip install local-semantic-rag[faiss]

from local_semantic_rag import FAISSVectorStore

store = FAISSVectorStore(
    dimension=384
)

Semantic Search

You can use the high-level API:

results = kb.search(
    "PHP web framework",
    top_k=5,
)

Or use the lower-level Retriever:

from local_semantic_rag import (
    Retriever,
    InMemoryVectorStore,
    SentenceTransformerEmbedding,
)

embedder = SentenceTransformerEmbedding()
store = InMemoryVectorStore()

retriever = Retriever(
    embedding_model=embedder,
    vector_store=store,
    top_k=5,
)

results = retriever.search("PHP framework")

Metadata Filtering

Search can be combined with metadata filters:

results = retriever.search(
    "API documentation",
    filters={
        "category": "documentation"
    },
)

Supported operators include:

=
!=
in
not in
contains
startswith
endswith

Example:

filters = {
    "language": {
        "op": "in",
        "value": ["en", "es"],
    },
    "category": {
        "op": "!=",
        "value": "draft",
    },
}

Reranking

For higher retrieval precision, you can use a cross-encoder reranker:

from local_semantic_rag import CrossEncoderReranker

reranker = CrossEncoderReranker(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
)

Then attach it to the retriever:

retriever = Retriever(
    embedding_model=embedder,
    vector_store=store,
    top_k=10,
    reranker=reranker,
)

Custom RAG Prompts

You can customize the prompt used by the RAG pipeline:

from local_semantic_rag import PromptTemplate

template = PromptTemplate(
    template="Context:\n{context}\n\nQuestion: {question}",
    system=(
        "You are an expert assistant. "
        "Answer accurately using the provided context."
    ),
    fallback="I don't have enough context to answer this.",
)

Use it with the RAG pipeline:

rag = RAGPipeline(
    retriever=kb,
    llm=llm,
    prompt_template=template,
)

Available placeholders:

{context}
{question}
{system}

Evaluation

local-semantic-rag provides retrieval evaluation metrics:

• Precision@K
• Recall@K
• MRR
• Hit Rate@K

Example:

from local_semantic_rag.evaluation import evaluate_retrieval

test_cases = [
    {
        "query": "PHP framework",
        "expected_ids": [
            "laravel_doc"
        ],
    },
    {
        "query": "Python framework",
        "expected_ids": [
            "django_doc"
        ],
    },
]

metrics = evaluate_retrieval(
    retriever=retriever,
    test_cases=test_cases,
    k=5,
)

print(metrics)

Example output:

{
    "precision": 0.8,
    "recall": 0.7,
    "mrr": 0.85,
    "hit_rate": 0.9
}

Extensibility

local-semantic-rag is built around abstract interfaces, making it easy to replace individual components.

Component	Extension
Document Loader	DocumentLoader
Chunker	Chunker
Embedding	EmbeddingModel
Vector Store	VectorStore
Reranker	Reranker
LLM	LLM

For example, create a custom embedding implementation:

from local_semantic_rag import EmbeddingModel
from local_semantic_rag.types import Embedding
from typing import List

class MyCustomEmbedder(EmbeddingModel):

    def __init__(self):
        self._dim = 768

    def embed_documents(
        self,
        texts: List[str],
    ) -> List[Embedding]:
        return [
            [0.0] * self._dim
            for _ in texts
        ]

    def embed_query(
        self,
        text: str,
    ) -> Embedding:
        return [0.0] * self._dim

    @property
    def dimension(self) -> int:
        return self._dim

Use it with the knowledge base:

kb = KnowledgeBase(
    embedding_model=MyCustomEmbedder()
)

Project Structure

local-semantic-rag/
│
├── local_semantic_rag/
│   ├── __init__.py
│   ├── documents/
│   ├── chunking/
│   ├── embeddings/
│   ├── vectorstores/
│   ├── retrieval/
│   ├── llm/
│   ├── evaluation/
│   ├── pipeline/
│   └── cli.py
│
├── docs/
│   ├── index.md
│   ├── getting-started.md
│   ├── architecture.md
│   ├── embeddings.md
│   ├── semantic-search.md
│   ├── rag.md
│   ├── vector-stores.md
│   ├── llm-providers.md
│   ├── evaluation.md
│   └── extending.md
│
├── tests/
│
├── pyproject.toml
├── README.md
├── LICENSE
└── CONTRIBUTING.md

Design Principles

Modular

Use only the components you need and replace implementations when required.

Extensible

Core components are defined using abstract base classes.

Library-First

The framework is designed to be imported into Python applications rather than being limited to CLI usage.

Local-First

Embeddings, vector search, and LLM generation can all run locally.

Type-Safe

The project uses Python type hints throughout the core APIs.

Performance

local-semantic-rag includes several performance-focused design decisions:

• Lazy loading of embedding and LLM models.
• Batch document embedding.
• Vectorized similarity search.
• Optional FAISS acceleration.
• Configurable retrieval limits.
• Optional reranking.
• Lightweight core dependencies.

Security & Privacy

local-semantic-rag follows a local-first approach.

• Documents can remain entirely on your machine.
• No external API is required for the core RAG workflow.
• Local embeddings can be generated without cloud services.
• Local LLMs can be used through Ollama or Transformers.
• File validation helps prevent invalid or malicious input.

If you choose to integrate a remote embedding, vector database, or LLM provider, data handling will depend on that provider.

Documentation

Detailed documentation is available in the docs/ directory.

• Getting Started
• Architecture
• Embeddings
• Semantic Search
• RAG Pipeline
• Vector Stores
• LLM Providers
• Evaluation
• Extending the Framework

Development

Clone the repository:

git clone https://github.com/awais69735/local-semantic-rag.git
cd local-semantic-rag

Create a virtual environment:

python -m venv .venv

Activate it on Linux/macOS:

source .venv/bin/activate

Activate it on Windows:

.venv\Scripts\activate

Install the package in editable mode:

pip install -e .

Install development dependencies if available:

pip install -e ".[dev]"

Run tests:

pytest

Contributing

Contributions are welcome.

1. Fork the repository.
2. Create a feature branch.
3. Implement your changes.
4. Add or update tests.
5. Run the test suite.
6. Submit a pull request.

See CONTRIBUTING.md for contribution guidelines.

Roadmap

Potential future improvements include:

• Streaming LLM responses.
• Additional vector database integrations.
• Advanced RAG evaluation metrics.
• Hybrid keyword + semantic search.
• Improved document preprocessing.
• More reranking models.
• Async APIs.
• Additional local LLM backends.
• Better chunking strategies.
• Production-oriented observability and tracing.

License

This project is licensed under the terms specified in LICENSE.

Acknowledgements

local-semantic-rag builds on the Python open-source ecosystem, including:

• Sentence Transformers
• Hugging Face Transformers
• FAISS
• Ollama
• Pydantic

Status

🚧 Active Development

local-semantic-rag is designed as a lightweight foundation for building private, local-first semantic search and RAG applications in Python.

⭐ If you find this project useful, consider starring the repository and contributing improvements.