local-semantic-rag

A local-first Retrieval-Augmented Generation (RAG) library for Python.

Build semantic search and RAG applications using local embeddings, vector stores, and local LLMs — without sending your documents to external APIs.

Features

• 🔍 Semantic Search – Search documents using vector embeddings.
• 🧠 RAG Pipeline – Retrieve relevant context and generate grounded answers.
• 📄 Multiple Document Formats – TXT, PDF, Markdown, HTML, DOCX, CSV, XML, JSON, JSONL, and Excel.
• ✂️ Flexible Chunking – Fixed-size, sentence-based, and recursive chunking.
• 🔢 Local Embeddings – Powered by Sentence Transformers.
• 🗄️ Vector Stores – In-memory and optional FAISS support.
• 🔄 Reranking – Optional cross-encoder reranking.
• 🤖 Local LLMs – Ollama and Hugging Face Transformers.
• 📊 Evaluation – Precision@K, Recall@K, MRR, and Hit Rate.
• 🧩 Extensible – Replace or implement any core component.
• 🔒 Privacy First – Documents remain on your machine unless you explicitly use a remote service.
• 🐍 Library First – Designed to be imported into your Python applications.
• 💻 CLI Included – Index, search, and ask questions directly from the terminal.

Architecture

Documents
    │
    ▼
Document Loaders
    │
    ▼
Chunking
    │
    ▼
Embeddings
    │
    ▼
Vector Store
    │
    ▼
Retriever
    │
    ├── Metadata Filtering
    │
    └── Optional Reranking
    │
    ▼
RAG Pipeline
    │
    ▼
Local LLM
    │
    ▼
Answer + Sources

Installation

Core Package

pip install local-semantic-rag

All Optional Dependencies

pip install local-semantic-rag[all]

Individual Extras

pip install local-semantic-rag[pdf]
pip install local-semantic-rag[docx]
pip install local-semantic-rag[html]
pip install local-semantic-rag[markdown]
pip install local-semantic-rag[faiss]
pip install local-semantic-rag[excel]
pip install local-semantic-rag[transformers]
pip install local-semantic-rag[ollama]

Quick Start

1. Create a Knowledge Base

from local_semantic_rag import (
    Document,
    KnowledgeBase,
    SentenceTransformerEmbedding,
)

kb = KnowledgeBase(
    embedding_model=SentenceTransformerEmbedding()
)

2. Add Documents

kb.add_documents([
    Document(
        id="doc1",
        content="Laravel is a PHP framework."
    ),
    Document(
        id="doc2",
        content="Django is a Python framework."
    ),
])

3. Search

results = kb.search("PHP framework", top_k=5)

for result in results:
    print(result.document.content)
    print(f"Score: {result.score:.4f}")

4. Save the Index

kb.save("./my_index")

5. Load and Search Later

kb = KnowledgeBase.load("./my_index")

results = kb.search("PHP framework")

for result in results:
    print(result.document.content)

RAG with Ollama

local-semantic-rag can use Ollama for completely local RAG generation.

Install Ollama Support

pip install local-semantic-rag[ollama]

Install Ollama and pull a model:

ollama pull llama3.2

Create a RAG Pipeline

from local_semantic_rag import (
    KnowledgeBase,
    RAGPipeline,
    OllamaLLM,
)

kb = KnowledgeBase.load("./my_index")

llm = OllamaLLM(
    model="llama3.2"
)

rag = RAGPipeline(
    retriever=kb,
    llm=llm,
)

response = rag.ask("What is Laravel?")

print(response.answer)

for source in response.sources:
    print(
        f"- {source.document.id} "
        f"(score: {source.score:.4f})"
    )

Command Line Interface

local-semantic-rag also provides a simple CLI.

Index Documents

local-semantic-rag index ./documents --output ./my_index

Semantic Search

local-semantic-rag search ./my_index "PHP framework"

Ask a Question

local-semantic-rag ask ./my_index "What is Laravel?" --llm llama3.2

Supported Document Formats

local-semantic-rag supports loading documents from multiple formats:

Embeddings

The default embedding model is:

all-MiniLM-L6-v2

It provides:

• 384-dimensional embeddings
• Small model size
• Fast local inference
• Good general-purpose semantic search

Custom Embedding Model

from local_semantic_rag import SentenceTransformerEmbedding

embedder = SentenceTransformerEmbedding(
    model_name="all-MiniLM-L12-v2",
    device="cpu",
)

GPU can be enabled with:

embedder = SentenceTransformerEmbedding(
    model_name="all-MiniLM-L12-v2",
    device="cuda",
)

Models are lazy-loaded and downloaded only when first used.

Vector Stores

In-Memory

Good for development, testing, and small datasets.

from local_semantic_rag import InMemoryVectorStore

store = InMemoryVectorStore()

FAISS

For larger datasets and high-performance similarity search:

pip install local-semantic-rag[faiss]

from local_semantic_rag import FAISSVectorStore

store = FAISSVectorStore(
    dimension=384
)

Semantic Search

You can use the high-level API:

results = kb.search(
    "PHP web framework",
    top_k=5,
)

Or use the lower-level Retriever:

from local_semantic_rag import (
    Retriever,
    InMemoryVectorStore,
    SentenceTransformerEmbedding,
)

embedder = SentenceTransformerEmbedding()
store = InMemoryVectorStore()

retriever = Retriever(
    embedding_model=embedder,
    vector_store=store,
    top_k=5,
)

results = retriever.search("PHP framework")

Metadata Filtering

Search can be combined with metadata filters:

results = retriever.search(
    "API documentation",
    filters={
        "category": "documentation"
    },
)

Supported operators include:

=
!=
in
not in
contains
startswith
endswith

Example:

filters = {
    "language": {
        "op": "in",
        "value": ["en", "es"],
    },
    "category": {
        "op": "!=",
        "value": "draft",
    },
}

Reranking

For higher retrieval precision, you can use a cross-encoder reranker:

from local_semantic_rag import CrossEncoderReranker

reranker = CrossEncoderReranker(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
)

Then attach it to the retriever:

retriever = Retriever(
    embedding_model=embedder,
    vector_store=store,
    top_k=10,
    reranker=reranker,
)

Custom RAG Prompts

You can customize the prompt used by the RAG pipeline:

from local_semantic_rag import PromptTemplate

template = PromptTemplate(
    template="Context:\n{context}\n\nQuestion: {question}",
    system=(
        "You are an expert assistant. "
        "Answer accurately using the provided context."
    ),
    fallback="I don't have enough context to answer this.",
)

Use it with the RAG pipeline:

rag = RAGPipeline(
    retriever=kb,
    llm=llm,
    prompt_template=template,
)

Available placeholders:

{context}
{question}
{system}

Evaluation

local-semantic-rag provides retrieval evaluation metrics:

• Precision@K
• Recall@K
• MRR
• Hit Rate@K

Example:

from local_semantic_rag.evaluation import evaluate_retrieval

test_cases = [
    {
        "query": "PHP framework",
        "expected_ids": [
            "laravel_doc"
        ],
    },
    {
        "query": "Python framework",
        "expected_ids": [
            "django_doc"
        ],
    },
]

metrics = evaluate_retrieval(
    retriever=retriever,
    test_cases=test_cases,
    k=5,
)

print(metrics)

Example output:

{
    "precision": 0.8,
    "recall": 0.7,
    "mrr": 0.85,
    "hit_rate": 0.9
}

Extensibility

local-semantic-rag is built around abstract interfaces, making it easy to replace individual components.

For example, create a custom embedding implementation:

from local_semantic_rag import EmbeddingModel
from local_semantic_rag.types import Embedding
from typing import List

class MyCustomEmbedder(EmbeddingModel):

    def __init__(self):
        self._dim = 768

    def embed_documents(
        self,
        texts: List[str],
    ) -> List[Embedding]:
        return [
            [0.0] * self._dim
            for _ in texts
        ]

    def embed_query(
        self,
        text: str,
    ) -> Embedding:
        return [0.0] * self._dim

    @property
    def dimension(self) -> int:
        return self._dim

Use it with the knowledge base:

kb = KnowledgeBase(
    embedding_model=MyCustomEmbedder()
)

Project Structure

local-semantic-rag/
│
├── local_semantic_rag/
│   ├── __init__.py
│   ├── documents/
│   ├── chunking/
│   ├── embeddings/
│   ├── vectorstores/
│   ├── retrieval/
│   ├── llm/
│   ├── evaluation/
│   ├── pipeline/
│   └── cli.py
│
├── docs/
│   ├── index.md
│   ├── getting-started.md
│   ├── architecture.md
│   ├── embeddings.md
│   ├── semantic-search.md
│   ├── rag.md
│   ├── vector-stores.md
│   ├── llm-providers.md
│   ├── evaluation.md
│   └── extending.md
│
├── tests/
│
├── pyproject.toml
├── README.md
├── LICENSE
└── CONTRIBUTING.md

Design Principles

Modular

Use only the components you need and replace implementations when required.

Extensible

Core components are defined using abstract base classes.

Library-First

The framework is designed to be imported into Python applications rather than being limited to CLI usage.

Local-First

Embeddings, vector search, and LLM generation can all run locally.

Type-Safe

The project uses Python type hints throughout the core APIs.

Performance

local-semantic-rag includes several performance-focused design decisions:

• Lazy loading of embedding and LLM models.
• Batch document embedding.
• Vectorized similarity search.
• Optional FAISS acceleration.
• Configurable retrieval limits.
• Optional reranking.
• Lightweight core dependencies.

Security & Privacy

local-semantic-rag follows a local-first approach.

• Documents can remain entirely on your machine.
• No external API is required for the core RAG workflow.
• Local embeddings can be generated without cloud services.
• Local LLMs can be used through Ollama or Transformers.
• File validation helps prevent invalid or malicious input.

If you choose to integrate a remote embedding, vector database, or LLM provider, data handling will depend on that provider.

Documentation

Detailed documentation is available in the docs/ directory.

• Getting Started
• Architecture
• Embeddings
• Semantic Search
• RAG Pipeline
• Vector Stores
• LLM Providers
• Evaluation
• Extending the Framework

Development

Clone the repository:

git clone https://github.com/awais69735/local-semantic-rag.git
cd local-semantic-rag

Create a virtual environment:

python -m venv .venv

Activate it on Linux/macOS:

source .venv/bin/activate

Activate it on Windows:

.venv\Scripts\activate

Install the package in editable mode:

pip install -e .

Install development dependencies if available:

pip install -e ".[dev]"

Run tests:

pytest

Contributing

Contributions are welcome.

1. Fork the repository.
2. Create a feature branch.
3. Implement your changes.
4. Add or update tests.
5. Run the test suite.
6. Submit a pull request.

See CONTRIBUTING.md for contribution guidelines.

Roadmap

Potential future improvements include:

• Streaming LLM responses.
• Additional vector database integrations.
• Advanced RAG evaluation metrics.
• Hybrid keyword + semantic search.
• Improved document preprocessing.
• More reranking models.
• Async APIs.
• Additional local LLM backends.
• Better chunking strategies.
• Production-oriented observability and tracing.

License

This project is licensed under the terms specified in LICENSE.

Acknowledgements

local-semantic-rag builds on the Python open-source ecosystem, including:

• Sentence Transformers
• Hugging Face Transformers
• FAISS
• Ollama
• Pydantic

Status

🚧 Active Development

local-semantic-rag is designed as a lightweight foundation for building private, local-first semantic search and RAG applications in Python.

⭐ If you find this project useful, consider starring the repository and contributing improvements.