ragwire 1.3.1

RAGWire — Production-grade RAG toolkit for document ingestion and retrieval with hybrid search support

RAGWire

Production-grade RAG toolkit for document ingestion and retrieval

---

Features

• Document Loading — PDF, DOCX, XLSX, PPTX and more via MarkItDown
• LLM Metadata Extraction — extracts company, doc type, fiscal period using your LLM; fully customisable via YAML
• Smart Text Splitting — markdown-aware and recursive chunking strategies
• Multiple Embedding Providers — Ollama, OpenAI, OpenRouter, HuggingFace, Google, FastEmbed
• Qdrant Vector Store — dense, sparse, and hybrid search
• Advanced Retrieval — similarity, MMR, and hybrid search with metadata filtering
• SHA256 Deduplication — at both file and chunk level
• Directory Ingestion — ingest an entire folder with one call, with optional recursive scan
• Env Var Substitution — use ${VAR} in config.yaml for secrets

Architecture

Installation

pip install ragwire

# With Ollama support (local, no API key)
pip install "ragwire[ollama]"

# With OpenRouter support (LLM + embeddings; requires Python >= 3.10)
pip install "ragwire[openrouter]"

# With all providers
pip install "ragwire[all]"

Quick Start

from ragwire import RAGWire

rag = RAGWire("config.yaml")

# Ingest files — SHA256 deduplication, safe to re-run
stats = rag.ingest_documents(["data/Apple_10k_2025.pdf", "data/Microsoft_10k_2025.pdf"])
print(f"Processed: {stats['processed']}, Skipped: {stats['skipped']}, Chunks: {stats['chunks_created']}")

# Or ingest an entire directory
stats = rag.ingest_directory("data/", recursive=True)

# Basic retrieval — returns list of LangChain Document objects
results = rag.retrieve("What is the total revenue?", top_k=5)
for doc in results:
    print(doc.page_content[:300])
    print(doc.metadata["company_name"])   # str, lowercased — e.g. "apple"
    print(doc.metadata["fiscal_year"])    # list[int] — e.g. [2025]  ← NOT a plain int
    print(doc.metadata["file_name"])      # str — e.g. "Apple_10k_2025.pdf"

# Retrieval with explicit metadata filters
results = rag.retrieve(
    "What is the net income?",
    filters={"company_name": "apple", "fiscal_year": 2025}  # pass year as int
)

# OR logic within a field — matches any of the listed values
results = rag.retrieve("Compare revenue trends", filters={"fiscal_year": [2023, 2024, 2025]})

# Agent-controlled filtering (recommended for AI agents)
filters = rag.extract_filters("Apple's revenue in 2025")
# → {"company_name": "apple", "fiscal_year": 2025} or None
results = rag.retrieve("Apple's revenue in 2025", filters=filters)

Configuration

Copy config.example.yaml to config.yaml and edit. Secrets can be injected via environment variables:

vectorstore:
  url: "https://your-cluster.qdrant.io"
  api_key: "${QDRANT_API_KEY}"

llm:
  provider: "openai"
  model: "gpt-5.4-nano"
  api_key: "${OPENAI_API_KEY}"

Full example:

embeddings:
  provider: "ollama"
  model: "qwen3-embedding:0.6b"
  base_url: "http://localhost:11434"

llm:
  provider: "ollama"
  model: "qwen3.5:9b"
  num_ctx: 16384

vectorstore:
  url: "http://localhost:6333"
  collection_name: "my_docs"
  use_sparse: true

retriever:
  search_type: "hybrid"
  top_k: 5
  auto_filter: false   # set true to enable LLM-based filter extraction from every query

Embedding Providers

# Ollama (local)
embeddings:
  provider: "ollama"
  model: "qwen3-embedding:0.6b"

# OpenAI
embeddings:
  provider: "openai"
  model: "text-embedding-3-small"

# OpenRouter (free-tier models available)
embeddings:
  provider: "openrouter"
  model: "nvidia/llama-nemotron-embed-vl-1b-v2:free"
  api_key: "${OPENROUTER_API_KEY}"

# HuggingFace (local)
embeddings:
  provider: "huggingface"
  model_name: "sentence-transformers/all-MiniLM-L6-v2"

# Google
embeddings:
  provider: "google"
  model: "models/embedding-001"

Component Usage

from ragwire import (
    MarkItDownLoader,
    get_splitter,
    get_markdown_splitter,
    get_embedding,
    QdrantStore,
    MetadataExtractor,
    hybrid_search,
    mmr_search,
)

# Load a document
loader = MarkItDownLoader()
result = loader.load("document.pdf")

# Split text
splitter = get_markdown_splitter(chunk_size=10000, chunk_overlap=2000)
chunks = splitter.split_text(result["text_content"])

# Embeddings
embedding = get_embedding({"provider": "ollama", "model": "qwen3-embedding:0.6b"})

# Vector store
store = QdrantStore(config={"url": "http://localhost:6333"}, embedding=embedding)
store.set_collection("my_collection")
vectorstore = store.get_store()

Architecture

ragwire/
├── core/          # Config loader + RAGWire orchestrator
├── loaders/       # MarkItDown document converter
├── processing/    # Text splitters + SHA256 hashing
├── metadata/      # Pydantic schema + LLM extractor
├── embeddings/    # Multi-provider embedding factory
├── vectorstores/  # Qdrant wrapper with hybrid search
├── retriever/     # Similarity, MMR, hybrid retrieval
└── utils/         # Logging

Troubleshooting

Error	Fix
Qdrant connection refused	docker run -p 6333:6333 qdrant/qdrant
markitdown[pdf] missing	pip install "markitdown[pdf]"
Ollama model not found	ollama pull <model-name>
fastembed missing	pip install fastembed (needed for hybrid search)
Embedding dimension mismatch	Set force_recreate: true in config once, then back to false

License

MIT © 2026 KGP Talkie Private Limited

Links

   

• 🌐 Website: kgptalkie.com
• 📖 Docs: laxmimerit.github.io/RAGWire
• 💻 GitHub: github.com/laxmimerit/ragwire
• 📧 Email: [e-mail]