easy-rag-pipeline 0.1.3

A reusable and configurable RAG (Retrieval-Augmented Generation) pipeline.

easy-rag-pipeline

A simple, flexible, and production-ready Retrieval-Augmented Generation (RAG) pipeline for Python. Supports OpenAI, Groq, OpenRouter, Gemini, and HuggingFace models for both LLM and embeddings. Built for easy integration and rapid prototyping.

Maintainer & Author: Engr. Hamza

---

Features

• Plug-and-play RAG pipeline for text, PDF, and web sources
• Supports OpenAI, Groq, OpenRouter, Gemini, and HuggingFace
• FAISS vector store for fast retrieval
• Easy configuration via YAML
• High-level utility functions for quick usage
• Modular: index once, query many times

---

Installation

pip install easy-rag-pipeline

---

Quickstart Example

1. Basic RAG from a file

from easy_rag_pipeline.api import rag_from_file

answer = rag_from_file(
              query="What is Retrieval-Augmented Generation (RAG)?",
              file_path="examples/sample_document.txt",
              config_path="examples/config.yaml"
)
print(answer)

2. RAG from a website

from easy_rag_pipeline.api import rag_from_url

answer = rag_from_url(
              query="What is RAG?",
              url="https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
              config_path="examples/config.yaml"
)
print(answer)

3. RAG from a string of text

from easy_rag_pipeline.api import rag_from_text

answer = rag_from_text(
              query="What is RAG?",
              text="Retrieval-Augmented Generation (RAG) is a technique...",
              config_path="examples/config.yaml"
)
print(answer)

4. Index a file and save the vector store

from easy_rag_pipeline.api import index_file

index_file(
              file_path="examples/sample_document.txt",
              config_path="examples/config.yaml",
              save_path="vector_store"
)

5. Load a saved index and query it

from easy_rag_pipeline.api import load_index_and_query

answer = load_index_and_query(
              query="What is RAG?",
              index_path="vector_store",
              config_path="examples/config.yaml"
)
print(answer)

---

Configuration

Edit examples/config.yaml to set your LLM, embedding, and vector store providers. Example:

llm:
       provider: "groq"  # or "openai", "openrouter", "gemini"
       model: "llama3-8b-8192"
       temperature: 0.7
embedding:
       provider: "huggingface"  # or "openai", "groq", "openrouter"
       model: "sentence-transformers/all-MiniLM-L6-v2"
vector_store:
       provider: "faiss"
chunking:
       chunk_size: 1000
       chunk_overlap: 100
retrieval:
       k: 5

---

API Reference

High-level utility functions (in easy_rag_pipeline.api):

• rag_from_file(query, file_path, config_path)
• rag_from_url(query, url, config_path)
• rag_from_text(query, text, config_path)
• index_file(file_path, config_path, save_path)
• load_index_and_query(query, index_path, config_path)

Advanced usage

You can use the lower-level pipeline functions for more control:

from easy_rag_pipeline.pipeline import create_and_persist_vector_store, query_rag_pipeline, simple_rag_pipeline

---

Supported Providers

• OpenAI: GPT-3.5, GPT-4, text-embedding-ada-002, etc.
• Groq: Llama3, Mixtral, and more
• OpenRouter: Claude, Llama, and more (OpenAI-compatible API)
• Gemini: Google Gemini models
• HuggingFace: Local and hosted embedding models

---

Setting API Keys

You can provide API keys in two ways:

1. Using a .env file (recommended)

Create a .env file in your project root:

OPENAI_API_KEY=sk-...
GROQ_API_KEY=gsk-...
GOOGLE_API_KEY=...  # for Gemini
OPENROUTER_API_KEY=...  # for OpenRouter

The pipeline will automatically load these using python-dotenv if installed.

2. Setting API keys in your code

You can set the API key directly in your config dictionary before running the pipeline:

config = load_config("examples/config.yaml")
config['llm']['api_key'] = "sk-..."  # or your GROQ/OPENROUTER key
config['embedding']['api_key'] = "sk-..."  # if needed for embeddings
answer = rag_from_file(
       query="What is RAG?",
       file_path="examples/sample_document.txt",
       config_path="examples/config.yaml"
)

Or set the environment variable in your script:

import os
os.environ["OPENAI_API_KEY"] = "sk-..."
os.environ["GROQ_API_KEY"] = "gsk-..."

This will be picked up automatically by the pipeline.

---

License

MIT

---

Maintainer

Engr. Hamza

A flexible, configurable, and easy-to-use Retrieval-Augmented Generation (RAG) pipeline in a box.

set PYTHONUTF8=1

---

Easy RAG Pipeline is a production-grade framework for building and deploying RAG applications. It handles the entire workflow from document ingestion to answer generation, allowing you to plug in different LLMs, embedding models, and vector stores with a simple, configuration-driven setup.

✨ Key Features

• 🔌 Pluggable Architecture: Easily switch between LLMs (OpenAI, Groq, Gemini, OpenRouter), embedding models (OpenAI, HuggingFace), and vector stores (FAISS).
• ⚙️ Configuration-Driven: No hard-coding. Manage all your settings—from chunk sizes to model names—through a single config.yaml file.
• 🚀 Efficient & Practical: Includes two pipeline modes: a simple all-in-one for quick demos, and an advanced two-step process (index then query) for production efficiency.
• 📚 Multi-Format Ingestion: Out-of-the-box support for loading documents from PDFs, text files, and websites.
• 📦 Ready-to-Use: Comes with a CLI example and an interactive Streamlit demo to get you started in minutes.
• 🔧 Extensible by Design: Clean, modular code that's easy to extend with your own custom components.

🏗️ Architecture

The pipeline follows a standard, modular RAG architecture that is easy to understand and build upon.

[Source Document: PDF, TXT, URL]
             |
             v
      [1. Ingest & Load]
             |
             v
      [2. Chunk Documents]
             |
             v
      [3. Generate Embeddings]  <-- (Pluggable: OpenAI, HuggingFace)
             |
             v
      [4. Store in Vector DB]   <-- (Pluggable: FAISS)
             |
             +-----------------------+
             |                       |
             v                       v
[User Query] -> [5. Retrieve Docs]  [Vector Database]
             |
             v
[Retrieved Context + Query]
             |
             v
      [6. Generate Answer]      <-- (Pluggable: OpenAI, Groq, Gemini)
             |
             v
         [Answer]

🚀 Getting Started

python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate

Install all required dependencies.

```bash
pip install -r requirements.txt

Finally, install the package in editable mode, which allows you to modify the code and see changes instantly.

pip install -e .

2. Configuration

This project is managed by a central configuration file and environment variables for your secrets.

a. Set up API Keys:

Copy the example environment file and add your secret API keys. The pipeline will automatically load the correct key based on the provider you choose in config.yaml.

cp .env.example .env

Now, edit .env and add your keys:

# For OpenAI
OPENAI_API_KEY="sk-..."

# For Groq
GROQ_API_KEY="gsk_..."

# For Google Gemini
GOOGLE_API_KEY="AIzaSy..."

# For OpenRouter
OPENROUTER_API_KEY="sk-or-..."

b. Select your Components:

Open examples/config.yaml to select the models and components you want to use. For example, to switch to Groq's Llama3 model:

# examples/config.yaml

llm:
  provider: "groq"
  model: "llama3-8b-8192"
  temperature: 0.7

3. Run the Demo

You can test the pipeline with either the basic CLI example or the interactive Streamlit app.

a. Basic CLI Example:

This script will run the simple, all-in-one pipeline on the included sample document.

python examples/basic_rag.py

b. Interactive Streamlit Demo:

For a more visual experience, launch the Streamlit app.

streamlit run examples/streamlit_demo.py

This will open the demo in your web browser.

⚙️ Advanced Usage

For production scenarios, re-indexing your documents on every query is inefficient. The library provides a two-step process for this:

1. Index Your Data: Run a script to process and store your documents in a persistent vector store.
2. Query the Store: Run your application to load the pre-indexed store and query it repeatedly.

Click to see an example of the advanced workflow

# script_to_index.py
from easy_rag_pipeline import create_and_persist_vector_store, load_config

# Load config
config = load_config("examples/config.yaml")

# Create and save a FAISS vector store from a PDF
create_and_persist_vector_store(
    source_path="path/to/my_document.pdf",
    source_type="pdf",
    config=config,
    save_path="my_vector_store"
)

# --------------------------------------------------

# script_to_query.py
from easy_rag_pipeline import query_rag_pipeline, load_config
from langchain_community.vectorstores import FAISS
from easy_rag_pipeline.embed import get_embedding_function

# Load config and embedding function
config = load_config("examples/config.yaml")
embedding_function = get_embedding_function(config['embedding'])

# Load the persisted vector store
vector_store = FAISS.load_local("my_vector_store", embedding_function, allow_dangerous_deserialization=True)

# Query the pipeline
query = "What was the main finding of the document?"
answer = query_rag_pipeline(query, vector_store, config)
print(answer)

🔧 Extensibility

The library is designed to be easily extended.

• To add a new LLM:
  1. Install the required package (e.g., pip install langchain-anthropic).
  2. Add an elif provider == "anthropic": block in easy_rag_pipeline/generate.py.
  3. Update config.py to load the ANTHROPIC_API_KEY.
• To add a new Document Loader:
  1. Add the loader function (e.g., load_docx) in easy_rag_pipeline/ingest.py.
  2. Update the pipeline.py functions to accept the new source_type.

🗺️ Roadmap

This project is under active development. Future enhancements include:

• Support for more vector databases (Chroma, Pinecone, Weaviate).
• Asynchronous pipeline for improved performance.
• Hybrid retrieval combining keyword search (BM25) and semantic search.
• Support for image and multi-modal RAG.
• Integration with RAG evaluation frameworks.

🤝 Contributing

Contributions are welcome! Please feel free to submit a pull request or open an issue to discuss your ideas.

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.