RAG (Retrieval-Augmented Generation) System

A Python-based RAG system that processes text files, generates embeddings, and stores them in a Postgres database with pgvector for efficient similarity search.

For detailed information, see:

• Design Document - System architecture and requirements
• Developer Guide - Detailed setup and development instructions

Features

• File ingestion (source code, Markdown, plain text)
• Text chunking with configurable overlap
• Vector embeddings via OpenAI or Hugging Face
• Postgres + pgvector for vector storage and search
• Project-based organization of documents
• Comprehensive Taskfile for development workflows
• Dockerized Postgres database with pgvector

Getting Started

Prerequisites

• Python 3.10+
• Docker
• Poetry (will be installed automatically by the setup script)

Poetry Management

This project uses Poetry for dependency management. Key Poetry commands are wrapped in Taskfile tasks:

• Install dependencies:

task install

• Set up development environment:

task setup-dev

• Update dependencies:

task update-deps

• Export requirements files:

task export-reqs

• Check dependency status:

task verify-deps

The project includes both runtime and development dependencies specified in pyproject.toml.

Installation

1. Clone the repository:

git clone https://github.com/SpillwaveSolutions/vector-rag
cd vector-rag

2. Set up the development environment:

task setup-dev

3. Configure environment variables:

cp environment/.env.example .env
# Edit .env with your settings:
# - Database credentials
# - OpenAI API key (if using OpenAI embeddings)

Running the System

Start the Database

The system uses a Dockerized Postgres database with pgvector:

task db:up

Run Examples

• With mock embeddings (no API key required):

task demo:mock

• With OpenAI embeddings (requires API key in .env):

task demo:openai

Interactive Database Access

To access the database directly:

task psql

Testing

The project includes comprehensive tests:

• Run all tests:

task test:all

• Run integration tests:

task test:integration

• Run tests with coverage report:

task test:coverage

• Run a specific test:

task test:single -- tests/path/to/test_file.py::test_name

Development Workflow

Code Formatting and Linting

task format  # Runs black and isort
task typecheck  # Runs mypy
task lint  # Runs all code quality checks

Dependency Management

• Update dependencies:

task update-deps

• Export requirements files:

task export-reqs

Database Management

• Recreate the database from scratch:

task db:recreate

• Stop the database:

task db:down

Configuration

The system is configured through environment variables in .env. Key settings include:

• DB_*: Database connection settings
• OPENAI_API_KEY: Required for OpenAI embeddings
• LOCAL_EMBEDDING: Set to true to use local SentenceTransformers
• EMBEDDINGS_DIM: Vector dimension (384 for local, 1536 for OpenAI)
• CHUNK_SIZE/CHUNK_OVERLAP: Text chunking parameters

License

MIT License

Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

Please ensure all tests pass and code is properly formatted before submitting PRs.