qdrant-loader 0.8.1

A tool for collecting and vectorizing technical content from multiple sources and storing it in a QDrant vector database.

QDrant Loader

A powerful data ingestion engine that collects and vectorizes technical content from multiple sources for storage in QDrant vector database. Part of the QDrant Loader monorepo ecosystem.

🚀 What It Does

QDrant Loader is the data ingestion engine that:

• Collects content from Git repositories, Confluence, JIRA, documentation sites, and local files
• Converts files automatically from 20+ formats including PDF, Office docs, and images
• Processes intelligently with smart chunking, metadata extraction, and change detection
• Stores efficiently in QDrant vector database with optimized embeddings
• Updates incrementally to keep your knowledge base current

🔄 Supported Data Sources

Source	Description	Key Features
Git	Code repositories and documentation	Branch selection, file filtering, commit metadata
Confluence	Cloud & Data Center/Server	Space filtering, hierarchy preservation, attachment processing
JIRA	Cloud & Data Center/Server	Project filtering, issue tracking, attachment support
Public Docs	External documentation sites	CSS selector extraction, version detection
Local Files	Local directories and files	Glob patterns, recursive scanning, file type filtering

📄 File Conversion Support

Automatically converts diverse file formats using Microsoft's MarkItDown:

Supported Formats

• Documents: PDF, Word (.docx), PowerPoint (.pptx), Excel (.xlsx)
• Images: PNG, JPEG, GIF, BMP, TIFF (with optional OCR)
• Archives: ZIP files with automatic extraction
• Data: JSON, CSV, XML, YAML
• Audio: MP3, WAV (transcription support)
• E-books: EPUB format
• And more: 20+ file types supported

Key Features

• Automatic detection: Files are converted when enable_file_conversion: true
• Attachment processing: Downloads and converts attachments from all sources
• Fallback handling: Graceful handling when conversion fails
• Metadata preservation: Original file information maintained
• Performance optimized: Configurable size limits and timeouts

📦 Installation

From PyPI (Recommended)

pip install qdrant-loader

From Source (Development)

# Clone the monorepo
git clone https://github.com/martin-papy/qdrant-loader.git
cd qdrant-loader

# Install in development mode
pip install -e packages/qdrant-loader

With MCP Server

For complete AI integration:

# Install both packages
pip install qdrant-loader qdrant-loader-mcp-server

⚡ Quick Start

1. Workspace Setup (Recommended)

# Create workspace directory
mkdir my-qdrant-workspace && cd my-qdrant-workspace

# Download configuration templates
curl -o config.yaml https://raw.githubusercontent.com/martin-papy/qdrant-loader/main/packages/qdrant-loader/conf/config.template.yaml
curl -o .env https://raw.githubusercontent.com/martin-papy/qdrant-loader/main/packages/qdrant-loader/conf/.env.template

2. Environment Configuration

Edit .env file:

# QDrant Configuration
QDRANT_URL=http://localhost:6333
QDRANT_COLLECTION_NAME=my_docs
QDRANT_API_KEY=your_api_key  # Required for QDrant Cloud

# LLM Configuration (new unified approach)
LLM_PROVIDER=openai
LLM_BASE_URL=https://api.openai.com/v1
LLM_API_KEY=your_openai_key
LLM_EMBEDDING_MODEL=text-embedding-3-small
LLM_CHAT_MODEL=gpt-4o-mini

# Legacy (still supported)
OPENAI_API_KEY=your_openai_key

# State Management
STATE_DB_PATH=./state.db

3. Data Source Configuration

Edit config.yaml:

# Global configuration
global_config:
  chunking:
    chunk_size: 1500
    chunk_overlap: 200
  
  llm:
    provider: "openai"
    base_url: "https://api.openai.com/v1"
    api_key: "${LLM_API_KEY}"
    models:
      embeddings: "text-embedding-3-small"
      chat: "gpt-4o-mini"
    request:
      batch_size: 100
    embeddings:
      vector_size: 1536
  
  file_conversion:
    max_file_size: 52428800  # 50MB
    conversion_timeout: 300
    markitdown:
      enable_llm_descriptions: false

# Multi-project configuration
projects:
  my-project:
    project_id: "my-project"
    display_name: "My Documentation Project"
    description: "Project description"
    
    sources:
      git:
        my-repo:
          base_url: "https://github.com/your-org/your-repo.git"
          branch: "main"
          include_paths:
            - "**/*.md"
            - "**/*.py"
          exclude_paths:
            - "**/node_modules/**"
          token: "${REPO_TOKEN}"
          enable_file_conversion: true

      localfile:
        local-docs:
          base_url: "file://./docs"
          include_paths:
            - "**/*.md"
            - "**/*.pdf"
          enable_file_conversion: true

4. Load Your Data

# Initialize QDrant collection
qdrant-loader init --workspace .

# Load data from configured sources
qdrant-loader ingest --workspace .

# Check project status
qdrant-loader project --workspace . status

🔧 Configuration

Environment Variables

Variable	Description	Default	Required
QDRANT_URL	QDrant instance URL	http://localhost:6333	Yes
QDRANT_API_KEY	QDrant API key	None	Cloud only
QDRANT_COLLECTION_NAME	Collection name	documents	Yes
LLM_API_KEY	LLM API key (unified)	None	Yes
OPENAI_API_KEY	OpenAI API key (legacy)	None	Legacy
STATE_DB_PATH	State database path	./state.db	Yes

Source-Specific Variables

Git Repositories

REPO_TOKEN=your_github_token

Confluence (Cloud)

CONFLUENCE_URL=https://your-domain.atlassian.net/wiki
CONFLUENCE_SPACE_KEY=SPACE
CONFLUENCE_TOKEN=your_token
CONFLUENCE_EMAIL=your_email

Confluence (Data Center/Server)

CONFLUENCE_URL=https://your-confluence-server.com
CONFLUENCE_SPACE_KEY=SPACE
CONFLUENCE_PAT=your_personal_access_token

JIRA (Cloud)

JIRA_URL=https://your-domain.atlassian.net
JIRA_PROJECT_KEY=PROJ
JIRA_TOKEN=your_token
JIRA_EMAIL=your_email

JIRA (Data Center/Server)

JIRA_URL=https://your-jira-server.com
JIRA_PROJECT_KEY=PROJ
JIRA_PAT=your_personal_access_token

🎯 Usage Examples

Basic Commands

# Show current configuration
qdrant-loader config --workspace .

# Initialize collection (one-time setup)
qdrant-loader init --workspace .

# Ingest data from all configured sources
qdrant-loader ingest --workspace .

# Check project status
qdrant-loader project status --workspace .

# List all projects
qdrant-loader project list --workspace .

# Show help
qdrant-loader --help

Advanced Usage

# Specify configuration files individually
qdrant-loader --config config.yaml --env .env ingest

# Debug logging
qdrant-loader ingest --workspace . --log-level DEBUG

# Force full re-ingestion
qdrant-loader init --workspace . --force
qdrant-loader ingest --workspace .

# Process specific project
qdrant-loader ingest --workspace . --project my-project

# Process specific source type
qdrant-loader ingest --workspace . --source-type git

# Enable performance profiling
qdrant-loader ingest --workspace . --profile

Project Management

# Validate project configurations
qdrant-loader project validate --workspace .

# Validate specific project
qdrant-loader project validate --workspace . --project-id my-project

# Show project status in JSON format
qdrant-loader project status --workspace . --format json

# Show specific project status
qdrant-loader project status --workspace . --project-id my-project

🏗️ Architecture

Core Components

• Source Connectors: Pluggable connectors for different data sources
• File Processors: Conversion and processing pipeline for various file types
• Chunking Engine: Intelligent text segmentation with configurable overlap
• Embedding Service: Flexible embedding generation with multiple providers
• State Manager: SQLite-based tracking for incremental updates
• QDrant Client: Optimized vector storage and retrieval

Data Flow

Data Sources → File Conversion → Text Processing → Chunking → Embedding → QDrant Storage
     ↓              ↓               ↓            ↓          ↓           ↓
Git Repos      PDF/Office      Preprocessing   Smart     OpenAI      Vector DB
Confluence     Images/Audio    Metadata        Chunks    Local       Collections
JIRA           Archives        Extraction      Overlap   Custom      Incremental
Public Docs    Documents       Filtering       Context   Providers   Updates
Local Files    20+ Formats     Cleaning        Tokens    Endpoints   State Tracking

🔍 Advanced Features

Incremental Updates

• Change detection for all source types
• Efficient synchronization with minimal reprocessing
• State persistence across runs
• Conflict resolution for concurrent updates

Performance Optimization

• Batch processing for efficient embedding generation
• Rate limiting to respect API limits
• Parallel processing for multiple sources
• Memory management for large datasets

Error Handling

• Robust retry mechanisms for transient failures
• Graceful degradation when sources are unavailable
• Detailed logging for troubleshooting
• Recovery strategies for partial failures

🧪 Testing

# Run all tests
pytest packages/qdrant-loader/tests/

# Run with coverage
pytest --cov=qdrant_loader packages/qdrant-loader/tests/

# Run specific test categories
pytest -m "unit" packages/qdrant-loader/tests/
pytest -m "integration" packages/qdrant-loader/tests/

🤝 Contributing

This package is part of the QDrant Loader monorepo. See the main contributing guide for details.

Development Setup

# Clone and setup
git clone https://github.com/martin-papy/qdrant-loader.git
cd qdrant-loader

# Install in development mode
pip install -e "packages/qdrant-loader[dev]"

# Run tests
pytest packages/qdrant-loader/tests/

📚 Documentation

• Complete Documentation - Comprehensive guides and references
• Getting Started - Quick start and core concepts
• User Guides - Detailed usage instructions
• Developer Docs - Architecture and API reference

🆘 Support

• Issues - Bug reports and feature requests
• Discussions - Community Q&A
• Documentation - Comprehensive guides

📄 License

This project is licensed under the GNU GPLv3 - see the LICENSE file for details.

---

Ready to load your data? Check out the Quick Start Guide or explore the complete documentation.