qdrant-loader 1.0.1

A tool for collecting and vectorizing technical content from multiple sources and storing it in a QDrant vector database.

QDrant Loader

A powerful data ingestion engine that collects and vectorizes technical content from multiple sources for storage in QDrant vector database. Part of the QDrant Loader monorepo ecosystem.

For full setup and configuration, start with the documentation links below.

🚀 What It Does

• Collects content from Git repositories, Confluence, JIRA, documentation sites, and local files
• Converts files automatically from 20+ formats including PDF, Office docs, and images
• Processes intelligently with smart chunking, metadata extraction, and change detection
• Stores efficiently in QDrant vector database with optimized embeddings
• Updates incrementally to keep your knowledge base current

🗄️ Supported Data Sources

Source	Description	Key Features
Git	Code repositories and documentation	Branch selection, file filtering, commit metadata
Confluence	Cloud & Data Center/Server	Space filtering, hierarchy preservation, attachment processing
JIRA	Cloud & Data Center/Server	Project filtering, issue tracking, attachment support
Public Docs	External documentation sites	CSS selector extraction, version detection
Local Files	Local directories and files	Glob patterns, recursive scanning, file type filtering

📄 File Conversion Support

Automatically converts diverse file formats using Microsoft's MarkItDown:

Supported Formats

• Documents: PDF, Word (.docx), PowerPoint (.pptx), Excel (.xlsx)
• Images: PNG, JPEG, GIF, BMP, TIFF (with optional OCR)
• Archives: ZIP files with automatic extraction
• Data: JSON, CSV, XML, YAML
• Audio: MP3, WAV (transcription support)
• E-books: EPUB format
• And more: 20+ file types supported

For detailed source setup and conversion behavior, see:

• Data source guides - Source-specific setup for Git, Confluence, Jira, local files, and public docs.
• File conversion guide - Supported formats, conversion behavior, and practical tuning options.

Key Features

• Automatic detection: Files are converted when enable_file_conversion: true
• Attachment processing: Downloads and converts attachments from all sources
• Fallback handling: Graceful handling when conversion fails
• Metadata preservation: Original file information is maintained
• Performance optimized: Configurable limits for size, timeouts, and throughput

🏗️ Architecture

Core Components

• Source Connectors: Pluggable connectors for different data sources
• File Processors: Conversion and processing pipeline for various file types
• Chunking Engine: Intelligent text segmentation with configurable overlap
• Embedding Service: Flexible embedding generation with multiple providers
• State Manager: SQLite-based tracking for incremental updates
• QDrant Client: Optimized vector storage and retrieval

Data Flow

Data Sources → File Conversion → Text Processing → Chunking → Embedding → QDrant Storage
	↓              ↓               ↓            ↓          ↓           ↓
Git Repos      PDF/Office      Preprocessing   Smart     OpenAI      Vector DB
Confluence     Images/Audio    Metadata        Chunks    Local       Collections
JIRA           Archives        Extraction      Overlap   Custom      Incremental
Public Docs    Documents       Filtering       Context   Providers   Updates
Local Files    20+ Formats     Cleaning        Tokens    Endpoints   State Tracking

🔍 Advanced Features

Incremental Updates

• Change detection for all source types
• Efficient synchronization with minimal reprocessing
• State persistence across runs
• Conflict resolution for concurrent updates

Performance Optimization

• Batch processing for efficient embedding generation
• Rate limiting to respect API limits
• Parallel processing for multiple sources
• Memory management for large datasets

Error Handling

• Robust retry mechanisms for transient failures
• Graceful degradation when sources are unavailable
• Detailed logging for troubleshooting
• Recovery strategies for partial failures

Implementation details for tuning and troubleshooting are covered in:

• Configuration reference - Full settings model, defaults, and production-ready examples.
• Common workflows - Proven end-to-end paths for ingestion, maintenance, and operations.
• Troubleshooting guide - Common failure patterns and step-by-step fixes.

📦 Installation

pip install qdrant-loader

For detailed installation instructions, see:

• Installation details - Platform-specific install methods and dependency requirements.

⚙️ Configuration

For detailed configuration setup, see:

• Basic Configuration - Getting started with configuration
• Configuration reference - Configuration model, options, and practical examples.
• Data source guides - Source-specific setup for Git, Confluence, Jira, local files, and more.
• Environment Variables - Environment variable reference and naming conventions.
• LLM Provider Guide - Configure provider-specific LLM details

🧪 CLI

qdrant-loader --help

⚡ Quick Start

# Initialize collection and metadata structures
qdrant-loader init --workspace .

# Ingest from configured projects/sources
qdrant-loader ingest --workspace .

# Check workspace status
qdrant-loader project --workspace . status

For full workspace bootstrapping (.env, config.yaml, and source templates), see Quick start.

📚 Documentation

• Getting Started - Quick start and core concepts
• Monorepo overview - Project structure, packages, and top-level navigation across the repository.
• Quick start - Fast setup path from install to first successful ingestion.
• User Guides - Detailed usage instructions
• Developer hub - Developer guides for architecture, testing, deployment, and contribution workflows.

🆘 Support

• Issues - Bug reports and feature requests
• Discussions - Community Q&A

🤝 Contributing

See CONTRIBUTING - Contribution guidelines, development standards, and pull request process.

📄 License

This project is licensed under the GNU GPLv3 - see the LICENSE file for details.

---

Ready to get started? Check out our Quick Start Guide or browse the complete documentation.