qdrant-loader 1.0.0

A tool for collecting and vectorizing technical content from multiple sources and storing it in a QDrant vector database.

QDrant Loader

A powerful data ingestion engine that collects and vectorizes technical content from multiple sources for storage in QDrant vector database. Part of the QDrant Loader monorepo ecosystem.

For full setup and configuration, start with the documentation links below.

🚀 What It Does

• Collects content from Git repositories, Confluence, JIRA, public documentation sites, and local files
• Converts supported file types (via MarkItDown) when enabled per source
• Chunks, embeds, and stores content in QDrant for semantic retrieval
• Supports incremental ingestion workflows through the CLI

For detailed source setup and conversion behavior, see:

• Data source guides - Source-specific setup for Git, Confluence, Jira, local files, and public docs.
• File conversion guide - Supported formats, conversion behavior, and practical tuning options.

📄 File Conversion Support

Automatically converts diverse file formats using Microsoft's MarkItDown:

Supported Formats

• Documents: PDF, Word (.docx), PowerPoint (.pptx), Excel (.xlsx)
• Images: PNG, JPEG, GIF, BMP, TIFF (with optional OCR)
• Archives: ZIP files with automatic extraction
• Data: JSON, CSV, XML, YAML
• Audio: MP3, WAV (transcription support)
• E-books: EPUB format
• And more: 20+ file types supported

Key Features

• Automatic detection: Files are converted when enable_file_conversion: true
• Attachment processing: Downloads and converts attachments from all sources
• Fallback handling: Graceful handling when conversion fails
• Metadata preservation: Original file information is maintained
• Performance optimized: Configurable limits for size, timeouts, and throughput

🏗️ Data Flow

Data Sources → File Conversion → Text Processing → Chunking → Embedding → QDrant Storage
	↓              ↓               ↓            ↓          ↓           ↓
Git Repos      PDF/Office      Preprocessing   Smart     OpenAI      Vector DB
Confluence     Images/Audio    Metadata        Chunks    Local       Collections
JIRA           Archives        Extraction      Overlap   Custom      Incremental
Public Docs    Documents       Filtering       Context   Providers   Updates
Local Files    20+ Formats     Cleaning        Tokens    Endpoints   State Tracking

🔍 Advanced Features

Incremental Updates

• Change detection for all source types
• Efficient synchronization with minimal reprocessing
• State persistence across runs
• Conflict resolution for concurrent updates

Performance Optimization

• Batch processing for efficient embedding generation
• Rate limiting to respect API limits
• Parallel processing for multiple sources
• Memory management for large datasets

Error Handling

• Robust retry mechanisms for transient failures
• Graceful degradation when sources are unavailable
• Detailed logging for troubleshooting
• Recovery strategies for partial failures

Implementation details for tuning and troubleshooting are covered in:

• Configuration reference - Full settings model, defaults, and production-ready examples.
• Common workflows - Proven end-to-end paths for ingestion, maintenance, and operations.
• Troubleshooting guide - Common failure patterns and step-by-step fixes.

📦 Installation

pip install qdrant-loader

🧪 CLI

qdrant-loader --help

⚡ Quick Start

# Initialize collection and metadata structures
qdrant-loader init --workspace .

# Ingest from configured projects/sources
qdrant-loader ingest --workspace .

# Check workspace status
qdrant-loader project --workspace . status

For full workspace bootstrapping (.env, config.yaml, and source templates), see Quick start.

📚 Canonical Documentation

• Monorepo overview - Project structure, packages, and top-level navigation across the repository.
• Quick start - Fast setup path from install to first successful ingestion.
• Installation details - Platform-specific install methods and dependency requirements.
• Configuration reference - Configuration model, options, and practical examples.
• Data source guides - Source-specific setup for Git, Confluence, Jira, local files, and more.

🤝 Contributing

See CONTRIBUTING - Contribution guidelines, development standards, and pull request process.