vector-db-query 1.0.0

CLI application for vector database queries using LLMs via MCP

Vector DB Query

🚀 Semantic Search for Your Documents with AI Integration

A powerful CLI tool that indexes your documents and enables natural language search with LLM integration via MCP

🌟 Key Features

Vector DB Query is a comprehensive solution for building searchable knowledge bases from your documents:

📄 Enhanced Document Processing

• 40+ File Formats: PDF, Word, Excel, PowerPoint, HTML, Markdown, JSON, XML, Images (with OCR), and more
• OCR Support: Extract text from images (PNG, JPG, TIFF, BMP) with configurable languages and confidence thresholds
• Archive Support: Process ZIP, TAR, and compressed archives automatically
• Smart Chunking: Multiple strategies including sliding window, semantic, and paragraph-based
• Metadata Extraction: Preserve document structure, authorship, dates, and custom tags
• Format-Specific Processing: Tailored extraction for each file type (formulas from Excel, speaker notes from PowerPoint, etc.)

🔍 Advanced Semantic Search

• Natural language queries with vector similarity
• Hybrid search combining keyword and semantic matching
• Advanced filtering by file type, date, score, and metadata
• Result reranking and highlighting
• Export results in multiple formats (JSON, CSV, Markdown)

🎨 Rich Interactive CLI

• Beautiful terminal UI powered by Rich and Textual
• Visual file browser with real-time preview
• Interactive query builder with autocomplete
• Live progress tracking with detailed statistics
• Customizable themes and output formats

🤖 AI Integration

• MCP server for Claude and other AI assistants
• Secure API with JWT authentication
• Rate limiting and request monitoring
• Standardized tool interface for document operations
• Real-time processing feedback

⚙️ Flexible Configuration

• YAML-based configuration with environment overrides
• CLI commands for configuration management
• Support for multiple configuration profiles
• Validation and health checks
• Hot-reloading of settings

📊 Monitoring & Management

• Real-time monitoring dashboard (Streamlit)
• System metrics and resource usage tracking
• Processing queue management
• Log aggregation and analysis
• PM2 integration for process management

⚡ Performance & Scalability

• Parallel processing with configurable workers
• Memory-efficient chunking and streaming
• Smart caching system
• Connection pooling for database operations
• Batch processing optimization

🔗 Data Source Integration (New!)

• Gmail Integration: Sync emails via IMAP/OAuth2 with folder selection and filtering
• Fireflies.ai Integration: Automatic meeting transcript sync via API and webhooks
• Google Drive Integration: Search and sync Gemini transcripts and documents
• Smart Deduplication: Cross-source duplicate detection using content hashing
• NLP Processing: Entity extraction, sentiment analysis, and key phrase detection
• Selective Processing: Configurable filters for targeted content processing
• Real-time Monitoring: Dashboard integration for tracking sync status
• Setup Wizard: Interactive configuration for easy onboarding

📋 Requirements

• Python 3.9 or higher
• 4GB RAM minimum (8GB recommended)
• Qdrant vector database (local or cloud)
• API key for embeddings (Google, OpenAI, etc.)
• Optional: Tesseract for OCR support
• Optional: Docker for containerized deployment

🚀 Quick Start

Installation

# Install from PyPI
pip install vector-db-query

# Or install from source
git clone https://github.com/your-org/vector-db-query.git
cd vector-db-query
pip install -e .

# Install with OCR support
pip install vector-db-query[ocr]
# Also install Tesseract:
# macOS: brew install tesseract
# Ubuntu: sudo apt-get install tesseract-ocr
# Windows: Download from https://github.com/UB-Mannheim/tesseract/wiki

# Install with all features
pip install vector-db-query[all]

# Install additional language packs for OCR
# Ubuntu/Debian:
sudo apt-get install tesseract-ocr-fra  # French
sudo apt-get install tesseract-ocr-deu  # German
sudo apt-get install tesseract-ocr-spa  # Spanish

# macOS:
brew install tesseract-lang

Setup

# 1. Start Qdrant (using Docker)
docker run -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage qdrant/qdrant

# 2. Configure the application
vector-db-query config setup

# 3. Process your first documents
vector-db-query process ~/Documents/my-files --recursive

# 4. Search your documents
vector-db-query query "machine learning algorithms"

# 5. Or use interactive mode for the full experience
vector-db-query interactive start

📖 Usage

Interactive Mode (Recommended)

The interactive mode provides a rich terminal interface:

vector-db-query interactive start

Features:

• 📁 Visual file browser with multi-format preview
• 🔍 Interactive query builder with AI suggestions
• 📊 Beautiful result viewer with syntax highlighting
• ⚙️ Settings editor with live validation
• 📚 Built-in tutorials and examples
• 🎯 Format-specific processing options

Command Line Mode

Processing Documents

# Process all supported formats
vector-db-query process /path/to/documents --recursive

# Process specific formats only
vector-db-query process /path/to/docs --formats pdf,docx,xlsx

# Process with OCR for images
vector-db-query process /path/to/images --ocr --ocr-lang eng

# Show all supported formats
vector-db-query formats

# Check format support for specific files
vector-db-query formats /path/to/file.xyz

# Process only Excel and PowerPoint files
vector-db-query process /path/to/docs --formats xlsx,pptx --recursive

# Dry run to see what would be processed
vector-db-query process /path/to/docs --dry-run --verbose

Querying Documents

# Simple natural language query
vector-db-query query "explain the authentication flow"

# Advanced search with filters
vector-db-query query "Python async" --filter file_type=py --limit 20

# Hybrid search with keyword weight
vector-db-query query "API endpoints" --hybrid --keyword-weight 0.4

# Export results in different formats
vector-db-query query "documentation" --export results.json --format json
vector-db-query query "configuration" --export results.md --format markdown

# Show query statistics
vector-db-query query "machine learning" --stats

Configuration Management

# Show current configuration
vector-db-query config show
vector-db-query config show --format table
vector-db-query config show --section document_processing

# Get/Set configuration values
vector-db-query config get document_processing.chunk_size
vector-db-query config set document_processing.chunk_size 2000 --type int

# Validate configuration
vector-db-query config validate

# Show supported file formats
vector-db-query config formats

# Add custom format
vector-db-query config add-format .custom

# Export/Import configuration
vector-db-query config export --output my-config.yaml
vector-db-query config load custom-config.yaml --merge

# Show environment variable mappings
vector-db-query config env

Monitoring and Management

# Start monitoring dashboard (requires monitoring dependencies)
vector-db-query monitor
# Or install monitoring dependencies first:
# pip install vector-db-query[monitoring]

# View system status
vector-db-query status

# View processing logs
vector-db-query logging show --tail 100
vector-db-query logging search "ERROR" --context 5

# Manage processes with PM2
./scripts/pm2-manage.sh start all
./scripts/pm2-manage.sh status
./scripts/pm2-manage.sh logs mcp-server

MCP Server for AI Assistants

Enable AI assistants like Claude to search your documents:

# Initialize MCP configuration
vector-db-query mcp init

# Start MCP server
vector-db-query mcp start

# Create API client
vector-db-query mcp auth create-client "claude-assistant"

# Check server status
vector-db-query mcp status

# Test with sample query
vector-db-query mcp test --query "find Python examples"

The MCP server provides tools for:

• Searching documents with natural language
• Processing new files in real-time
• Getting collection statistics
• Managing the vector database
• Monitoring system health

⚙️ Configuration

The application uses a flexible YAML-based configuration system:

# config.yaml example
app:
  name: "Vector DB Query System"
  log_level: "INFO"

document_processing:
  chunk_size: 1000
  chunk_overlap: 200
  max_file_size_mb: 100
  
  file_formats:
    documents: [".pdf", ".doc", ".docx", ".txt", ".md"]
    spreadsheets: [".xlsx", ".xls", ".csv"]
    images: [".png", ".jpg", ".jpeg", ".gif", ".bmp"]
    # ... more formats
  
  ocr:
    enabled: true
    language: "eng"
    confidence_threshold: 60.0

vector_db:
  host: "localhost"
  port: 6333
  collection_name: "documents"

embedding:
  model: "embedding-001"
  dimensions: 768

# ... more settings

Environment Variables

Override configuration with environment variables:

export VECTOR_DB_LOG_LEVEL=DEBUG
export QDRANT_HOST=remote-server.com
export QDRANT_PORT=6334
export EMBEDDING_MODEL=text-embedding-ada-002
export OCR_LANGUAGE=eng+fra+deu
export CHUNK_SIZE=1500

🧩 Supported File Formats

Documents

• PDF (.pdf) - Full text extraction with layout preservation
• Microsoft Word (.doc, .docx) - Text, tables, headers/footers, and comments
• OpenDocument Text (.odt) - ODT format support
• Rich Text Format (.rtf) - RTF document processing
• Plain Text (.txt, .text) - With encoding detection
• Markdown (.md, .markdown) - Preserves structure and formatting

Spreadsheets

• Microsoft Excel (.xlsx, .xls) - Extracts:
  • Cell values and formulas
  • Comments and notes
  • Multiple sheets
  • Table structures
• CSV (.csv, .tsv) - Tabular data processing
• OpenDocument Spreadsheet (.ods) - ODS format support

Presentations

• Microsoft PowerPoint (.pptx, .ppt) - Extracts:
  • Slide content and titles
  • Speaker notes
  • Table data
  • Slide numbers and structure
• OpenDocument Presentation (.odp) - ODP format support

Email

• Email Messages (.eml) - Extracts:
  • Headers (From, To, Subject, Date)
  • Body content (text/HTML)
  • Attachments (processed recursively)
  • Thread detection
• Mailbox (.mbox) - Multi-message archive support
• Outlook Message (.msg) - MSG format support

Web & Markup

• HTML (.html, .htm, .xhtml) - Features:
  • Script/style removal
  • Text extraction with structure
  • Link preservation
  • Optional markdown conversion
• XML (.xml) - Structured data extraction

Configuration & Data

• JSON (.json) - Pretty-printed extraction
• YAML (.yaml, .yml) - Multi-document support
• INI/Config (.ini, .cfg, .conf) - Section-based extraction
• TOML (.toml) - TOML format support
• Log Files (.log) - Features:
  • Pattern extraction
  • Summary generation
  • Configurable line limits

Images (with OCR)

Requires Tesseract installation:

• PNG (.png) - Lossless image format
• JPEG (.jpg, .jpeg) - Common photo format
• TIFF (.tiff, .tif) - Multi-page support
• BMP (.bmp) - Bitmap images
• GIF (.gif) - Graphics format

Archives

• ZIP (.zip)
• TAR (.tar, .tar.gz, .tar.bz2, .tar.xz)
• 7-Zip (.7z)

Logs

• Log Files (.log)

🔧 Advanced Features

OCR Configuration

# Install Tesseract
# macOS
brew install tesseract

# Ubuntu/Debian
sudo apt-get install tesseract-ocr

# Install additional languages
sudo apt-get install tesseract-ocr-fra tesseract-ocr-deu

# Configure OCR in vector-db-query
vector-db-query config set document_processing.ocr.enabled true
vector-db-query config set document_processing.ocr.language "eng"
vector-db-query config set document_processing.ocr.confidence_threshold 60.0

Format-Specific Configuration

Configure processing behavior for each file format:

# config/default.yaml
document_processing:
  format_settings:
    excel:
      extract_formulas: true
      extract_comments: true
      process_all_sheets: true
      max_rows_per_sheet: 10000
    
    powerpoint:
      extract_speaker_notes: true
      extract_slide_numbers: true
      include_master_slides: false
    
    email:
      extract_attachments: true
      thread_detection: true
      sanitize_content: true
      include_headers: true
    
    html:
      remove_scripts: true
      remove_styles: true
      convert_to_markdown: false
      preserve_links: true
    
    logs:
      summarize: true
      extract_patterns: true
      max_lines: 10000

Or use environment variables:

export VECTOR_DB_EXCEL_EXTRACT_FORMULAS=true
export VECTOR_DB_EXCEL_MAX_ROWS=5000
export VECTOR_DB_EMAIL_EXTRACT_ATTACHMENTS=true
export VECTOR_DB_HTML_CONVERT_MARKDOWN=true
export VECTOR_DB_LOG_SUMMARIZE=true

Batch Processing

# Python script for batch processing
from vector_db_query import DocumentProcessor

processor = DocumentProcessor(
    chunk_size=1500,
    chunk_overlap=300,
    parallel_workers=8
)

# Process with progress callback
def on_progress(current, total, file_name):
    print(f"Processing {file_name}: {current}/{total}")

documents = processor.process_directory(
    "/path/to/documents",
    recursive=True,
    progress_callback=on_progress
)

Custom Embeddings

# Use custom embedding models
from vector_db_query import EmbeddingService

# Configure custom model
embedding_service = EmbeddingService(
    model="custom-model",
    api_key="your-api-key",
    dimensions=1536
)

# Process with custom embeddings
processor = DocumentProcessor(
    embedding_service=embedding_service
)

📊 Monitoring Dashboard

The built-in monitoring dashboard provides real-time insights:

# Start the dashboard
vector-db-query monitor start

# Access at http://localhost:8501

Features:

• System resource usage (CPU, Memory, Disk)
• Processing queue status
• Document processing statistics
• Error logs and alerts
• Performance metrics

🐳 Docker Support

Run everything in containers:

# Build the image
docker build -t vector-db-query .

# Run with docker-compose
docker-compose up -d

# Access services
# - API: http://localhost:5000
# - Qdrant: http://localhost:6333
# - Dashboard: http://localhost:8501

🧪 Testing

# Run all tests
pytest

# Run specific test categories
pytest tests/test_readers/
pytest tests/test_cli/

# Run with coverage
pytest --cov=vector_db_query

# Run integration tests
pytest tests/integration/ --integration

📚 Documentation

Guides

• Getting Started Guide
• File Formats Guide - Detailed information about all supported formats
• Usage Examples - Practical examples for common use cases
• Configuration Guide
• CLI Features Guide

API Documentation

• Document Readers API - API reference for all document readers
• Full API Reference

Integration & Deployment

• MCP Integration Guide
• Monitoring Setup Guide
• Monitoring System Guide

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📊 Data Sources

The Data Sources feature enables automatic synchronization of content from multiple external sources into your vector database:

Quick Start

# Run interactive setup wizard
vdq setup

# Or use quick start guide
vdq quickstart

# Start syncing data
vdq datasources sync

# Monitor sync status
vdq monitor

Key Capabilities

Gmail Integration

• OAuth2 authentication for secure access
• Folder selection (INBOX, Sent, Drafts, etc.)
• Advanced filtering (sender whitelist/blacklist, patterns)
• Attachment processing
• Thread detection and grouping

Fireflies.ai Integration

• API-based transcript sync
• Real-time webhook support
• Meeting duration and platform filters
• Speaker identification
• Automatic summary extraction

Google Drive Integration

• OAuth2 authentication
• Pattern-based file search (e.g., "Notes by Gemini")
• Folder-specific sync
• Shared drive support
• File type filtering

Advanced Processing

• Deduplication: Content-based hashing to prevent duplicates
• NLP Analysis: Extract entities, sentiment, and key phrases
• Selective Processing: Rule-based filtering system
• Performance: Parallel processing with rate limiting
• Monitoring: Real-time dashboard with metrics

Configuration

The system can be configured via:

• Interactive setup wizard: vdq setup
• Configuration file: config/default.yaml
• Environment variables for sensitive data
• Web UI through monitoring dashboard

Documentation

• Setup Wizard Guide
• Data Sources Operations Guide
• Troubleshooting Guide
• Deployment Guide
• Maintenance Procedures

🙏 Acknowledgments

• Qdrant for the excellent vector database
• Rich for beautiful terminal formatting
• Textual for the interactive TUI
• MCP for AI integration standards
• All our contributors and users!

---

Built with ❤️ by the Vector DB Query Team

GitHub • Documentation • Issues