Skip to main content

🐍 Doc Loader - Python Package

Python PyPI License: MIT Documentation

A comprehensive Python toolkit for extracting and processing documentation from multiple file formats (PDF, TXT, JSON, CSV, DOCX) into a universal JSON structure, optimized for vector stores and RAG (Retrieval-Augmented Generation) systems.

🚀 Features

  • ✅ Universal JSON Output: Consistent format across all document types
  • ✅ Multiple Format Support: PDF, TXT, JSON, CSV, DOCX
  • ✅ Native Performance: Rust backend with PyO3 bindings
  • ✅ Intelligent Text Processing: Smart chunking, cleaning, and metadata extraction
  • ✅ Vector Store Ready: Optimized output for embedding and indexing
  • ✅ Rich Metadata: Comprehensive document and chunk-level metadata
  • ✅ Language Detection: Automatic language detection capabilities

📦 Installation

pip install extracteur-docs-rs

🔧 Usage

Quick Start

import extracteur_docs_rs as doc_loader

# Process any supported file format
result = doc_loader.process_file("document.pdf", chunk_size=500)

print(f"Chunks: {result.chunk_count()}")
print(f"Words: {result.total_word_count()}")
print(f"Supported formats: {doc_loader.supported_extensions()}")

Advanced Usage

import extracteur_docs_rs as doc_loader

# Create a processor with custom parameters
processor = doc_loader.PyUniversalProcessor()
params = doc_loader.PyProcessingParams(
    chunk_size=400,
    overlap=60,
    clean_text=True,
    extract_metadata=True
)

# Process a file
result = processor.process_file("document.txt", params)

# Process text content directly
text_result = processor.process_text_content("Your text here...", params)

# Export to JSON
json_output = result.to_json()

📋 Output Format

All processors generate a standardized JSON structure:

{
  "document_metadata": {
    "filename": "document.pdf",
    "document_type": "PDF",
    "file_size": 1024000,
    "title": "Document Title",
    "author": "Author Name"
  },
  "chunks": [
    {
      "id": "pdf_chunk_0",
      "content": "Extracted text content...",
      "chunk_index": 0,
      "metadata": {
        "size": 1000,
        "language": "en",
        "confidence": 0.95
      }
    }
  ],
  "processing_info": {
    "processor": "PdfProcessor",
    "processing_time_ms": 150,
    "total_chunks": 5,
    "total_content_size": 5000
  }
}

📊 Supported Formats

  • PDF: Text extraction with metadata, page tracking
  • TXT: Encoding detection, language detection, smart chunking
  • JSON: Hierarchical analysis, key extraction, schema inference
  • CSV: Header detection, column analysis, data completeness
  • DOCX: Document structure, style preservation, metadata

🔧 API Reference

Main Functions

# Process any file format
result = doc_loader.process_file(file_path, **options)

# Get supported file extensions
extensions = doc_loader.supported_extensions()

PyUniversalProcessor Class

processor = doc_loader.PyUniversalProcessor()

# Process file with custom parameters
result = processor.process_file(file_path, params)

# Process text content directly
result = processor.process_text_content(text, params)

PyProcessingParams Class

params = doc_loader.PyProcessingParams(
    chunk_size=1000,          # Maximum chunk size
    overlap=100,              # Overlap between chunks
    clean_text=True,          # Enable text cleaning
    extract_metadata=True,    # Extract rich metadata
    detect_language=True      # Enable language detection
)

Result Methods

# Get chunk count
count = result.chunk_count()

# Get total word count
words = result.total_word_count()

# Export to JSON string
json_str = result.to_json()

# Get processing info
info = result.get_processing_info()

🔗 Integration Examples

RAG/Embedding Pipeline

import extracteur_docs_rs as doc_loader
from sentence_transformers import SentenceTransformer

# Process document
result = doc_loader.process_file("document.pdf")

# Extract chunks for embedding
chunks = [chunk["content"] for chunk in result.to_json()["chunks"]]

# Generate embeddings
model = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = model.encode(chunks)

Batch Processing

import extracteur_docs_rs as doc_loader
import os

def process_directory(directory_path):
    results = []
    for filename in os.listdir(directory_path):
        if any(filename.endswith(ext) for ext in doc_loader.supported_extensions()):
            file_path = os.path.join(directory_path, filename)
            result = doc_loader.process_file(file_path)
            results.append(result)
    return results

# Process all documents in a directory
results = process_directory("./documents/")

REST API Integration

from flask import Flask, request, jsonify
import extracteur_docs_rs as doc_loader

app = Flask(__name__)

@app.route('/process', methods=['POST'])
def process_document():
    if 'file' not in request.files:
        return jsonify({'error': 'No file provided'}), 400
    
    file = request.files['file']
    if file.filename == '':
        return jsonify({'error': 'No file selected'}), 400
    
    # Save temporary file
    temp_path = f"/tmp/{file.filename}"
    file.save(temp_path)
    
    try:
        # Process document
        result = doc_loader.process_file(temp_path)
        return jsonify(result.to_json())
    finally:
        os.remove(temp_path)

if __name__ == '__main__':
    app.run()

📈 Performance

  • Fast Processing: Rust backend for optimal performance
  • Memory Efficient: Streaming processing for large files
  • Concurrent Support: Thread-safe processors
  • Scalable: Suitable for production workloads

🔗 Links

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Metadata

Release files for extracteur-docs-rs 0.3.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for extracteur-docs-rs 0.3.2
File Interpreter ABI Platform Actions
extracteur_docs_rs-0.3.2-cp313-cp313-manylinux_2_34_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.34+ x86-64 Details for extracteur_docs_rs-0.3.2-cp313-cp313-manylinux_2_34_x86_64.whl

Release files / extracteur_docs_rs-0.3.2-cp313-cp313-manylinux_2_34_x86_64.whl

Download URL extracteur_docs_rs-0.3.2-cp313-cp313-manylinux_2_34_x86_64.whl
Size 1.5 MB
Tags CPython 3.13 Linux glibc 2.34+ x86-64
SHA-256 checksum
How to use checksums
dfb811a37385533b8578f3393215bab0e7570f786a5306fc8bac2e11fac88481
BLAKE2b-256 checksum
How to use checksums
2d9adf54abbe7d46389f9c2c68de9664b7635c73f7654ccbfadd8be1f30d2295
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.3

Security for extracteur-docs-rs 0.3.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page