pdf2tex 0.2.0

High-performance RAG-based PDF to LaTeX conversion module

PDF2TeX

High-performance RAG-based PDF to LaTeX conversion module for large documents (2000+ pages).

Features

• Intelligent PDF Extraction: Multi-path content processing with PyMuPDF, Nougat, and PaddleOCR
• Math-First Approach: 95%+ accuracy on mathematical content using neural equation recognition
• RAG-Powered Generation: Context-aware LaTeX synthesis with Hugging Face LLMs
• Distributed Processing: Ray-based parallel processing for high throughput
• Chapter-Based Output: One .tex file per chapter with master document

Architecture

PDF Input → Extract + OCR → Chunk + Index → RAG + LLM → LaTeX Output
                ↓                ↓              ↓
            Ray Distributed Workers Pool
                        ↓
        Qdrant | Redis | MinIO | Postgres

Quick Start

Prerequisites

• Python 3.11+
• Docker & Docker Compose
• NVIDIA GPU (recommended)
• Hugging Face API token

Installation

# Clone repository
git clone https://github.com/pdf2tex/pdf2tex.git
cd pdf2tex

# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -e ".[dev]"

# Start infrastructure
docker-compose up -d

# Run conversion
pdf2tex convert input.pdf --output ./output

Configuration

Create a .env file:

# Hugging Face
HUGGINGFACE_TOKEN=hf_xxxxxxxxxxxxx

# Database
POSTGRES_URL=postgresql+asyncpg://pdf2tex:password@localhost:5432/pdf2tex

# Vector Store
QDRANT_URL=http://localhost:6333

# Redis
REDIS_URL=redis://localhost:6379

# MinIO
MINIO_ENDPOINT=localhost:9000
MINIO_ACCESS_KEY=minioadmin
MINIO_SECRET_KEY=minioadmin

# Ray
RAY_ADDRESS=auto

Usage

CLI

# Convert a PDF
pdf2tex convert document.pdf --output ./output

# Resume failed conversion
pdf2tex resume doc_abc123

# Check status
pdf2tex status doc_abc123

API

# Start API server
uvicorn pdf2tex.api.app:app --host 0.0.0.0 --port 8000

# Submit document
curl -X POST http://localhost:8000/documents \
  -F "file=@textbook.pdf"

# Check status
curl http://localhost:8000/documents/doc_abc123

Python SDK

from pdf2tex import PDF2TeX

converter = PDF2TeX()
result = await converter.convert("textbook.pdf", output_dir="./output")
print(f"Converted {result.total_pages} pages in {result.duration}")

Project Structure

pdf2tex/
├── src/pdf2tex/
│   ├── extraction/     # PDF parsing, OCR, math extraction
│   ├── chunking/       # Text splitting, chapter detection
│   ├── rag/            # Embeddings, vector store, retrieval
│   ├── generation/     # LLM integration, LaTeX synthesis
│   ├── pipeline/       # Orchestration, distributed workers
│   └── api/            # FastAPI endpoints
├── tests/
├── docker-compose.yml
└── pyproject.toml

Performance

Document Size	Processing Time	Workers
500 pages	~20 min	10
1000 pages	~40 min	20
2000 pages	~72 min	20

License

MIT License - see LICENSE for details.