pdf2text-agent 0.1.1

A Python tool to extract text from various PDF types using direct extraction and OCR, with built-in memory management.

PDF-to-Text Agent

The PDF-to-Text Agent is a Python application designed to extract textual content from various types of PDF documents. It intelligently analyzes PDFs to determine if they are text-based, scanned (image-based), or mixed, and then applies the optimal extraction strategy. The agent incorporates memory management techniques to handle large files efficiently and provides options for different processing modes and output formats.

Table of Contents

• Features
• Quick Start
• Directory Structure
• Setup and Installation
• Usage
• Configuration
• Contributing
• License

Features

• Versatile PDF Handling: Supports text-based, scanned (OCR), and mixed PDF documents.
• Intelligent Analysis: Automatically analyzes PDF content to choose the most effective extraction method (direct text, OCR, or hybrid).
• Configurable Processing Modes: Offers different modes (Fast, Balanced, Quality) to balance speed and extraction accuracy.
• Memory Efficient: Implements chunking for large file processing and includes an active memory manager to prevent crashes and optimize resource usage.
• Multiple Output Formats: Extracted content can be saved as plain text (.txt) or structured JSON (.json) files.
• Batch Processing: Capable of processing multiple PDF files from a directory in a single run.
• Customizable Configuration: Core settings (memory limits, OCR language, output paths, etc.) are configurable via pdf2text/config.py and command-line arguments.
• Stress Testing Tool: Includes tools/stress_test.py for performance testing and aiding in the detection of potential memory leaks.
• Detailed Logging: Provides comprehensive logs for monitoring and troubleshooting.

Quick Start

This guide will get you up and running with the PDF-to-Text Agent quickly.

1. Prerequisites:

   • Python 3.7+ installed.
   • pip (Python package installer) installed.
   • Tesseract OCR installed (see Setup and Installation for details).

2. Clone & Setup:

   # Clone the repository (if you haven't already)
   # git clone <repository_url>
   # cd <repository_directory>
   
   # Create and activate a virtual environment
   python -m venv .venv
   # On Windows:
   # .venv\Scripts\activate
   # On macOS/Linux:
   source .venv/bin/activate
   
   # Install the package and dependencies
   pip install .
   

3. Run on a Sample PDF: The project includes a sample PDF. Try converting it:

   pdf2text-agent pdf2text/sample_pdf/sample.pdf
   

   This will process sample.pdf using default settings (balanced mode, outputting both text and JSON files) and save the results in the pdf2text/output/ directory.

4. Check the Output: Look for sample_extracted.txt and sample_extracted.json in pdf2text/output/text_files/ and pdf2text/output/json_files/ respectively.

For more detailed instructions, options, and batch processing, please refer to the Usage section.

Directory Structure

.
├── pdf2text/
│   ├── main.py                 # Main command-line interface entry point
│   ├── config.py               # Application configuration settings
│   │
│   ├── core/                   # Core application logic
│   │   ├── agent.py            # Orchestrates the PDF processing pipeline
│   │   ├── file_manager.py     # Handles file I/O operations
│   │   └── memory_manager.py   # Manages memory usage and cleanup
│   │
│   ├── analyzers/              # Modules for PDF analysis
│   │   ├── pdf_analyzer.py     # Analyzes PDF type and characteristics
│   │   └── memory_estimator.py # Estimates memory requirements for processing
│   │
│   ├── extractors/             # Modules for text extraction
│   │   ├── text_extractors.py  # Extracts text from text-based PDFs (uses PyMuPDF)
│   │   ├── ocr_extractors.py   # Extracts text from scanned PDFs using OCR (Tesseract)
│   │   └── hybrid_extractors.py # Combines text and OCR extraction strategies
│   │
│   ├── utils/                  # Utility modules
│   │   └── logger.py           # Logging configuration
│   │
│   ├── sample_pdf/             # Contains sample PDF files for testing
│   │   └── sample.pdf
│   └── output/                 # Default directory for generated output files
│       ├── text_files/         # Plain text (.txt) outputs
│       ├── json_files/         # JSON (.json) outputs
│       └── ...                 # Other potential output subdirectories (logs, temp)
│
├── tools/
│   └── stress_test.py          # Script for stress testing and memory leak detection
│
├── requirements.txt            # Python package dependencies
├── README.md                   # This file
├── .gitignore                  # Specifies intentionally untracked files
├── pyproject.toml              # Project metadata (often for build systems like Poetry/Flit)
└── uv.lock                     # Lock file for uv pip installer (if used)

Setup and Installation

Follow these steps to set up and run the PDF-to-Text Agent:

1. Get the Code: If you haven't done so already (e.g., via the Quick Start), clone the repository or download the source code.

   # Example for cloning:
   # git clone <repository_url> # Replace with actual URL if hosted
   # cd <repository_directory>
   

   If you have the files locally, navigate to the project's root directory.

2. Create a Python Virtual Environment (Recommended):

   python -m venv .venv
   

   Activate the virtual environment:

   • On Windows:

     .venv\Scripts\activate
     

   • On macOS and Linux:

     source .venv/bin/activate
     

3. Install the Package: Ensure your virtual environment is activated. There are a couple of ways to install the pdf2text-agent:

       *   **Editable install from local source (for development)**:
           This is useful if you are making changes to the code. From the project's root directory:
           ```bash
           pip install -e .
           ```
       *   **Standard install from local source**:
           From the project's root directory:
           ```bash
           pip install .
           ```
       *   **(Future) Install from PyPI**:
           Once the package is published to the Python Package Index (PyPI), you'll be able to install it like any other package:
           ```bash
           # pip install pdf2text-agent # (This is a placeholder until published)
           ```
       Installing the package will automatically handle the dependencies listed in `pyproject.toml`. The `requirements.txt` file can still be useful for specific development environments or for reference.
   

4. Install Tesseract OCR: This application uses Tesseract OCR for processing scanned documents. You need to install Tesseract on your system.

   • Linux (Debian/Ubuntu):

     sudo apt-get update
     sudo apt-get install tesseract-ocr
     sudo apt-get install libtesseract-dev # For development headers if needed
     # Install language packs as needed, e.g., for English:
     sudo apt-get install tesseract-ocr-eng
     

   • macOS (using Homebrew):

     brew install tesseract
     # Install language packs, e.g., for English:
     brew install tesseract-lang
     

   • Windows: Download the Tesseract installer from the official Tesseract at UB Mannheim page. During installation, make sure to: * Add Tesseract to your system PATH. * Select and install the language data packs you need (e.g., English).
   • Verify Installation: After installation, open a new terminal/command prompt and type tesseract --version. You should see the Tesseract version information.
   • For more details and other operating systems, refer to the official Tesseract installation documentation.

5. Configuration (Optional):

   • Default settings are provided in pdf2text/config.py. You can modify this file to change default behaviors such as output directories, memory limits, or OCR languages.

Usage

Ensure your virtual environment is activated and Tesseract OCR is installed before running the agent.

Command-line Interface

The PDF-to-Text Agent is run from the command line using the pdf2text-agent command (once the package is installed). The basic command structure is:

pdf2text-agent <input_path_or_dir> [options]

• <input_path_or_dir>: This is a positional argument. It should be the path to a single PDF file, or if the --batch option is used, it should be the path to a directory containing PDF files.

Common Options:

The following options are available to customize the processing:

• --mode [fast|balanced|quality]: Sets the processing mode.
  • fast: Prioritizes speed over extraction accuracy. Useful for quick previews or when OCR quality is less critical.
  • balanced: Offers a balance between speed and accuracy. This is the default mode.
  • quality: Prioritizes the highest possible extraction accuracy, which may be slower. Recommended for documents requiring careful OCR.
• --output [text|json|both]: Specifies the output format.
  • text: Saves a plain .txt file containing the extracted text.
  • json: Saves a detailed .json file with extracted content, metadata, and processing information.
  • both: Saves both .txt and .json files. This is the default.
• --language <language_code>: Specifies the language(s) for OCR (Optical Character Recognition). For example, use eng for English, deu for German, or eng+fra for English and French. The default is eng.
• --memory-limit <MB>: Overrides the maximum memory (in Megabytes) the application should attempt to use. This can be useful for processing very large files or in memory-constrained environments.
• --verbose or -v: Enables verbose logging, providing more detailed output about the processing steps, including configuration details and memory usage.
• --config: Displays the current configuration settings (derived from pdf2text/config.py and any command-line overrides) and then exits without processing any files.

Output files are saved in the pdf2text/output/ directory by default. This can be configured in pdf2text/config.py.

Examples:

• Process a single PDF with default settings (balanced mode, output both text and JSON):

  pdf2text-agent documents/report.pdf
  

• Process a scanned PDF in quality mode and save only as a text file:

  pdf2text-agent scans/image_based.pdf --mode quality --output text
  

• Process a German PDF using OCR (default output is both text and JSON):

  pdf2text-agent invoices/invoice_de.pdf --language deu
  

• Display the current configuration:

  pdf2text-agent --config
  

• Process a PDF with verbose output:

  pdf2text-agent important_document.pdf --verbose
  

Batch Processing Multiple PDFs

To process all PDF files within a specific directory, use the --batch flag along with the directory path.

pdf2text-agent <path_to_your_pdf_directory> --batch [options]

You can combine batch processing with other options like --mode or --output:

• Process all PDFs in the project_files/reports/ directory using fast mode and saving only JSON output:

  pdf2text-agent project_files/reports/ --batch --mode fast --output json
  

Running the Stress Test Tool

The application includes a stress testing tool to help identify potential memory leaks by repeatedly processing PDFs.

python tools/stress_test.py --pdf_dir <path_to_directory_with_test_pdfs> [--iterations <number_of_runs>] [--output_dir <path_for_stress_test_outputs>]

• --pdf_dir: (Required) Path to a directory containing a diverse set of PDF files for testing.
• --iterations: (Optional) Number of times the batch processing will be repeated. Defaults to 5.
• --output_dir: (Optional) If specified, output files from the stress test runs will be saved here in iteration-specific subdirectories. Otherwise, temporary directories are used and cleaned up.

Example:

python tools/stress_test.py --pdf_dir pdf2text/sample_pdf/ --iterations 10

Monitor the console output for memory usage statistics after each iteration. A consistent increase in memory usage might indicate a leak.

Configuration

Beyond the command-line arguments, more granular control over the application's behavior can be achieved by modifying the settings in pdf2text/config.py.

This file includes options such as:

• Default OCR language.
• Memory management thresholds and limits.
• Default output directory paths.
• Logging levels and formatting.
• Parameters for PDF analysis and memory estimation.

Please refer to the comments within pdf2text/config.py for detailed explanations of each configuration option.

Contributing

Contributions are welcome! If you'd like to improve the PDF-to-Text Agent or add new features:

1. Fork the repository.
2. Create a new branch for your feature or bug fix.
3. Make your changes.
4. Ensure your code adheres to good practices and includes relevant comments or documentation.
5. If adding new features or fixing bugs, consider adding or updating tests.
6. Submit a pull request with a clear description of your changes.

We appreciate your help in making this tool better!

License

This project is licensed under the MIT License.