Omni-Pre-Processor 0.5.8

Omni Pre-Processor: Document content extraction package

OPP - Omni Pre-Processor

Document content extraction for DOCX, PPTX, PDF, XLSX, CSV, JSON, XML, HTML, EPUB, EML, MSG, and Image (OCR).

Features

• Multi-format extraction - DOCX, PPTX, PDF, XLSX, CSV, JSON, XML, HTML, EPUB, EML, MSG, Image, IPYNB, YouTube URL
• Inline formatting tracking - Bold, italic, underline, strikethrough preserved in XLIFF as <bx>/<ex> tags for downstream formatting restoration
• Image OCR - Tesseract and RapidOCR with graceful fallback
• Email extraction - EML (RFC 822) and MSG (Outlook) with attachment recursion
• Audio/Video transcription - Whisper-based ASR
• Format auto-detection - Magic bytes detection (extension not required)
• Resource management - MD5 deduplication, UUID naming for images
• Pipeline orchestrator - detect → extract → manage → report
• CLI interface - Full command-line with batch support
• Output formats - Markdown and XLIFF 1.2/2.0
• Manifest metadata - JSON manifest with source info, extraction stats, and image data
• Skeleton preservation - Original DOCX/PPTX ZIP structure preserved for downstream XLIFF→DOCX/PPTX backfill

Installation

# Core package
pip install -e .

# With office/data formats (XLSX, CSV, JSON, XML)
pip install -e ".[office]"

# With email and OCR (EML, MSG, Tesseract, RapidOCR)
pip install -e ".[email]"

Quick Start

Python API

from opp import DOCXExtractor, PDFExtractor, PPTXExtractor
from opp.detector import detect_format
from opp.pipeline import OPPPipeline

# Direct extraction
extractor = DOCXExtractor()
result = extractor.extract("document.docx")
print(result.content)

# Auto-detection
fmt, confidence = detect_format("document.docx")
print(f"Format: {fmt.value}, Confidence: {confidence}")

# Full pipeline
pipeline = OPPPipeline(resource_storage_dir="./resources")
result = pipeline.process_file("document.docx")
print(f"Extracted: {len(result.content)} chars, {result.images_stored} images")

CLI

# Extract to Markdown
opp --target-format=md document.docx

# Extract to XLIFF for translation
opp --target-format=xlf --source-lang=en --target-lang=zh document.docx

# Generate both MD and XLIFF
opp --target-format=both --source-lang=en --target-lang=zh document.docx

# Custom output directory
opp --target-format=md --output-dir ./output document.docx

# Image OCR
opp --ocr-engine tesseract scan.png

# Batch processing
opp --batch file1.docx file2.pdf file3.pptx

Windows Batch Scripts

Script	Description
md.bat	Convert to Markdown
en2cn_xliff.bat	English source → Chinese XLIFF
cn2en_xliff.bat	Chinese source → English XLIFF

md.bat "document.docx"
md.bat "folder"

en2cn_xliff.bat "english.docx"
cn2en_xliff.bat "中文.docx"

Supports drag-drop of files and folders. Logs saved to logs/.

Output Files

Every extraction produces a manifest.json and optionally a skeleton.zip:

output_dir/
├── document.md              # Extracted Markdown
├── document.xlf             # Extracted XLIFF (translation-ready)
├── document_manifest.json   # Metadata about source and extraction
└── document.skeleton.zip    # Original DOCX/PPTX ZIP (for backfill)

manifest.json

Records source file info, extraction outputs, and resources:

{
  "manifest_version": "1.0",
  "generated_at": "2026-05-22T14:30:00Z",
  "tool": "OPP",
  "tool_version": "0.2.0",
  "source": {
    "file_path": "/path/to/spec.docx",
    "original_filename": "spec.docx",
    "format": "DOCX",
    "file_size_bytes": 45824,
    "file_hash_md5": "a1b2c3d4e5f6..."
  },
  "extraction": {
    "source_lang": "en",
    "target_lang": "zh",
    "outputs": {
      "markdown": { "path": "spec.md", "paragraph_count": 150, "table_count": 3 },
      "xliff": { "path": "spec.xlf", "trans_unit_count": 42 }
    },
    "images": [
      { "mime_type": "image/png", "width": 800, "height": 600, "data_size_bytes": 24580 }
    ],
    "warnings": []
  },
  "resources": { "storage_dir": "resources", "image_count": 5 },
  "skeleton": {
    "path": "spec.skeleton.zip",
    "format": "ZIP",
    "key_files": ["word/document.xml", "word/styles.xml", "[Content_Types].xml"]
  }
}

skeleton.zip

Preserves the original OOXML ZIP structure for DOCX/PPTX files. This enables downstream ORF tools to perform XLIFF→DOCX/PPTX backfill by replacing content in the preserved skeleton.

Format	Key Files Preserved
DOCX	word/document.xml, word/styles.xml, word/numbering.xml, word/settings.xml, [Content_Types].xml
PPTX	All files under ppt/ prefix (slides, layouts, media)

Project Structure

src/opp/
├── detector.py           # Format auto-detection
├── extractors/           # Document extractors
│   ├── docx.py
│   ├── pptx.py
│   ├── pdf.py
│   ├── xlsx.py
│   ├── csv.py
│   ├── json.py
│   ├── xml.py
│   ├── email.py
│   └── image_ocr.py
├── channels/             # Output formatters
│   ├── table_channel.py   # DataFrame → Markdown table
│   └── keyvalue_channel.py # dict → XLIFF
├── xliff/                # XLIFF 1.2/2.0 generator
├── pipeline.py           # OPPPipeline orchestrator
├── resource_manager.py   # Image deduplication
└── cli.py               # Command-line interface

Architecture

                     ┌─────────────────────────────────────────┐
                     │              OPPPipeline                  │
                     │  detect_format() → Extractor → Report   │
                     └─────────────────────────────────────────┘

┌──────────┐    ┌───────────┐    ┌────────────────┐    ┌──────────────┐
│ detector │───▶│ extractors│───▶│resource_manager│───▶│error_handler │
│  magic   │    │  DOCX/...  │    │  MD5 + UUID    │    │ HTML/text    │
└──────────┘    └───────────┘    └────────────────┘    └──────────────┘

Development

pip install -e ".[dev]"
pytest tests/ -v --cov=src/opp --cov-report=term-missing

Test Coverage

Module	Tests
detector	13
resource_manager	18
error_handler	18
integration	25
cli	18
e2e	52
xliff	40+
extractors	140+
inline formatting	53
manifest generation	6
skeleton preservation	6
Total	544+

Batch Testing

Test files available in batch_test/ covering all formats.

opp --target-format=both --source-lang=en --target-lang=zh --output-dir=output batch_test/

Pipeline — Omni Localization Suite

OPP is Step 1 of the Omni Localization Suite pipeline:

┌────────────────────────────────────────────────────────────────────────┐
│                     OMNI LOCALIZATION SUITE                             │
│                                                                        │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐               │
│  │     OPP     │───▶│     OL      │───▶│     ORF      │               │
│  │  (提取)     │    │   (翻译)    │    │   (回写)    │               │
│  └─────────────┘    └─────────────┘    └─────────────┘               │
│                                                                        │
│  Step 1: OPP        Step 2: OL            Step 3: ORF                  │
│  Extract →          Translate →           Backfill →                  │
│  MD + XLIFF +       MD + XLIFF            DOCX/PPTX                   │
│  skeleton.zip                                                    │
└────────────────────────────────────────────────────────────────────────┘

Complete Workflow

# Step 1: OPP - Extract document to MD/XLIFF + skeleton.zip
opp --target-format=both --source-lang=en --target-lang=zh document.docx
# Output: document.md, document.xlf, document_manifest.json, document.skeleton.zip

# Step 2: OL - Translate to target language
ol translate-md document.md -s en -t zh -o translated/

# Step 3: ORF - Backfill translated content to target format
orf apply-xliff document.docx --xliff translated/document.xlf --output result.docx

Related Projects

• OL (Omni-Localizer) - NEXT STEP after OPP. Translates MD/XLIFF produced by OPP.
• ORF (Omni-Re-Formatter) - Backfills translated content to DOCX/PPTX/EPUB.

For AI Agents

OPP outputs standardized artifacts for downstream processing:

Artifact	Description	Used By
{name}.md	Markdown with YAML frontmatter (source_lang, target_lang)	OL (translate-md)
{name}.xlf	XLIFF 1.2/2.0 with <bx>/<ex> inline tags	OL (translate-xliff)
{name}_manifest.json	Metadata: source info, output paths, resources	ORF (manifest parser)
{name}.skeleton.zip	Original DOCX/PPTX ZIP structure	ORF (XLIFF→DOCX backfill)

MCP Tools Available: extract_document, batch_extract, detect_format, generate_markdown, generate_xliff

MCP Server (Agent-Facing)

The OPP MCP server provides document extraction capabilities to AI agents via the Model Context Protocol. AI assistants can use these tools to process documents without needing to understand OPP's internal architecture.

Why Use the MCP Server?

• Agent integration - Connect OPP to any MCP-compatible AI assistant
• stdio transport - Communication over standard input/output for security
• 5 extraction tools - Cover all major document formats
• Path security - Directory allowlist prevents unauthorized file access

Installation

# Install OPP with MCP server support
pip install -e ".[mcp]"

Quick Start

Start the server manually:

python -m opp.mcp.server

Auto-start with uvx:

uvx opp-mcp-server

Auto-start with npx:

npx opp-mcp-server

Hermes Configuration

Add OPP to your Hermes agent configuration:

agents:
  my-agent:
    tools:
      - name: opp
        type: code
        config:
          server_command: uvx opp-mcp-server
          allowed_directories:
            - /path/to/documents
            - /path/to/output

Available Tools

Tool	Description
extract_document	Extract content from a single document file. Supports DOCX, PPTX, PDF, XLSX, CSV, JSON, XML, HTML, EPUB, EML, MSG, and images. Returns markdown or structured content.
batch_extract	Process multiple files in one request. Takes an array of file paths and processes them sequentially. Returns extraction results for each file.
detect_format	Identify the file format of a document using magic bytes detection. Works regardless of file extension. Returns format name and confidence score.
generate_markdown	Convert a document to markdown format. Specify source and target languages for proper text processing.
generate_xliff	Convert a document to XLIFF format for translation workflows. Requires source-lang and target-lang parameters.

Security

The MCP server enforces path validation to prevent unauthorized file access.

Allowlist configuration:

# Via environment variable
export OPP_MCP_ALLOWED_DIRS="/allowed/documents,/allowed/output"

# Via configuration file
# Create opp_mcp_config.yaml with:
# security:
#   allowed_directories:
#     - /path/to/documents
#     - /path/to/output

**Configuration file** (`opp_mcp_config.yaml`):

```yaml
security:
  allowed_directories:
    - /mnt/d/贯维/Documents
    - /mnt/d/贯维/Output
    - ./documents

server:
  host: localhost
  port: 8765

extraction:
  default_target_format: md
  ocr_engine: tesseract

Environment Variables

Variable	Description	Default
OPP_ALLOWED_DIRECTORIES	Comma-separated list of allowed directories	Required
OPP_RESOURCE_STORAGE_DIR	Directory for extracted images	./resources
OPP_OCR_ENGINE	OCR engine to use	tesseract
OPP_LOG_LEVEL	Logging level	INFO