Super Document Processor

A high-fidelity, multi-format document parsing and chunking engine tailored for Retrieval-Augmented Generation (RAG) pipelines. It extracts structured text, preserves tables, optionally transcribes embedded charts and images with Vision LLMs, and outputs boundary-aligned chunks.

Available as a Python library, CLI tool, and FastAPI microservice (with Docker support).

---

Supported Formats

Category	Formats	Features Extracted
Documents	.pdf, .docx	Headings, paragraphs, markdown/HTML tables, hyperlinks, embedded images
Presentations	.pptx	Slide text, shapes, notes, multi-table separation, smart-art, charts
Emails	.msg	Headers (From, To, Subject, Date) and formatted email body
Images	.jpg, .jpeg, .png, .jfif	Full VLM vision analysis and table transcription
Code & Text	.txt, .md, .json, .csv, .py, .js, .ts, .html, etc.	Natural paragraph and token-bounded chunking

---

Quickstart

1. Local Setup

git clone https://github.com/Cinderfox/Super-Document-Processor.git
cd Super-Document-Processor

# Setup environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .\.venv\Scripts\activate
pip install -r requirements.txt

# Configure environment (optional, required only for Vision LLMs)
cp .env.example .env

2. Docker Setup

docker build -t super-document-processor:latest .
docker run -d --name doc-processor -p 8000:8000 -v ${PWD}/data:/data super-document-processor:latest

---

1. Python Library Usage

from super_document_processor import process_file

# 1. Standard text & table extraction (zero LLM calls, ultra-fast)
chunks = process_file("annual_report.pdf")

# 2. Extract with Vision LLM (describes embedded charts, diagrams, and photos)
chunks_with_vision = process_file(
    file_path="pitch_deck.pptx",
    process_images=True,
    model="google-gemini-3-flash"  # e.g. "gpt-4o", "google-gemini-3-flash", etc.
)

for chunk in chunks:
    print(f"--- Chunk {chunk['chunkIndex']} ({chunk['filename']}) ---")
    print(chunk["content"])

Chunk Data Structure

[
  {
    "content": "==================== Slide 1 ====================\nSlide Text:\n- Performance Scorecard...",
    "chunkIndex": 1,
    "filename": "pitch_deck.pptx"
  }
]

Direct Chunker Access & Default Limits

Text-based document processors (.docx, .msg, .txt etc) use ChunkCollector for token-safe output partitioning:

• Default Max Size: 6,000 characters per chunk
• Default Overlap: 0 characters (zero duplicated text across chunks)

from super_document_processor.utils.chunker import collector, chunk_text

# Using standalone utility with custom values:
chunks = chunk_text(text="Your long text...", chunk_size=6000, overlap=200)

# Or collecting and retrieving chunked texts:
collector.add("First section text...")
chunks = collector.get_chunked_texts(max_size=6000, overlap=200)
collector.clear()

PPTX Slide Rendering & Image Conversion

Export PowerPoint slides directly to SVGs or PNGs:

from utils.pptx_converter import convert_pptx, create_pptx_render_zip

# 1. Convert all slides to SVGs (returns list of clean SVG strings)
res = convert_pptx("presentation.pptx", format="svg")
for s in res["slides"]:
    print(f"Slide {s['slide']}: {s['svg'][:80]}...")

# 2. Convert specific slide to high-res PNG bytes
res_png = convert_pptx("presentation.pptx", format="png", slide=1, scale=2.0)
with open("slide_1.png", "wb") as f:
    f.write(res_png["raw_bytes"])

# 3. Create a downloadable ZIP containing all rendered slides
zip_bytes = create_pptx_render_zip("presentation.pptx", format="png")
with open("presentation_pngs.zip", "wb") as f:
    f.write(zip_bytes)

---

2. CLI Usage

# Basic extraction to terminal (stdout)
python main.py document.pdf

# Save raw JSON chunks to a file
python main.py report.docx --json -o output.json

# Process embedded images using a Vision Model
python main.py slides.pptx --process-images --model google-gemini-3-flash --json -o output.json

# Convert all PPTX slides to clean SVG files in a directory
python main.py slides.pptx --convert-pptx --format svg -o ./output_svgs

# Convert specific slide to PNG
python main.py slides.pptx --convert-pptx --format png --slide 1 -o slide_1.png

# Convert PPTX and package all slides into a ZIP archive
python main.py slides.pptx --convert-pptx --format png --as-zip -o slides.zip

# Start the REST API server
python main.py --serve --port 8000

Flag	Short	Default	Description
file		None	Path to document or image file
--json		False	Output as raw JSON chunks or JSON slide array
--output	-o	stdout	Destination file path or directory for output
--convert-pptx	-c	False	Convert PPTX slides to SVG or PNG images
--format	-f	svg	Target conversion format (svg or png)
--slide		All	Specific 1-based slide index to convert
--scale		2.0	Resolution scale factor for PPTX rendering
--as-zip		False	Bundle all converted slides into a single .zip file
--process-images		False	Enable AI vision processing for images/tables
--model	-m	None	Vision model name (required if --process-images is set)
--serve	-s	False	Start the FastAPI HTTP microservice
--port	-p	8000	Server listening port

---

3. REST API Microservice

Start the server with python main.py --serve --port 8000 or via Docker.

• Swagger Docs: http://localhost:8000/docs
• Health Check: http://localhost:8000/health

1. Document Parsing & Chunking: POST /process

Accepts either a file upload (multipart/form-data) or a local/volume file path (application/json). Providing both or neither returns an HTTP 400 error.

Option A: File Upload (Multipart Form)

# Basic upload
curl -X POST "http://localhost:8000/process" \
     -F "file=@/path/to/contract.pdf"

# With Vision LLM processing
curl -X POST "http://localhost:8000/process?process_images=true&model=google-gemini-3-flash" \
     -F "file=@/path/to/presentation.pptx"

Option B: Local File Path or Mounted Volume Path (JSON)

# Running locally:
curl -X POST "http://localhost:8000/process" \
     -H "Content-Type: application/json" \
     -d '{"file_path": "./test_docs/report.pdf"}'

# Running in Docker:
curl -X POST "http://localhost:8000/process" \
     -H "Content-Type: application/json" \
     -d '{"file_path": "/data/report.pdf", "process_images": true, "model": "gpt-4o"}'

---

2. PPTX to SVG / PNG Conversion: POST /convert/pptx

Renders PowerPoint slides into SVG or PNG images.

Stream Single Slide Directly as Image (image/svg+xml or image/png)

Ideal for embedding directly in <img> tags or displaying in browsers:

# Get Slide 1 as direct SVG
curl -X POST "http://localhost:8000/convert/pptx?slide=1&format=svg" \
     -F "file=@presentation.pptx" > slide_1.svg

# Get Slide 2 as direct high-res PNG
curl -X POST "http://localhost:8000/convert/pptx?slide=2&format=png&scale=2.0" \
     -F "file=@presentation.pptx" > slide_2.png

Download All Slides as a ZIP Archive

curl -X POST "http://localhost:8000/convert/pptx?format=png&as_zip=true" \
     -F "file=@presentation.pptx" -o presentation_slides.zip

Convert Mounted Volume Path via JSON Body

curl -X POST "http://localhost:8000/convert/pptx" \
     -H "Content-Type: application/json" \
     -d '{"file_path": "/data/presentation.pptx", "format": "svg"}'

Python Client Example

import requests

url = "http://localhost:8000/convert/pptx"

# 1. Download Slide 1 directly as image
with open("pitch_deck.pptx", "rb") as f:
    resp = requests.post(url, files={"file": f}, params={"slide": 1, "format": "png"})
    with open("slide_1.png", "wb") as out:
        out.write(resp.content)

# 2. Get all slides as JSON payload (contains SVGs or base64 PNGs)
with open("pitch_deck.pptx", "rb") as f:
    resp = requests.post(url, files={"file": f}, params={"format": "svg", "response_type": "json"})
    data = resp.json()
    print(f"Rendered {data['total_slides']} slides.")

---

3. Async Processing: POST /process-async & GET /jobs/{job_id}

For large files or heavy multimodal extraction, submit jobs asynchronously to prevent client gateway timeouts. No Redis required (backed by persistent /data/jobs storage).

1. Submit Job (POST /process-async)

curl -X POST "http://localhost:8000/process-async" \
     -F "file=@large_document.pdf"

{
  "job_id": "e8d7a12b-34c5-4f67-8901-abcdef123456",
  "filename": "large_document.pdf",
  "status": "processing",
  "poll_url": "/jobs/e8d7a12b-34c5-4f67-8901-abcdef123456"
}

2. Poll Status & Fetch Result (GET /jobs/{job_id})

curl "http://localhost:8000/jobs/e8d7a12b-34c5-4f67-8901-abcdef123456"

• In Progress: {"status": "processing", "progress": 0}
• Completed: Returns the full JSON payload ("status": "completed", total_chunks, chunks). The job file is automatically deleted from disk upon successful retrieval.

---

4. Docker Deployment

The container image includes all rendering dependencies (libicu, ca-certificates, etc.).

Run as an API Server

# Port 8000 exposed, local ./data folder mounted to /data in container
docker run -d \
  --name doc-processor \
  -p 8000:8000 \
  --env-file .env \
  -v ${PWD}/data:/data \
  super-document-processor:latest

Run as One-Off CLI Tool & Save Output to Host

Files created inside Docker default to the container's isolated layer. To save the output JSON directly on your host machine, use either Volume Mounting or Shell Redirection:

# Method 1: Mount folder and write with -o (appears in your host folder)
docker run --rm -v ${PWD}:/data super-document-processor:latest \
  python main.py /data/report.pdf --json -o /data/output.json

# Method 2: Stream stdout directly to host file (works with docker run or docker exec)
docker run --rm -v ${PWD}:/data super-document-processor:latest \
  python main.py /data/report.pdf --json > output.json

# Method 3: Run against an already active container
docker exec doc-processor python main.py /data/report.pdf --json > output.json

Docker Compose

version: "3.8"

services:
  doc-processor:
    image: super-document-processor:latest
    container_name: doc-processor
    restart: unless-stopped
    ports:
      - "8000:8000"
    env_file:
      - .env
    volumes:
      - ./data:/data

docker compose up -d

---

Environment Variables

Variable	Default	Description
PORT	8000	HTTP port when running in server mode
OPENAI_API_KEY	None	API key for Vision LLM extraction
OPENAI_BASE_URL	https://api.openai.com/v1	Custom gateway URL (LiteLLM, Ollama, Gemini, Groq, etc.)

---

License

MIT License © Cinderfox