rc-docparser

Published on PyPI as rc-docparser; import it as docparser (import docparser).

Convert research literature (.docx, .xlsx, .pdf, .html, .pptx, .epub, .txt, .md, .csv) into a clean, reproducible Markdown + JSON corpus, with optional vision-language captioning of every embedded figure via OpenRouter, OpenAI, Gemini, a local server, or a fully-local model.

                              ┌────────────────┐
  data/raw/*.docx             │   docparser    │       data/parsed/<slug>/
  data/raw/*.xlsx             │  - parse_docx  │       document.md
  data/raw/*.pdf      ─────►  │  - parse_xlsx  │ ────► document.json
  data/raw/*.html             │  - parse_pdf   │
  data/raw/*.pptx             │  - parse_html  │       data/assets/<slug>/
  data/raw/*.epub             │  - parse_pptx  │       img-*.png
  data/raw/*.txt|md|csv       │  - parse_epub  │
                              │  - VLM caption │
                              └────────────────┘

Install

The package is published on PyPI as rc-docparser; the Python import name is docparser (i.e. import docparser).

pip install rc-docparser              # core: docx + xlsx + txt/md + csv/tsv
pip install 'rc-docparser[pdf]'       # + PyMuPDF for PDFs
pip install 'rc-docparser[html]'      # + trafilatura + bs4 for HTML
pip install 'rc-docparser[pptx]'      # + python-pptx for PowerPoint
pip install 'rc-docparser[epub]'      # + EbookLib + bs4 for EPUB
pip install 'rc-docparser[vlm]'       # + requests for API VLM captions
pip install 'rc-docparser[all]'       # everything above (recommended)

Higher-fidelity / heavier features are separate opt-in extras (so the core install stays small and MIT):

pip install 'rc-docparser[tables]'       # + pdfplumber for PDF table extraction
pip install 'rc-docparser[ocr]'          # + rapidocr-onnxruntime for scanned PDFs
pip install 'rc-docparser[pymupdf4llm]'  # PyMuPDF4LLM PDF backend (AGPL/commercial)
pip install 'rc-docparser[docling]'      # IBM Docling PDF backend (MIT)
pip install 'rc-docparser[marker]'       # Datalab Marker PDF backend (GPL-3.0)
pip install 'rc-docparser[localvlm]'     # transformers/torch local captioning

rc-docparser requires Python 3.10+.

Quick start (library)

from docparser import WorkspaceLayout, run_all

layout = WorkspaceLayout.under("./project")   # data/raw, data/parsed, data/assets, .cache
layout.ensure()

run_all(layout, use_vlm=False)                # parse everything in data/raw

For a single file:

from docparser import parse_path, WorkspaceLayout

layout = WorkspaceLayout.under(".")
payload = parse_path("paper.pdf", layout)
print(payload["stats"])

Quick start (CLI)

# parse a single file
docparser parse paper.pdf --workspace ./out --no-vlm

# walk a whole directory
docparser parse-all --workspace ./project --no-vlm

# enable VLM captioning (requires OPENROUTER_API_KEY in env or .env)
export OPENROUTER_API_KEY=sk-or-v1-...
docparser parse-all --workspace ./project --max-images 50

# higher-fidelity PDF: pick a backend, OCR scanned pages, extract tables
docparser parse paper.pdf --pdf-backend docling --ocr auto --pdf-tables --no-vlm

# caption with a different provider
docparser parse-all --workspace ./project --vlm-provider openai --vlm-model gpt-4o-mini

docparser version

What gets captured

.docx

• Walks the document body in document order (paragraphs + tables + drawings).
• Preserves heading hierarchy (section_path) on every block.
• Extracts every embedded image to data/assets/<slug>/ with a stable img-<seq>-<sha10>.<ext> name.
• Detects figure/table captions (style Caption or text matching Figure 1: … / Fig. 1. / Table 1.) and associates the caption with the preceding image.
• Captures context_before and context_after for every image so the VLM has document-grounded context.

.xlsx

• Iterates every sheet, every row, every column.
• For each cell stores: address, row/col indices, value, openpyxl data_type, number_format, hyperlink, comment, and the formula (from a second pass with data_only=False).
• Stores merged_ranges, frozen_panes, and any embedded images.
• Markdown rendering uses the first non-empty row as a header heuristic and preserves multi-line cells with <br>.

.pdf

• Page-by-page text extraction in reading order via PyMuPDF's blocks API.
• Best-effort heading detection from font size (≥120% of the body-text median promotes a line to a heading; bold flag tracked).
• Embedded raster images extracted via doc.extract_image(xref).
• Pluggable backends (backend="pymupdf4llm" | "docling" | "marker") route conversion to a higher-fidelity engine; their Markdown is normalized into the same block schema. Images are still extracted via PyMuPDF.
• OCR (ocr="auto" | "force", [ocr] extra) recognizes text on scanned / low-text pages; OCR'd blocks carry "ocr": true.
• Tables (extract_tables=True, [tables] extra) emit real table blocks via pdfplumber.

.html

• Article-grade body extraction via trafilatura.
• Plus a structural BeautifulSoup walk that emits typed blocks (heading / paragraph / list / table / image) so downstream RAG layers can rely on the JSON.

.pptx

• Walks slides in presentation order; each slide becomes a section.
• Emits per-slide headings (slide title), bulleted text frames (with list level), tables, pictures, and speaker notes.
• Embedded pictures extracted and optionally captioned.

.epub

• Walks the spine in reading order; per-chapter BeautifulSoup structural walk.
• Captures metadata (title/author/language), headings, paragraphs, lists, tables, and embedded images (resolved from the EPUB image manifest).

.txt / .md and .csv / .tsv (core, no extras)

• Plain text is split into paragraph blocks; Markdown is passed through and also decomposed into heading / list / code / paragraph blocks.
• CSV/TSV: delimiter sniffing, header detection, a Markdown table, and one JSON record per row.

Images ([vlm] extra)

Each image is sent to a vision-language model (default provider OpenRouter, model anthropic/claude-sonnet-4) along with its surrounding caption + context. Any OpenAI-compatible provider works via --vlm-provider (openrouter / openai / gemini / local), or use a fully-local transformers model with --vlm-provider transformers ([localvlm] extra). The model returns a strict JSON object:

{
  "caption":           "one-sentence figure caption",
  "description":       "2–5 sentence paragraph",
  "visible_text":      "OCR-style transcription",
  "tags":              ["world-model", "diagram", "..."],
  "image_kind":        "diagram | plot | screenshot | photo | equation | table | ...",
  "domain_relevance":  "how this relates to the document's topic"
}

Results are cached on disk at <cache_dir>/vlm/<model>/<sha1>.json, keyed by SHA-1 of the image bytes × model, so re-runs are free until the source image bytes change.

Configuration (.env)

Var	Default	Purpose
DOCPARSER_VLM_PROVIDER	openrouter	openrouter / openai / gemini / local
OPENROUTER_API_KEY	required for OpenRouter	OpenRouter key (sk-or-...)
OPENROUTER_VLM_MODEL	anthropic/claude-sonnet-4	any vision-capable OpenRouter model
OPENROUTER_BASE_URL	https://openrouter.ai/api/v1	override for a proxy
OPENROUTER_REFERER / OPENROUTER_TITLE	repo URL / docparser	OpenRouter attribution headers
OPENAI_API_KEY / OPENAI_VLM_MODEL	required for OpenAI / gpt-4o-mini	OpenAI provider
GEMINI_API_KEY / GEMINI_VLM_MODEL	required for Gemini / gemini-1.5-flash	Gemini provider
DOCPARSER_VLM_BASE_URL	http://localhost:11434/v1	base URL for the local provider
DOCPARSER_VLM_API_KEY / DOCPARSER_VLM_MODEL	— / llava	key + model for the local provider
DOCPARSER_LOCAL_VLM_MODEL	Salesforce/blip-image-captioning-large	model for the transformers backend

API reference (highlights)

• WorkspaceLayout(raw_dir, parsed_dir, assets_dir, cache_dir) — dataclass describing where parser output lives. Use .under(root) for the default data/raw + data/parsed + data/assets + .cache layout under a root.
• parse_docx(source, layout=None, *, captioner=None, write_outputs=True) → payload dict.
• parse_xlsx(source, layout=None, *, captioner=None, write_outputs=True) → payload dict.
• parse_pdf(source, layout=None, *, captioner=None, write_outputs=True, extract_images=True, backend="builtin", ocr="off", extract_tables=False) → payload dict. (requires [pdf]; backends/OCR/tables require their extras)
• parse_html(source, layout=None, *, captioner=None, write_outputs=True, use_trafilatura=True) → payload dict. source may be a path or http(s):// URL. (requires [html])
• parse_pptx(source, layout=None, *, captioner=None, write_outputs=True) → payload dict. (requires [pptx])
• parse_epub(source, layout=None, *, captioner=None, write_outputs=True) → payload dict. (requires [epub])
• parse_text(source, layout=None, ...) / parse_csv(source, layout=None, ...) — core parsers for .txt/.md and .csv/.tsv.
• parse_path(source, layout=None, **kwargs) — dispatches by extension; PDF-only kwargs (backend, ocr, extract_tables) are forwarded to PDFs.
• run_all(layout, *, use_vlm=True, only=None, max_images=None, continue_on_error=False, vlm_provider=None, vlm_model=None, pdf_backend="builtin", ocr="off", extract_tables=False) — walks layout.raw_dir, parses everything supported, writes a top-level CORPUS.md and data/parsed/corpus.json.
• caption_image(image_bytes, *, mime, doc_name, nearby_caption, context, provider=None, model=None, layout=None, ...) → VLMResult. (requires [vlm])

Development

git clone https://github.com/Research-Commons/docparser
cd docparser
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all,dev]"
pytest -ra
ruff check src tests
mypy
python -m build           # produces dist/*.whl + *.tar.gz
twine check dist/*

Publishing

CI runs lint + mypy + tests on Python 3.10-3.12 and builds the distribution on every push/PR (.github/workflows/ci.yml). Pushing a version tag (e.g. v0.2.0) triggers .github/workflows/publish.yml, which builds and uploads to PyPI via Trusted Publishing (OIDC, no stored token) — configure a trusted publisher for the project on PyPI first. To publish manually instead:

python -m build
twine check dist/*
twine upload dist/*

License

MIT — see LICENSE.