docpluck

PDF, DOCX, and HTML text extraction and normalization for academic papers.

Built from cross-project experience across 8,000+ PDFs spanning psychology, medicine, economics, physics, and biology. Achieves 100% accuracy on 29 manually verified ground-truth passages (see BENCHMARKS.md).

Supports three input formats:

PDF via pdftotext default mode (with pdfplumber SMP recovery)
DOCX via mammoth (DOCX → HTML → text, preserving Shift+Enter soft breaks)
HTML via beautifulsoup4 + lxml (block/inline-aware tree-walk)

All three formats feed into the same normalization pipeline and quality scoring.

Install

# PDF only (pdfplumber)
pip install docpluck

# + DOCX support (adds mammoth)
pip install docpluck[docx]

# + HTML support (adds beautifulsoup4 + lxml)
pip install docpluck[html]

# Everything
pip install docpluck[all]

System requirement for extract_pdf(): poppler-utils (provides the pdftotext binary). DOCX and HTML are pure Python — no system dependencies.

# Linux / WSL
apt-get install poppler-utils

# macOS
brew install poppler

# Windows
# Download from https://github.com/oschwartz10612/poppler-windows/releases
# Add bin/ to PATH

Install from GitHub (like R's remotes::install_github()):

pip install git+https://github.com/giladfeldman/docpluck.git

# Pinned version
pip install "docpluck>=1.3.0"

Quick Start

from docpluck import (
    extract_pdf, extract_docx, extract_html,
    normalize_text, NormalizationLevel, compute_quality_score,
)

# 1. Extract text from any supported format
with open("paper.pdf", "rb") as f:
    text, method = extract_pdf(f.read())

# Or from DOCX:
# with open("paper.docx", "rb") as f:
#     text, method = extract_docx(f.read())

# Or from HTML:
# with open("paper.html", "rb") as f:
#     text, method = extract_html(f.read())

print(f"Extracted {len(text):,} chars via {method}")

# 2. Normalize for statistical pattern matching
normalized, report = normalize_text(text, NormalizationLevel.academic)

print(f"Steps applied: {report.steps_applied}")
print(f"Changes made: {report.changes_made}")

# 3. Check quality
quality = compute_quality_score(normalized)
print(f"Quality: {quality['score']}/100 ({quality['confidence']})")
if quality["garbled"]:
    print("Warning: text may be corrupted (column merge or encoding failure)")

Structured extraction (v2.0)

For consumers that need tables and figures as structured data — meta-analysis tooling, statistical-claim extraction, dashboards — call extract_pdf_structured():

from docpluck import extract_pdf_structured

with open("paper.pdf", "rb") as f:
    result = extract_pdf_structured(f.read())

print(f"{result['page_count']} pages")
print(f"{len(result['tables'])} tables, {len(result['figures'])} figures")

for t in result["tables"]:
    print(f"  {t['label']} on page {t['page']} ({t['kind']}, confidence={t['confidence']})")
    if t["kind"] == "structured":
        print(f"    {t['n_rows']} rows × {t['n_cols']} cols")

Modes

# Default: caption-anchored fast path.
extract_pdf_structured(pdf_bytes)

# Thorough: scan every page for uncaptioned tables (slower).
extract_pdf_structured(pdf_bytes, thorough=True)

# Strip table/figure regions from `text` and replace with [Label: caption] markers.
extract_pdf_structured(pdf_bytes, table_text_mode="placeholder")

CLI

docpluck extract paper.pdf --structured > out.json
docpluck extract paper.pdf --structured --thorough --text-mode placeholder
docpluck extract paper.pdf --structured --html-tables-to ./out/

extract_pdf() (the v1 text-only path) is unchanged. New consumers opt in to the structured path; existing consumers see no behavioral change.

See docs/superpowers/specs/2026-05-06-table-extraction-design.md for the full schema and design rationale.

API Reference

`extract_pdf(pdf_bytes: bytes) → tuple[str, str]`

Extract text from PDF bytes.

Parameters:

pdf_bytes — Raw PDF file content as bytes

Returns: (text, method) tuple where:

text — Extracted plain text. Check text.startswith("ERROR:") for failure.
method — Engine used:
- "pdftotext_default" — standard extraction (fast, ~400ms)
- "pdftotext_default+pdfplumber_recovery" — SMP fallback triggered (~9s), used when pdftotext outputs U+FFFD replacement characters (common in Nature/Cell papers using Mathematical Italic fonts)

Requires: pdftotext binary on PATH.

with open("paper.pdf", "rb") as f:
    text, method = extract_pdf(f.read())

if text.startswith("ERROR:"):
    raise RuntimeError(f"Extraction failed: {text}")

`extract_docx(docx_bytes: bytes) → tuple[str, str]`

Extract text from DOCX (Word) file bytes via mammoth.

Parameters:

docx_bytes — Raw DOCX file content as bytes

Returns: (text, method) tuple where method is always "mammoth".

How it works: DOCX is converted to HTML first (preserving Shift+Enter soft breaks as <br> tags), then passed through the same block/inline-aware tree-walk used by extract_html(). This preserves paragraph structure, headings, lists, and soft breaks — which mammoth.extract_raw_text() would lose.

Requires: pip install docpluck[docx] (adds mammoth>=1.8.0).

Known limitations:

OMML equations (Office Math) are silently dropped. Inline stats written as plain text survive; stats inside equation objects do not.
Tracked changes: only deleted paragraphs are handled minimally.
Memory: peak usage is ~3–5× file size.

from docpluck import extract_docx

with open("paper.docx", "rb") as f:
    text, method = extract_docx(f.read())

`extract_html(html_bytes: bytes) → tuple[str, str]`

Extract text from HTML file bytes via beautifulsoup4 + lxml.

Parameters:

html_bytes — Raw HTML file content as bytes (UTF-8 decoded with error replacement)

Returns: (text, method) tuple where method is always "beautifulsoup".

How it works: Custom tree-walk that distinguishes block from inline elements:

Block elements (<p>, <div>, <h1>–<h6>, <li>, <td>, etc.) get newlines before and after.
Inline elements (<a>, <span>, <em>, etc.) get spaces before and after — critical for preventing merged words like "ChanORCID" when adjacent inline elements have no whitespace between them.
Ignored tags (<script>, <style>, <meta>, <svg>, <iframe>, etc.) are decomposed before walking.

Why not BeautifulSoup.get_text(): get_text() cannot distinguish block from inline elements — it applies a uniform separator everywhere, which either merges paragraphs or inserts spurious whitespace. The BeautifulSoup maintainer has confirmed this will not be fixed. A custom tree-walk is required.

Requires: pip install docpluck[html] (adds beautifulsoup4>=4.12.0 and lxml>=5.0.0).

from docpluck import extract_html, html_to_text

# From bytes
with open("article.html", "rb") as f:
    text, method = extract_html(f.read())

# From an already-decoded string
text = html_to_text("<p>Hello <a>world</a></p>")

`count_pages(pdf_bytes: bytes) → int`

Count pages in a PDF using byte pattern matching. No external binary required. PDF only — returns None is not applicable for DOCX/HTML.

with open("paper.pdf", "rb") as f:
    content = f.read()
    n = count_pages(content)
print(f"{n} pages")

`normalize_text(text: str, level: NormalizationLevel) → tuple[str, NormalizationReport]`

Apply the normalization pipeline at the specified level.

Parameters:

text — Raw extracted text
level — NormalizationLevel.none | NormalizationLevel.standard | NormalizationLevel.academic

Returns: (normalized_text, report) tuple.

Normalization levels:

Level	Steps	Use when
`none`	—	You want raw text, no modifications
`standard`	Core cleanup (`S*`) + document-shape cleanup (`F0/H0/T0/P0/P1/W0`) + recovery/ref joins (`R2/R3/A7`)	General text processing (NLP, search indexing)
`academic`	`standard` + statistical repairs (`A` and `W0`)	Statistical pattern matching, meta-analysis

from docpluck import normalize_text, NormalizationLevel

# Raw text
text, _ = normalize_text(raw, NormalizationLevel.none)

# General cleanup
text, report = normalize_text(raw, NormalizationLevel.standard)

# Full statistical repair (recommended for academic PDFs)
text, report = normalize_text(raw, NormalizationLevel.academic)

print(report.version)          # e.g., "1.9.35"
print(report.steps_applied)    # ["S0_smp_to_ascii", "S1_encoding_validation", ...]
print(report.changes_made)     # {"ligatures_expanded": 27, "dashes_normalized": 3, ...}

NormalizationReport fields:

Field	Type	Description
`level`	`str`	Level used: `"none"`, `"standard"`, or `"academic"`
`version`	`str`	Pipeline version (e.g. `"1.9.35"`)
`steps_applied`	`list[str]`	Step codes in order (e.g. `["S1_encoding_validation", "S3_ligature_expansion"]`)
`changes_made`	`dict[str, int]`	Character-level change counts per step

`compute_quality_score(text: str) → dict`

Compute extraction quality metrics.

Returns:

{
    "score": 85,                    # 0–100 composite score
    "common_word_ratio": 0.142,     # fraction of first 2000 words that are common English words
    "garbled": False,               # True if common_word_ratio < 0.02 (column merge / encoding failure)
    "confidence": "high",           # "high" (≥80), "medium" (≥50), "low" (<50)
    "details": {
        "ligatures_remaining": 0,   # count of ff/fi/fl ligature chars not yet expanded
        "garbled_chars": 0,         # count of U+FFFD replacement characters
        "non_ascii_ratio": 0.031,   # fraction of non-ASCII characters
    }
}

Interpreting the score:

Score	Meaning
≥ 80	High quality — proceed with analysis
50–79	Medium — check manually if precision matters
< 50	Low — likely garbled (column merge, encoding failure, or non-English)

quality = compute_quality_score(text)

if quality["garbled"]:
    print("Extraction likely failed — skipping this paper")
elif quality["score"] < 50:
    print(f"Low quality ({quality['score']}) — verify manually")

Integration Examples

ESCIcheck / effectcheck

from docpluck import extract_pdf, normalize_text, NormalizationLevel, compute_quality_score
import re

def extract_stats(pdf_path: str) -> list[dict]:
    with open(pdf_path, "rb") as f:
        text, method = extract_pdf(f.read())

    normalized, report = normalize_text(text, NormalizationLevel.academic)

    quality = compute_quality_score(normalized)
    if quality["garbled"]:
        return []  # Skip garbled papers

    # Now apply your statistical patterns to `normalized`
    # e.g. find t-tests, F-tests, correlations, p-values
    p_values = re.findall(r'p\s*[<=>]\s*\.?\d+', normalized)
    return p_values

Scimeto / MetaESCI (batch processing)

from docpluck import extract_pdf, normalize_text, NormalizationLevel, compute_quality_score
from pathlib import Path

def process_corpus(pdf_dir: str) -> list[dict]:
    results = []
    for pdf_path in Path(pdf_dir).glob("**/*.pdf"):
        with open(pdf_path, "rb") as f:
            text, method = extract_pdf(f.read())

        if text.startswith("ERROR:"):
            results.append({"file": pdf_path.name, "error": text})
            continue

        normalized, report = normalize_text(text, NormalizationLevel.academic)
        quality = compute_quality_score(normalized)

        results.append({
            "file": pdf_path.name,
            "chars": len(normalized),
            "method": method,
            "quality": quality["score"],
            "garbled": quality["garbled"],
        })

    return results

MetaMisCitations (URL-based)

import httpx
from docpluck import extract_pdf, normalize_text, NormalizationLevel

def extract_from_url(url: str) -> str:
    response = httpx.get(url, follow_redirects=True, timeout=30)
    response.raise_for_status()

    text, method = extract_pdf(response.content)
    normalized, _ = normalize_text(text, NormalizationLevel.academic)
    return normalized

What Gets Fixed

Standard normalization (`NormalizationLevel.standard`)

Artifact	Example (before → after)
Null bytes	`"Study\x00 results"` → `"Study results"`
Ligatures	`"signiﬁcant"` → `"significant"`
Unicode minus	`"r = −0.73"` → `"r = -0.73"`
Soft hyphen (invisible)	`"signifi\u00ADcant"` → `"significant"`
Non-breaking spaces	`"p\u00A0<\u00A0.001"` → `"p < .001"`
Full-width digits	`"ｐ＝０.００１"` → `"p = 0.001"`
Curly quotes	`"the "effect""` → `"the "effect""`
Hyphenation	`"signi-\nficant"` → `"significant"`
Repeated headers	Journal name repeated on every page → stripped
Page numbers	Standalone `12` on its own line → stripped

Academic normalization adds (`NormalizationLevel.academic`)

Artifact	Example (before → after)
Stat line breaks	`"p =\n.001"` → `"p = .001"`
Dropped decimals	`"p = 484"` → `"p = .484"`
European decimals	`"p = 0,05"` → `"p = 0.05"`
CI delimiters	`"[0.81; 1.92]"` → `"[0.81, 1.92]"`
Greek letters	`"η² = 0.12"` → `"eta2 = 0.12"`
Superscripts	`"r² = 0.54"` → `"r2 = 0.54"`
Footnote markers	`"p < .001¹"` → `"p < .001"`

System Requirements

Requirement	Version	Notes
Python	≥ 3.10
pdfplumber	≥ 0.11.0	Core pip dependency — installed automatically
poppler-utils	any recent	System package — for `extract_pdf()` only
mammoth	≥ 1.8.0	Optional (`[docx]`) — pure Python, no system deps
beautifulsoup4	≥ 4.12.0	Optional (`[html]`) — pure Python
lxml	≥ 5.0.0	Optional (`[html]`) — has prebuilt wheels

The normalization and quality functions (normalize_text, compute_quality_score) have no system requirements — pure Python, no external binaries. DOCX and HTML extraction are pure Python too; only PDF needs a system binary.

License

MIT. See LICENSE.

Citation

If you use docpluck in research, please cite:

Feldman, G. (2026). docpluck: PDF text extraction and normalization for academic papers.
https://github.com/giladfeldman/docpluck

Metadata

Release files for docpluck 2.4.125

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for docpluck 2.4.125
File	Size	Uploaded
docpluck-2.4.125.tar.gz	4.0 MB	Aug 5, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for docpluck 2.4.125
File	Interpreter	ABI	Platform	Reset
docpluck-2.4.125-py3-none-any.whl 406.0 kB Aug 5, 2026	Python 3	none	any	Details

Total release size: 4.4 MB

Release files / docpluck-2.4.125.tar.gz

Download URL	docpluck-2.4.125.tar.gz
Size	4.0 MB
Tags	Source
SHA-256 checksum How to use checksums	`89234479653326250896b9050c173515a1a220ca16309f772313e30975e8e59d`
BLAKE2b-256 checksum How to use checksums	`ddf79c81f5c35b1be23e6b7ac504b65ecd15a9df4cb2cabb23d033d8ee4c0eef`
Upload date	Aug 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/giladfeldman/docpluck
Publishing commit: github.com/giladfeldman/docpluck/tree/44107eab217dc60d52150b46ca8cfc3be2af8790

Workflow

Publishing configuration: github.com/giladfeldman/docpluck/blob/44107eab217dc60d52150b46ca8cfc3be2af8790/.github/workflows/publish.yml
Publishing logs: github.com/giladfeldman/docpluck/actions/runs/30998723693/attempts/1

Artifact

Subject: docpluck-2.4.125.tar.gz
SHA-256 checksum: 89234479653326250896b9050c173515a1a220ca16309f772313e30975e8e59d Verifying attestations

Release files / docpluck-2.4.125-py3-none-any.whl

Download URL	docpluck-2.4.125-py3-none-any.whl
Size	406.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`bfce96a93615df0ad0519d3e9dcdc58d2322e35cc365ffc0545f562b5b07bdb2`
BLAKE2b-256 checksum How to use checksums	`1efb5146ff64f69c3d705a48ba4a256eb3a66cb73033058d40048dbd567c0ee9`
Upload date	Aug 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/giladfeldman/docpluck
Publishing commit: github.com/giladfeldman/docpluck/tree/44107eab217dc60d52150b46ca8cfc3be2af8790

Workflow

Publishing configuration: github.com/giladfeldman/docpluck/blob/44107eab217dc60d52150b46ca8cfc3be2af8790/.github/workflows/publish.yml
Publishing logs: github.com/giladfeldman/docpluck/actions/runs/30998723693/attempts/1

Artifact

Subject: docpluck-2.4.125-py3-none-any.whl
SHA-256 checksum: bfce96a93615df0ad0519d3e9dcdc58d2322e35cc365ffc0545f562b5b07bdb2 Verifying attestations