Skip to main content

corpusfold

Turn the PDFs you actually have into a clean, traceable knowledge base.

Built for Chinese documents, where the hard part is almost never the embedding model — it is everything upstream of it.

from corpusfold import extract_pdf, strip_boilerplate, normalize_document, chunk_document, Index

doc = normalize_document(strip_boilerplate(extract_pdf("report.pdf")))
index = Index(chunk_document(doc))

for hit in index.search("近两月上市公司数量和平均涨幅", limit=3):
    print(hit.score, hit.label)   # report.pdf p.5
    print(hit.chunk.text[:80])

The pipeline

字符层   look-alike codepoints (⻆ vs 角)      → cjkfold ✅
抽取层   positions, tables, running headers   → ✅
版面层   分栏、阅读顺序                        → ✅
分块层   中文没有空格,标点才是语义边界        → ✅
检索层   bigram BM25 + 精确命中 + 可选向量     → ✅

corpusfold depends on cjkfold for the character layer and does not duplicate it.

Six things that decide whether this works

1. Positions, not just text

Header/footer removal, column detection, and "where did this sentence come from?" are all layout questions. None can be answered once coordinates are thrown away, so extraction captures a bounding box for every block and everything downstream keeps it.

2. A footer's page number defeats exact matching

请务必阅读正文后免责条款部分 2
请务必阅读正文后免责条款部分 3

Never equal, so naive dedup never fires — candidates are compared after digits are stripped. And the first page carries extra text, so it appears exactly once:

国海证券研究所 请务必阅读正文后免责条款部分     ← page 1
请务必阅读正文后免责条款部分 2                ← pages 2+

No frequency threshold catches that. A short candidate inside the header band that contains a known key is treated as boilerplate too — while a long paragraph repeating on every page is left alone.

3. A contents page outranks the body it points at

3、 风险提示 ............................... 8

A contents page contains exactly the words a reader would search for and none of the substance, so it wins. Dot-leader entries are dropped at chunk time. Ordinary prose ellipsis …… is unaffected.

4. A full-width element bridges the gutter

A two-column page read positionally interleaves the columns. But a title or a wide table spans the gutter — include it in column detection and the page projection is welded solid, so no gutter is ever found. Full-width blocks are excluded from detection and hoisted to the top of the page.

5. A table is not prose

Flattened into the text stream, a table's column headers drift away from its values. Tables are extracted as structured blocks, and the text inside their bounding box is dropped — otherwise the same numbers are indexed twice and a fragment can outrank the row it belongs to.

A table is atomic: splitting it mid-table scatters the headers. Only a table too large to read as one passage is sliced — and then every slice repeats the header, so a slice read alone still says what its numbers mean.

6. Chunk after finding boundaries, not before

The common mistake is to cut to 500 characters and hunt for a boundary afterwards — by then the sentence is already in half. The order is reversed:

split into sentences  →  merge up to a target size  →  never stop mid-paragraph

Chinese has no spaces, so sentence-final punctuation is the only reliable semantic boundary the text offers. Neighbouring chunks repeat their last sentence, so an answer cannot fall into the gap between two.

Retrieval

index = Index(chunks)
hit = index.search("紫色独角兽协议编号")[0]
hit.chunk.text                   # the passage
hit.label                        # 'report.pdf p.3-4' — provenance, always
hit.lexical, hit.exact, hit.dense  # components, kept separate
  • Character bigrams, not words. Chinese has no spaces and a tokenizer is a heavy dependency with failure modes of its own.
  • Lexical first, exact-match bonus on top. Dense scores are bad at literal tokens; cjkfold's character n-gram overlap puts them back.
  • Dense is optional and off by default. Pass any object with encode(texts) -> list[list[float]] as embedder=. OpenAIEmbedder talks to any OpenAI-compatible endpoint over urllib, so enabling it adds no dependency. A chunk with no literal overlap can still be retrieved on the dense score alone.

Identical passages collapse before results are returned.

Measured on a real document

A 15-page broker research report (dense Chinese, multi-column, heavy tables):

Stage Result
Extract 15 pages, 455 raw blocks, 17 tables
Structure table interiors de-duplicated → 326 blocks
Boilerplate 27 running header/footer blocks removed
Columns 5 of 15 pages detected as multi-column
Chunk 52 chunks, median 411 chars, max 775 — inside the hard maximum
Retrieve top hit for each query was body text, with page provenance

Install

pip install corpusfold

Working on it from a checkout instead:

uv venv .venv --python 3.11
uv pip install -e ".[dev]"

Command line

No code required:

corpusfold inspect 研报.pdf                      # 流水线对这份文档做了什么
corpusfold build 研报.pdf -o index.json           # 建索引
corpusfold search "近两月上市公司数量和平均涨幅" -i index.json

inspect — the fastest way to judge whether this is worth your time:

研报.pdf
  页数        15
  原始块      326
  表格        17 个
  页眉页脚    删掉 27 块
  多栏页      5 / 15
  分块        52 个,中位 411 字,最大 775 字
  预览
    [研报.pdf p.1] 2023 年01 月15 日专题研究新股研究…

search — every hit says where it came from:

1. 1.068   研报.pdf p.5-6
   资料来源:Wind,国海证券研究所…上市首日平均涨幅12.10%,上市以来平均涨幅8.31%…

Tests

.venv/bin/python -m pytest

64 tests. They generate their own PDFs and build Document objects directly — no fixture files, no Chinese font, no network.

Roadmap

  • Extraction: PDF → blocks with coordinates
  • Character cleanup (via cjkfold)
  • Running header/footer detection and removal
  • Contents pages kept out of the index
  • Multi-column detection and reading order
  • Tables as structured blocks, de-duplicated and sliced
  • Chinese-aware chunking with overlap and provenance
  • Hybrid retrieval: bigram BM25 + exact n-gram bonus + optional dense layer
  • Command line: inspect / build / search
  • Section headings as breadcrumbs on each chunk
  • OCR path for scanned pages

License

MIT

Metadata

Release files for corpusfold 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for corpusfold 0.1.1
File Size Uploaded Provenance Actions
corpusfold-0.1.1.tar.gz 32.3 kB Without provenance Details for corpusfold-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for corpusfold 0.1.1
File Interpreter ABI Platform Actions
corpusfold-0.1.1-py3-none-any.whl Python 3 none any Details for corpusfold-0.1.1-py3-none-any.whl

Total release size: 55.7 kB

Release files / corpusfold-0.1.1.tar.gz

Download URL corpusfold-0.1.1.tar.gz
Size 32.3 kB
Tags Source
SHA-256 checksum
How to use checksums
0ef3edff860468a3fc5e2cb1e959209ec4e0c57c8a2e7d5a1b8545e131993477
BLAKE2b-256 checksum
How to use checksums
7089c48dbcd4c6727dce477202e6c2fa9a99d6c21523f30d442765ac56aaeab5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / corpusfold-0.1.1-py3-none-any.whl

Download URL corpusfold-0.1.1-py3-none-any.whl
Size 23.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
843981dc3207b8b5fdcc48181d7715d138734e07515c783c0bc93f3a9496a115
BLAKE2b-256 checksum
How to use checksums
8a7ba3695f59d5804746c4a7514a1f0d583a43a0224b692375bd485a1214d9ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for corpusfold 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page