corpusfold
Turn the PDFs you actually have into a clean, traceable knowledge base.
Built for Chinese documents, where the hard part is almost never the embedding model — it is everything upstream of it.
from corpusfold import extract_pdf, strip_boilerplate, normalize_document, chunk_document, Index
doc = normalize_document(strip_boilerplate(extract_pdf("report.pdf")))
index = Index(chunk_document(doc))
for hit in index.search("近两月上市公司数量和平均涨幅", limit=3):
print(hit.score, hit.label) # report.pdf p.5
print(hit.chunk.text[:80])
The pipeline
字符层 look-alike codepoints (⻆ vs 角) → cjkfold ✅
抽取层 positions, tables, running headers → ✅
版面层 分栏、阅读顺序 → ✅
分块层 中文没有空格,标点才是语义边界 → ✅
检索层 bigram BM25 + 精确命中 + 可选向量 → ✅
corpusfold depends on cjkfold for the
character layer and does not duplicate it.
Six things that decide whether this works
1. Positions, not just text
Header/footer removal, column detection, and "where did this sentence come from?" are all layout questions. None can be answered once coordinates are thrown away, so extraction captures a bounding box for every block and everything downstream keeps it.
2. A footer's page number defeats exact matching
请务必阅读正文后免责条款部分 2
请务必阅读正文后免责条款部分 3
Never equal, so naive dedup never fires — candidates are compared after digits are stripped. And the first page carries extra text, so it appears exactly once:
国海证券研究所 请务必阅读正文后免责条款部分 ← page 1
请务必阅读正文后免责条款部分 2 ← pages 2+
No frequency threshold catches that. A short candidate inside the header band that contains a known key is treated as boilerplate too — while a long paragraph repeating on every page is left alone.
3. A contents page outranks the body it points at
3、 风险提示 ............................... 8
A contents page contains exactly the words a reader would search for and none
of the substance, so it wins. Dot-leader entries are dropped at chunk time.
Ordinary prose ellipsis …… is unaffected.
4. A full-width element bridges the gutter
A two-column page read positionally interleaves the columns. But a title or a wide table spans the gutter — include it in column detection and the page projection is welded solid, so no gutter is ever found. Full-width blocks are excluded from detection and hoisted to the top of the page.
5. A table is not prose
Flattened into the text stream, a table's column headers drift away from its values. Tables are extracted as structured blocks, and the text inside their bounding box is dropped — otherwise the same numbers are indexed twice and a fragment can outrank the row it belongs to.
A table is atomic: splitting it mid-table scatters the headers. Only a table too large to read as one passage is sliced — and then every slice repeats the header, so a slice read alone still says what its numbers mean.
6. Chunk after finding boundaries, not before
The common mistake is to cut to 500 characters and hunt for a boundary afterwards — by then the sentence is already in half. The order is reversed:
split into sentences → merge up to a target size → never stop mid-paragraph
Chinese has no spaces, so sentence-final punctuation is the only reliable semantic boundary the text offers. Neighbouring chunks repeat their last sentence, so an answer cannot fall into the gap between two.
Retrieval
index = Index(chunks)
hit = index.search("紫色独角兽协议编号")[0]
hit.chunk.text # the passage
hit.label # 'report.pdf p.3-4' — provenance, always
hit.lexical, hit.exact, hit.dense # components, kept separate
- Character bigrams, not words. Chinese has no spaces and a tokenizer is a heavy dependency with failure modes of its own.
- Lexical first, exact-match bonus on top. Dense scores are bad at literal
tokens;
cjkfold's character n-gram overlap puts them back. - Dense is optional and off by default. Pass any object with
encode(texts) -> list[list[float]]asembedder=.OpenAIEmbeddertalks to any OpenAI-compatible endpoint over urllib, so enabling it adds no dependency. A chunk with no literal overlap can still be retrieved on the dense score alone.
Identical passages collapse before results are returned.
Measured on a real document
A 15-page broker research report (dense Chinese, multi-column, heavy tables):
| Stage | Result |
|---|---|
| Extract | 15 pages, 455 raw blocks, 17 tables |
| Structure | table interiors de-duplicated → 326 blocks |
| Boilerplate | 27 running header/footer blocks removed |
| Columns | 5 of 15 pages detected as multi-column |
| Chunk | 52 chunks, median 411 chars, max 775 — inside the hard maximum |
| Retrieve | top hit for each query was body text, with page provenance |
Install
pip install corpusfold
Working on it from a checkout instead:
uv venv .venv --python 3.11
uv pip install -e ".[dev]"
Command line
No code required:
corpusfold inspect 研报.pdf # 流水线对这份文档做了什么
corpusfold build 研报.pdf -o index.json # 建索引
corpusfold search "近两月上市公司数量和平均涨幅" -i index.json
inspect — the fastest way to judge whether this is worth your time:
研报.pdf
页数 15
原始块 326
表格 17 个
页眉页脚 删掉 27 块
多栏页 5 / 15
分块 52 个,中位 411 字,最大 775 字
预览
[研报.pdf p.1] 2023 年01 月15 日专题研究新股研究…
search — every hit says where it came from:
1. 1.068 研报.pdf p.5-6
资料来源:Wind,国海证券研究所…上市首日平均涨幅12.10%,上市以来平均涨幅8.31%…
Tests
.venv/bin/python -m pytest
64 tests. They generate their own PDFs and build Document objects directly —
no fixture files, no Chinese font, no network.
Roadmap
- Extraction: PDF → blocks with coordinates
- Character cleanup (via
cjkfold) - Running header/footer detection and removal
- Contents pages kept out of the index
- Multi-column detection and reading order
- Tables as structured blocks, de-duplicated and sliced
- Chinese-aware chunking with overlap and provenance
- Hybrid retrieval: bigram BM25 + exact n-gram bonus + optional dense layer
- Command line:
inspect/build/search - Section headings as breadcrumbs on each chunk
- OCR path for scanned pages
License
MIT
Metadata
Release files for corpusfold 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| corpusfold-0.1.1.tar.gz | 32.3 kB | Without provenance | Details for corpusfold-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| corpusfold-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for corpusfold-0.1.1-py3-none-any.whl |
Total release size: 55.7 kB
Release files / corpusfold-0.1.1.tar.gz
| Download URL | corpusfold-0.1.1.tar.gz |
|---|---|
| Size | 32.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0ef3edff860468a3fc5e2cb1e959209ec4e0c57c8a2e7d5a1b8545e131993477
|
|
BLAKE2b-256 checksum How to use checksums |
7089c48dbcd4c6727dce477202e6c2fa9a99d6c21523f30d442765ac56aaeab5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / corpusfold-0.1.1-py3-none-any.whl
| Download URL | corpusfold-0.1.1-py3-none-any.whl |
|---|---|
| Size | 23.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
843981dc3207b8b5fdcc48181d7715d138734e07515c783c0bc93f3a9496a115
|
|
BLAKE2b-256 checksum How to use checksums |
8a7ba3695f59d5804746c4a7514a1f0d583a43a0224b692375bd485a1214d9ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for corpusfold 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.