zh-chapter-splitter

Zero-dependency, deterministic chapter splitter for Chinese web-novel .txt files — encoding detection, rule-fusion heading recognition, and a rich quality report. No LLM, no network, pure standard library. Battle-tested on a 4.85M-character web novel.

• No dependencies. Pure standard library (re, dataclasses, pathlib, statistics).
• Deterministic. The same bytes always split the same way — no model, no randomness.
• Encoding-aware. Auto-detects UTF-8, UTF-8-BOM, UTF-16 (LE/BE), GB18030, GBK, Big5.
• Rule fusion. Preset heading rules plus one self-derived rule compete by a 5-dimension structural score; the best split wins and over-long blocks are re-scanned for missed headings.
• Honest reports. Every split returns a diagnostics dict: confidence, problem category, per-chapter anomalies, author-note flags, weird-title flags, and ordinal gaps — nothing is silently deleted.

Install

pip install zh-chapter-splitter

Requires Python 3.10+.

Quickstart

from zh_chapter_splitter import split_chapters_with_report

text = open("novel.txt", encoding="utf-8").read()
chapters, report = split_chapters_with_report(text)

for ch in chapters[:3]:
    print(ch["chapter_number"], ch["title"], len(ch["content"]))

print(report["mode"], report["confidence"], report["problem_label"])

Split straight from a file (encoding auto-detected):

from zh_chapter_splitter import split_file

chapters, report, text = split_file("novel.txt")
print(report["encoding"])  # e.g. "gb18030"

Just detect the encoding, or just clean the text:

from zh_chapter_splitter import decode_bytes, clean_text

text, encoding = decode_bytes(open("novel.txt", "rb").read())
cleaned = clean_text(text)  # strip pirate ads / mojibake, normalize whitespace

Each chapter is a plain dict:

{
    "title": str,                    # heading line, e.g. "第12章 归途"
    "content": str,                  # chapter body
    "chapter_number": int,           # 1-based, contiguous
    "volume_title": str,             # owning volume heading, or ""
    "is_author_note": bool,          # author's aside (leave note / afterword)
    "exclude_from_extraction": bool, # == is_author_note
    "title_confidence": float,       # 0..1; low means a meme/quip title
    "content_descriptor": str,       # first-sentence descriptor for low-confidence titles
}

API

Lower-level building blocks are also exported: sanitize_corpus, sanitize_corpus_text, filter_non_content, annotate_weird_titles, adaptive_split, structural_score, split_by_heading_regex, extract_seq, build_candidate_rules, and the ChapterSplitter class (with its chapter_splitter singleton).

Split rules

split_rule defaults to "auto" (rule fusion). You can also force a preset: chapter_cn, corpus, chapter_en, number_dot, paren_num, or custom (with custom_pattern, where * stands for a chapter number, e.g. 卷一-第*章).

Command line

Installs a zh-chapter-split console script (pure argparse):

# Split into one .txt per chapter plus a report.json.
zh-chapter-split novel.txt -o out/

# Print the chapter-title list only, write nothing.
zh-chapter-split novel.txt --preview

# Force a preset rule or a custom template.
zh-chapter-split novel.txt -o out/ --split-rule chapter_cn
zh-chapter-split novel.txt -o out/ --split-rule custom --custom-pattern "卷一-第*章"

How it works

1. Decode & clean. Detect the encoding (BOM-aware), normalize newlines and full-width spaces, then delete whole advertising lines, strip inline promo fragments, and drop garbled (mojibake) lines — all deterministically.
2. Split. In auto mode, rule-fusion adaptive splitting competes against a set of built-in detectors (numbered sections, pagination headings, strong / weak headings, fixed-window fallback). Candidates are ranked by a 5-dimension structural score (ordinal continuity, size uniformity, count sanity, marker consistency, coverage) and the winner is chosen.
3. Post-process & annotate. Drop empty ghost chapters, window over-long blocks, dedupe exact repeats, then flag author notes and weird titles and reconcile ordinal gaps.
4. Report. Return the chapters plus a diagnostics dict for human review.

License

MIT — see LICENSE.

Acknowledgements

Extracted from the RPG Roleplay platform (https://github.com/felixchaos/rpg-roleplay-platform). 中文说明见 README.zh-CN.md.