arxiv-sectionizer

Extract semantic sections — Future Work, Limitations, Conclusion, Discussion, etc. — from arXiv LaTeX source archives. Zero runtime dependencies.

The arxiv package hands you metadata and a PDF URL. This library hands you the sections you actually want, pulled from LaTeX source with \input{} / \include{} / \subfile{} includes resolved.

Install

pip install arxiv-sectionizer

Quickstart

from arxiv_sectionizer import sectionize

# Accepts a tar.gz path, raw bytes, a .gz, or a .tex string.
for s in sectionize("2401.01234.tar.gz", kinds="future_work"):
    print(s.title)
    print(s.body[:300])

Want every section? Omit kinds:

sections = sectionize("paper.tar.gz")
for s in sections:
    print(s.kind or "-", s.level, s.title)

Lower-level API

from arxiv_sectionizer import read_archive, flatten, extract_sections, find_sections

files = read_archive("paper.tar.gz")          # dict[str, str] of every .tex member
tex = flatten(files)                          # resolve \input{}/\include{}/\subfile{}
sections = extract_sections(tex)              # all sections, with .kind classification
future = find_sections(tex, "future_work")    # only matching categories

Each Section is:

Section(
    title: str,
    body: str,
    level: int,          # 0=chapter, 1=section, 2=subsection, ...
    kind: str | None,    # "future_work" | "limitations" | "conclusion" | ...
    start: int,          # byte offsets into the flattened source
    end: int,
)

Custom categories

The default category map covers the usual targets. Override it with your own keyword map:

from arxiv_sectionizer import Categories, extract_sections

cats = Categories(mapping={
    "open_problems": ("open problem", "unsolved"),
    "ablation":      ("ablation",),
})
sections = extract_sections(tex, categories=cats, only_classified=True)

Default categories: future_work, limitations, conclusion, discussion, related_work, introduction, abstract, methods, results.

CLI

arxiv-sectionize paper.tar.gz --kind future_work,limitations
arxiv-sectionize paper.tar.gz --titles-only
arxiv-sectionize paper.tar.gz --json > sections.json

Why

Every paper-mining pipeline re-implements "give me this section from this paper" over LaTeX, and every implementation re-discovers the same edge cases:

• nested \subsection / \paragraph
• starred forms (\section*{...})
• titles with inline LaTeX commands (\section{Results on \textsc{LibriTTS}})
• split-file papers using \input{conclusion}
• comment-outed sections (% \section{Old})
• PDF-only submissions with no .tex source

This library handles those once.

License

MIT