paper-digest 0.1.0

AI-optimized arXiv paper reader. Converts academic papers into LLM-legible Markdown with structural annotations.

paper-digest

Convert arXiv papers to AI-optimized Markdown. Math-perfect, section-tagged, noise-free.

pip install git+https://github.com/jingxuan/paper-digest.git
paper-digest 2412.15185

Install

pip install git+https://github.com/jingxuan/paper-digest.git

Or install from local source:

git clone https://github.com/jingxuan/paper-digest.git
cd paper-digest
pip install -e .

Why

Feeding a PDF to Claude/GPT costs 30k+ tokens per paper. Most of it is formatting noise. If you're screening 100 papers, that's 3M tokens wasted.

paper-digest converts arXiv papers to clean Markdown with:

• LaTeX math preserved as-is — $\mathcal{L} = \sum_i \ell_i$, not garbled MathML
• Section tags — ## [METHOD] 3.1 Our Approach so your AI can skip to what matters
• Noise removed — no headers/footers, acknowledgements, or page numbers

A 30k-token PDF becomes ~10k chars of structured Markdown. Use --sections abstract,method to cut it to ~3k.

Usage

CLI

# Full paper
paper-digest 2412.15185

# Just abstract + method (saves ~74%)
paper-digest 2412.15185 --sections abstract,method

# JSON output
paper-digest 2412.15185 --format json

# Batch processing
paper-digest --batch ids.txt --output ./papers/

# Keep everything (don't trim acknowledgements/appendix)
paper-digest 2412.15185 --raw

# Prefer HTML path (faster, but math may have artifacts)
paper-digest 2412.15185 --prefer-html

Python API

from paper_digest import PaperDigest

pd = PaperDigest()

paper = pd.fetch("2412.15185")
print(paper.markdown)            # Full AI-optimized Markdown
print(paper.section("method"))   # Just the method section
print(paper.metadata)            # Title, authors, date, section tags

# Filter sections
paper = pd.fetch("2412.15185", sections=["abstract", "method"])

# Batch
results = pd.fetch_batch(["2412.15185", "2302.13971"])

Input formats

All of these work:

2412.15185
2412.15185v2
hep-ph/0301012
https://arxiv.org/abs/2412.15185
https://arxiv.org/pdf/2412.15185.pdf

How it works

arXiv ID → LaTeX source (default) or HTML → parse → tag sections → trim noise → Markdown

Two conversion paths, automatic fallback:

Path	Coverage	Math quality	Speed
LaTeX (default)	90%+ arXiv papers	Perfect (native LaTeX)	~3s
HTML (fallback)	2024+ papers	Good (occasional MathML artifacts)	~2s

Default is LaTeX-first because math formulas are kept verbatim from the source — no conversion, no loss. Use --prefer-html if you want speed over math fidelity.

Output format

---
arxiv_id: "2412.15185"
title: "Tiled Diffusion"
authors: ["Or Madar", "Ohad Fried"]
date: "2024-12-19"
sections: ["ABSTRACT", "INTRODUCTION", "METHOD", "EXPERIMENTS", "CONCLUSION", "REFERENCES"]
---

## [ABSTRACT] Abstract

Image tiling — the seamless connection of disparate images...

## [METHOD] 3 Method

Given a set of desired output images $\mathcal{I}=\{I_1, I_2, ..., I_N\}$...

Section tags

Tag	Matches
ABSTRACT	Abstract
INTRODUCTION	Introduction, Background, Motivation
RELATED_WORK	Related Work, Prior Work, Literature Review
METHOD	Method, Methodology, Our Approach, Model Architecture, ...
EXPERIMENTS	Experiments, Experimental Setup, Evaluation, ...
RESULTS	Results, Main Results, Analysis, Discussion
CONCLUSION	Conclusion, Summary, Future Work
REFERENCES	References, Bibliography
OTHER	Anything else (original title preserved)

Motivation

Transforming human knowledge from human-first and human-legible to LLM-first and LLM-legible is a beautiful space with so much potential... This would be a significantly richer source of legible, workable information for an LLM than just something like pdf-to-text (current prevailing practice), which simply asks the LLM to predict the textbook content top to bottom token by token (umm - lame).

— Andrej Karpathy, 2025-08-28

License

MIT