ar2md

Save a versioned arXiv paper as Markdown, local images, and the original PDF. Markdown comes from native arXiv HTML, with ar5iv fallback after an HTTP 404.

Requirements

• GNU Wget
• Pandoc 3.x for Markdown conversion; PDF downloading works without it

Set WGET or PANDOC to an executable path if needed.

Usage

Run without installing:

uvx ar2md 1706.03762v7

Or install the command:

uv tool install ar2md
ar2md 1706.03762v7

IDs must include a version. Versioned arXiv abs, html, and pdf URLs are also accepted; 1706.03762.v7 is normalized to 1706.03762v7.

ar2md https://arxiv.org/abs/1706.03762v7 --output-dir refs
ar2md 1706.03762v7 --markdown-only  # skip the PDF download

To run an unreleased commit:

uvx --from git+https://github.com/cs-giung/ar2md ar2md 1706.03762v7

Output

The default output directory is papers/:

papers/
├── 1706.03762v7/       # local images, only when present
├── 1706.03762v7.md     # HTML-derived Markdown
├── 1706.03762v7.pdf    # original arXiv PDF
└── index.jsonl        # metadata, paths, warnings, and acquisition errors

Outputs are regular files, not symlinks. The HTML mirror and temporary resources are removed afterward. Image paths are grouped under the versioned ID, including ar5iv paths that originally omit the version. PDFs always come from https://arxiv.org/pdf/<versioned-id> and must have a PDF header.

index.jsonl has one record per versioned ID, sorted by ID. Paths are relative to the output directory; source identifies the HTML endpoint, and html is null because HTML is not retained:

{"id": "1706.03762v7", "title": "Attention Is All You Need", "source": "https://arxiv.org/html/1706.03762v7", "html": null, "markdown": "1706.03762v7.md", "pdf": "1706.03762v7.pdf", "warnings": [], "errors": {}}

When migrating from pre-0.2 mirror output, use a new output directory. Existing symlinked output files and asset directories are refused; old mirrors are not deleted.

Independent acquisition and refreshes

The PDF is attempted first, then Markdown. Failure of one does not suppress the other attempt, including when Pandoc is missing:

Unavailable formats have null paths on a first attempt; title and source may also be null. On refresh, failed or skipped acquisitions retain their existing files and indexed paths. A failed Markdown refresh also retains its previous title, source, and warnings. errors records the latest failures by format. --markdown-only neither downloads nor deletes a PDF.

Successful results and the index are staged before publication. If publication fails, ar2md attempts to restore every replaced file. Failed or interrupted rollback retains recoverable backups under .ar2md-backup-* in the output directory; ordinary rollback errors report the location on stderr. Backups preserve relative filenames. Stop other writes and keep these backups until recovery is complete.

An invalid existing index or unsafe destination can prevent all publication. This is not crash recovery or support for concurrent writers. Unreferenced old assets are not deleted.

Saved paths go to stdout; errors and quality warnings go to stderr. Exit code 0 means every requested acquisition succeeded; 1 means an acquisition or publication failed, even if a partial result was saved. Quality warnings alone do not fail the command.

Conversion and limits

The Lua filter selects the paper body, removes arXiv UI, preserves math, figures, captions, and internal anchors, and reconstructs supported algorithms, footnotes, and equation layouts. Image objects are prepared for Pandoc without rewriting unrelated HTML. SVG assets remain external files.

Image discovery and path rewriting use Pandoc's syntax tree, including reference-style images. Code examples and escaped image syntax remain literal; HTML anchors remain HTML. No additional Markdown parser dependency is needed.

Checks flag unresolved citations and internal links, nested math delimiters, missing local images, and selected source graphics omitted from Markdown. Counts are printed and saved in warnings. These checks do not prove semantic completeness or equivalence to the PDF. There is no PDF text extraction, OCR, TeX fallback, or content fabrication. Network errors other than the native-HTML 404 do not trigger ar5iv fallback; upstream HTML can change even for a fixed paper version.

Development

uv sync --locked
uv run python -m unittest discover -s tests -v
uvx --from . ar2md 1706.03762v7

Pandoc-dependent tests are skipped when it is unavailable. PDF acquisition, CLI error handling, and publication tests still run. CI installs Pandoc explicitly. Saved artifacts and comparison methodology are documented in the comparison notes.

Publishing

Validate and rebuild from the current checkout, clearing stale distributions:

uv lock --check
uv run python -m unittest discover -s tests -v
uv build --clear
uvx twine check --strict dist/*

Publishing uses PyPI Trusted Publishing. Configure the ar2md project for GitHub repository cs-giung/ar2md, workflow publish.yml, and environment pypi. Create a GitHub release with a tag matching v<project.version> (v0.2.0 for this version). The workflow verifies the tag, runs tests, builds and validates distributions, then publishes without a long-lived API token.

License

MIT