Critical Reads

Some papers mistake their own story for evidence. Some are optimized for acceptance rather than truth. Others are careful, competent work on directions that do not matter. Critical Reads exists to tell them apart, preserve the insights that survive scrutiny, and choose research directions worth pursuing.

Body-read reviews of papers, posts, and talks. Each entry lives in entries/<citation-key>.md and carries its identity in frontmatter.

• Collections
• CLI
  • DBLP index
  • Configuration
• Adding a review
  • Getting the source in front of you
• Review guide
  • For robotics papers

Collections

Curated, ordered lists live in collections.json, the single source used by the viewer and validation. Every collection has a published or draft status. Published collections are stable reference catalogs; draft collections are provisional reading paths shown in a separate collapsed section. Select one to filter entries in its declared order.

CI publishes the generated viewer after GitHub Pages is enabled with GitHub Actions as its source.

CLI

Run make serve for the local viewer. Entry links open rendered HTML; .md URLs expose UTF-8 Markdown source. Install the CLI from PyPI, then authenticate with GitHub for review syncing and DBLP index downloads:

uv tool install paperstack-cli
gh auth login

The command groups have separate data and authority boundaries:

Review commands:

paperstack review show black2024pi0 --brief
paperstack review show arxiv:2410.24164 --json
paperstack review list --quality poor --tag vla
paperstack review search "flow matching"
paperstack review sync --force
paperstack review init <key> --id arxiv:NNNN.NNNNN --title "Verbatim title" --editor <name>
paperstack review check --style
paperstack review audit
paperstack review citations --fetch

Paper commands:

paperstack paper search "Attention Is All You Need" --source dblp
paperstack paper metadata arxiv:2106.09685
paperstack paper metadata arxiv:2410.15549 --source semantic_scholar --json
paperstack paper metadata doi:10.1109/CVPR.2016.90 --source crossref
paperstack paper read arxiv:2604.23073
paperstack paper read arxiv:2604.23073 --outline
paperstack paper read arxiv:2604.23073 --section 6
paperstack paper pdf arxiv:2602.09017

metadata accepts arxiv:, doi:, dblp:, and openreview: references. Semantic Scholar records include total, influential, and reference counts. read and pdf require an arxiv: reference. Metadata output keeps source records separate and includes provenance; it never selects or generates a citation.

Options are scoped to commands that use them. --json is available for structured records and --offline for commands with a local or cached path. PDF conversion requires the optional dependency:

uv tool install 'paperstack-cli[pdf]'

From a clone, replace paperstack with uv run paperstack. For PDF conversion in a clone, use uv run --extra pdf paperstack paper pdf <arxiv-ref>. review init requires a writable review tree; review check requires a clone containing scripts/build/check.sh. review citations --fetch updates citations.json for every arXiv-backed entry through Semantic Scholar's batch API; without --fetch, it only removes cached records for entries that no longer exist. The static viewer exposes a minimum-citation filter and shows counts alongside entries. SEMANTIC_SCHOLAR_API_KEY is optional but raises the API rate limit.

DBLP index

The optional index accelerates DBLP search and is required by review audit. It covers selected CS venues, not all of DBLP. The 2026.08 Parquet snapshot contains 285,521 structured records and is about 25 MiB. Installation is explicit:

paperstack index dblp status
paperstack index dblp install
paperstack index dblp update
paperstack index dblp remove --yes

install downloads the pinned Parquet snapshot through authenticated gh. update discovers the newest dblp-index-YYYY.MM paperstack Release and replaces an older index only after its SHA-256, schema, and embedded metadata checks pass. status reports the version, coverage, size, record count, and location. review audit reports title and venue matches without editing reviews. Snapshots with mismatched metadata or fewer than 250,000 records are rejected. Installed files are immutable and content-addressed; an atomic pointer switch preserves the previous index if an update is interrupted. See DBLP snapshot releases for the publishing procedure.

Configuration

Configuration comes from the process environment. The CLI also loads the nearest .env without overriding exported variables, so a clone can keep local configuration in a gitignored .env file.

Review lookup reads $PAPERSTACK_DIR, a surrounding clone, or a GitHub-backed cache, in that order. Scoped --offline flags serve cached review or paper data without network access. Exit codes are 0 for hits, 1 for no match, 2 for ambiguity, and 3 for unavailable data.

Adding a review

paperstack review init <key> --id arxiv:NNNN.NNNNN --title "Verbatim title" --editor <name>

This initializes an ungraded scaffold only. The review itself remains a reading and judgment task. Review files are stored internally as entries/<key>.md. See the review guide for editorial guidance.

• Name files <first-author surname><arXiv v1 year><first significant title word>, lowercase; suffix collisions with a, b, and so on.
• Use the established method name or full title as the # heading.
• Use a registered CURIE (arxiv:, doi:, hdl:, isbn:) for id, or a URL when none exists.
• tags are lowercase and singular. Reuse before inventing.
• Include only verified affiliations in lab. Use a person's name in editor, or model effort (harness) for an agent.
• Use CommonMark with GFM, including tables for tabular results.

Run make check; make style adds prose-length warnings. The scripts require jq and mikefarah's yq (go-yq on conda-forge).

Getting the source in front of you

The source fetcher uses latexpand from PATH, falling back to a vendored copy via Perl.

paperstack paper read arxiv:2604.23073            # the complete LaTeX body
paperstack paper read arxiv:2604.23073 --outline  # the section outline
paperstack paper read arxiv:2604.23073 --section 6
paperstack paper pdf arxiv:cs/9301101            # when there is no LaTeX source
yt-dlp --skip-download --write-auto-subs --sub-lang en -o talk <url>

• Prefer the LaTeX source; use the PDF fallback only when no source exists.
• Cross-check malformed tables with pdftotext -layout <pdf> -.
• Treat commented-out results as evidence only when their surviving values match the published version; a mismatched baseline may be an earlier run.
• Strip timestamps and duplicate cues from video captions.

Review guide

Read the body, then write the critical read, paper summary, reason to read, and one-liner, in that order.

• Keep the review body within 2,500 visible non-whitespace characters. Use prose for argument, bullets for independent points, and tables for repeated comparisons.
• Make the summary self-contained: a reader who has not read the paper should understand its core problem, approach, evidence, and findings. Choose the form that reads best; five bullets is one option, not a target.
• In the critical read, consult prior and subsequent work as needed, then focus on what matters for interpreting, trusting, or using the paper.
• Keep the one-liner to one sentence and Why read it to two. Why read it captures significance, originality, or practical value; use none when there is no reason.
• Quality measures how much of the title and abstract's main claim survives the evidence:
  • excellent: the claim stands and has lasting importance
  • good: the claim stands
  • fair: only a narrower claim stands
  • poor: the claim is not established
• Grade the advertised claim, not the narrower verdict. Use fair only when narrowing scope preserves the core claim; materially replacing it is poor.
• For SOTA or efficiency claims, check the strongest comparable result and name the denominator.
• If an official protocol fits the claim, an unjustified custom replacement caps Quality at fair unless matched, interpretable anchors restore comparability.
• Disclosed weaknesses still count when the paper claims past them. Side contributions do not raise the grade.
• On fair or poor, add Read it anyway. only with a checked citation count from Hugging Face or Semantic Scholar.

For robotics papers

See What Are We Actually Benchmarking in Robot Manipulation?.

• A benchmark counts only when success requires the claimed capability. LIBERO-only evidence caps Quality at poor; so does an uncounted real-world result added to it.
• Judge the hardest benchmark, note omissions, and account for benchmark age and test-set proximity.
• Treat margins within evaluation noise as ties; check SOTA claims against the VLA Evaluation Harness.
• Recover exact values and trial counts where possible; otherwise state that they are unavailable.
• Compare baselines only under the same training and evaluation protocol.