langchain-arxiv-retriever

A LangChain retriever for the arXiv API.

pip install langchain-arxiv-retriever

from langchain_arxiv import ArxivRetriever

retriever = ArxivRetriever()
docs = retriever.invoke('ti:"attention is all you need"')

One Document per paper, in arXiv's relevance order. Three knobs, one metadata schema, typed errors, nothing silent.

Documentation: https://j1c.github.io/langchain-arxiv-retriever/

Usage

Abstracts (default)

page_content is the paper's abstract — fast, no downloads:

retriever = ArxivRetriever()          # k=3 papers by default
docs = retriever.invoke("grokking transformers", k=10)   # per-call override

Full text

page_content is the whole paper. Text comes from arXiv's native HTML rendering when available and from the PDF otherwise; metadata["source_format"] tells you which ("html" or "pdf"):

full = ArxivRetriever(full_text=True, k=2)
docs = full.invoke("2404.19756")
docs[0].metadata["content_length"]   # 140369 — never silently cut

Need both modes? Instances are stateless and cheap — make two. For runtime switching inside a chain, use LangChain's standard configurable_fields mechanism on full_text.

Fetch by arXiv ID

A query consisting only of arXiv identifiers is fetched directly instead of searched:

retriever.invoke("1706.03762")            # new-style ID
retriever.invoke("quant-ph/9508027")      # old-style ID
retriever.invoke("2305.05665 2404.19756") # several at once

Query semantics

Queries go to the arXiv API verbatim. Everything in the arXiv API manual works: ti:, abs:, au:, cat:, all:, AND/OR/ANDNOT, quoted phrases.

Be aware that arXiv search is lexical, not semantic. Keyword-style queries retrieve well; conversational questions don't.

Metadata schema

Identical keys in both modes, snake_case, JSON-serializable:

Why not langchain-community's ArxivRetriever?

This package exists because langchain-community has been deprecated and is no longer maintained, and uses deprecated functions from arxiv-py. Lastly, the community implementation has accumulated footguns: