arxiv-retriever 1.2.3

"Automate your ArXiv paper search, retrieval, and summarization process."

Description

arxiv_retriever is a lightweight command-line tool designed to automate the retrieval of computer science papers from ArXiv. The retrieval can be done using specified ArXiv computer science archive categories, full or partial titles of papers, if available, or links to the papers. Paper retrieval can be refined by author.

NOTE: My tests indicate that when searching for a really long title, using the partial title and then refining by author yields better results, as opposed to searching with the full title or even searching with the full title and refining by author. However, the tests are not exhaustive.

This tool is built using Python and leverages the Typer library for the command-line interface and the Python ElementTree XML package for parsing XML responses from the arXiv API. It can be useful for researchers, engineers, or students who want to quickly retrieve an ArXiv paper or keep abreast of latest research in their field without leaving their terminal/workstation.

Although my current focus while building arxiv_retriever is the computer science archive, it can be easily used with categories from other areas on arxiv, e.g., math.CO.

Features

• Fetches the most recent papers from ArXiv by specified categories
• Search for papers on ArXiv using full or partial title
• Refine fetch and search by author for more precise results
• View paper details including title, authors, abstract, publication date, and links to paper's abstract and pdf pages
• Download papers after they are retrieved using fetch or search, or directly using download
• Easy-to-use command-line interface built with Typer
• Configurable number of results to fetch
• Built using only the standard library and tried and tested packages.

Environment Setup

You can optionally set an environment variable (an OpenAI API key) before using the program. This is used to authenticate with OpenAI for the paper summarization feature. If you do not want your papers summarized, you will not need to set the environment variable. Specify your choice when asked by the CLI. Specifying 'y' without the KEY set will lead to an error.

Optional Environment Variable

• Variable Name: OPENAI_API_KEY

Setting the Environment Variable

On Unix-like systems (Linux, macOS)

In your terminal, run:

export OPENAI_API_KEY=<key>

To ensure this works across all shell instances, add the above line to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc, or ~/.profile).

On Windows

1. Open the Start menu and search for "Environment Variables"
2. Click on the "Edit system environment variables" option.
3. In the System Properties window, click on the "Environment Variables" button
4. Under "User variables", click "New"
5. Set the variable name as OPENAI_API_KEY and the value as your API key.

Verifying the Environment Variable

To verify the environment variable is set correctly:

• On Unix-like systems:

    echo $OPENAI_API_KEY
  

• On Windows (command prompt):

  echo %OPENAI_API_KEY%
  

NOTE: Keep your API key confidential and do not share it publicly.

Installation

Install from PyPI (Recommended):

pip install --upgrade arxiv-retriever

Install from Source Distribution

If you need a specific version or want to install from a source distribution:

1. Download the source distribution (.tar.gz file) from PyPI or the GitHub releases page.

2. Install using pip:

   pip install axiv-x.y.z.tar.gz
   

   Replace x.y.z with the version number.

This method can be useful if you need a specific version or are in an environment without direct access to PyPI.

Install for Development and Testing

To install the latest development version from source:

1. Ensure you have Poetry installed. If not, install it by following the instructions at https://python-poetry.org/docs/#installation.
2. Clone the repository:

   git clone https://github.com/MimicTester1307/arxiv_retriever.git
   cd arxiv_retriever  
   

3. Install the project and its dependencies:

   poetry install
   

4. (Optional) To activate the virtual environment created by Poetry:

   poetry shell
   

5. (Optional) Run tests to ensure everything is set up correctly:

   poetry run pytest
   

6. Build the project:

   poetry build
   

7. Install the wheel file using pip:

   pip install dist/arxiv_retriever-1.0.0-py3-none-any.whl
   

Usage

After installation, use the package via the axiv command:

To view available commands:

axiv --help

To view arguments and options for available commands:

axiv <command> --help

Sample Usage

To retrieve the most recent computer science papers by categories, use the fetch command followed by the categories and options:

axiv fetch <categories> [--limit]

Outputs limit papers sorted by submittedDate in descending order

To filter results by author(s):

  axiv fetch <categories> [--limit] [--authors]

Outputs limit papers sorted by submittedDate in descending order, filtered by authors

To retrieve limit papers matching a specified title, use the search command followed by a title and options:

axiv search <title> [--limit]

Outputs limit papers sorted by relevance in descending order

To filter results by author(s):

  axiv search <title> [--limit] [--authors]

Outputs limit papers sorted by relevance in descending order, filtered by authors

Downloading your research papers

There are multiple ways to download your research paper using axiv:

• use axiv download <link> [--download_dir] to download the paper directly from the link
• confirm if you want to download the retrieved papers using fetch or search when asked by the CLI

With option 1, the file is named using the URL's basename, e.g. 2407.09298v1.pdf.

With options 2, the file is named using the title retrieved from the XML data when parsing.

NOTE: If the file name exists, it is overwritten.

Examples

Fetch the latest 5 papers in the cs.AI and cs.GL categories:

axiv fetch cs.AI cs.GL --limit 5

Fetch papers matching the title, "Attention is all you need", refined by author "Ashish":

axiv search "Attention is all you need" --limit 5 --authors "Ashish"

Download papers using links:

• download using link to abstract:

      axiv download https://arxiv.org/abs/2407.20214v1
  

• download using link to pdf:

  axiv download https://arxiv.org/pdf/2407.20214v1
  

Note on Package and Command Names

• Package Name: The package is named arxiv_retriever. This is the name you use when installing via pip or referring to the project.
• Command Name: After installation, you interact with the tool using the axiv command in your terminal.

This distinction allows for a more concise command while maintaining a descriptive package name.

Contributing

Contributions are welcome! Please fork the repository and submit a pull request for any features, bug fixes, or enhancements.

Note on Testing

Currently, 10 out of 11 tests pass, and even that required a bit of magic. Refactoring the tests for asynchrony was an interesting challenge. Discussions and contributions regarding the asynchronous implementation are particularly welcome.

You can contact me via email or leave a comment on the Notion project tracker.

License

This project is licensed under the MIT license. See the LICENSE file for more details.

Acknowledgements

• Typer for the command-line interface
• ElementTree for XML parsing
• arXiv API for providing access to paper metadata via a well-designed API
• Trio and HTTPx for the asynchronous features