Skip to main content

ArXiv-Miner: Mine/Scrape Arxiv-Papers To Structured Datasets

Project description


ArXiv Miner is a toolkit for mining research papers on CS ArXiv.

What is ArXiv-Miner

arxiv-miner is a quick handy library that helps power Sci-Genie. Sci-Genie is a search engine for quickly searching through full text of papers on CS ArXiv. arxiv-miner helps extract and parse LaTeX documents from CS ArXiv. It also supports storage and search of those parsed documents using Elasticsearch. The library can be applicable for all other domains like Math, Physics, Biology etc.


All documentation on how to install and use arxiv-miner is provided in the documentation website or inside the docs folder. Contribution guidelines are also provided there.

Why was ArXiv-Miner created ?

ArXiv Miner was created for easily scraping, parsing and searching research content on ArXiv. This library was created after stitching together solutions from the code of various tools like arxiv-sanity, arxiv-vanity/engrafo, arxivscraper, tex2py, cso-classifier and axcell. Parsed structure of the content can be useful in search or any scientific research mining/AI applications as a heuristic baseline.

Core Components of ArXiv-Miner

  • Scraping
  • Parsing
  • Indexing/Storage

Family Of Projects With ArXiv-Miner


This project was developed like a Cowboy coder over the COVID-19 pandemic. Hence, this may have bugs and not the most well optimized code. The primary reason for development was to aid CS and Machine Learning/AI research, but this tool can be extended to all 3M+ documents on ArXiv.

Call For Contributors

Any help with contributions to improve the project or fix bugs are completely welcome. Please read the contribution guide in the documentation.

Credits and Appreciation

This project like all others has been built on shoulders of giants. A big thanks to the creators of the following libraries/open source projects that aided the development of arxiv-miner, and it's family of projects:



Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

arxiv_miner-2.0.3.tar.gz (57.8 kB view hashes)

Uploaded source

Built Distribution

arxiv_miner-2.0.3-py3-none-any.whl (49.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page