pdf2doi 0.3

A python library/command-line tool to retrieve the DOI or other identifiers (e.g. arXiv) from pdf files of a publications.

pdf2doi

pdf2doi is a Python library to extract the DOI or other identifiers (e.g. arXiv) starting from the .pdf file of a publication (or from a folder containing several .pdf files). It exploits several methods (see below for detailed description) to find a possible identifier, and it validates any result via web queries to public archives (e.g. http://dx.doi.org). Additionally, it allows generating automatically bibtex entries. (Note: in the current version only the format of arXiv identifiers in use after 1 April 2007 is supported)

Description

Automatically associating a DOI or other identifiers (e.g. arXiv) to a pdf file can be either a very easy or a very difficult (sometimes nearly impossible) task, depending on how much care was placed in crafting the file. In the simplest case (which typically applies to most recent publications) it is enough to look into the file metadata. For older publications, the identifier is often found within the pdf text and it can be extracted with the help of regular expressions. In the unluckiest cases, the only method left is to google some details of the publication (e.g. the title or parts of the text) and hope that a valid identifier is contained in one of the first results.

The pdf2doi library applies sequentially all these methods (starting from the simpler one) until a valid identifier is found and validated. Specifically, for a given .pdf file it will, in order,

1. Look into the metadata of the .pdf file (extracted via the library PyPDF2) and see if any string matches the pattern of a DOI or an arXiv ID. Priority is given to the metadata which contain the word 'doi' in their label.

2. Check if the name of the pdf file contains any sub-string that matches the pattern of a DOI or an arXiv ID.

3. Scan the text inside the .pdf file, and check for any string that matches the pattern of a DOI or an arXiv ID. The text is extracted with the libraries PyPDF2 and textract.

4. Try to find possible titles of the publication. In the current version, possible titles are identified via the library pdftitle, and by the file name. For each possible title a google search is performed and the plain text of the first results is scanned for valid identifiers.

5. As a last desperate attempt, the first N=1000 characters of the pdf text are used as a query for a google search (the value of N can be set by the variable config.N_characters_in_pdf). The plain text of the first results is scanned for valid identifiers.

When a valid identifier is found with any method different than the first one, the identifier is also added to the metadata of the pdf file with key='/identifier'. In this way future lookups of this file will be able to extract the identifier with the first method, speeding up the search. This feature can be disabled by the user (in case edits to the pdf file are not desired).

Installation

Use the package manager pip to install pdf2doi.

pip install pdf2doi

Usage

pdf2doi can be used either as a stand-alone application invoked from the command line, or by importing it in your python project.

Usage inside a python script:

The function pdf2doi can be used to look for the identifier of a pdf file by applying all the available methods. Setting verbose=True will increase the output verbosity, documenting all steps performed.

import pdf2doi
result = pdf2doi.pdf2doi('.\examples\PhysRevLett.116.061102.pdf',verbose=True)
print('\n')
print(result['identifier'])
print(result['identifier_type'])
print(result['method'])

The previous code produces the output

................
File: .\examples\PhysRevLett.116.061102.pdf
Looking for a valid identifier in the document infos...
Could not find a valid identifier in the document info.
Looking for a valid identifier in the file name...
Could not find a valid identifier in the file name.
Looking for a valid identifier in the document text...
Extracting text with the library PyPdf...
Validating the possible DOI 10.1103/PhysRevLett.116.061102 via a query to dx.doi.org...
The DOI 10.1103/PhysRevLett.116.061102 is validated by dx.doi.org. A bibtex entry was also created.
A valid DOI was found in the document text.

10.1103/PhysRevLett.116.061102
DOI
document_text

The output variable result is a dictionary containing the identifier and other relevant information,

result['identifier'] =      DOI or other identifier (or None if nothing is found)
result['identifier_type'] = string specifying the type of identifier (e.g. 'doi' or 'arxiv')
result['validation_info'] = Additional info on the paper. If the online validation is enabled, then result['validation_info']
                            will typically contain a bibtex entry for this paper. Otherwise it will just contain True                         
result['path'] =            path of the pdf file
result['method'] =          method used to find the identifier

The first argument passed to the function pdf2doi can also be a directory. In this case the function will look for all valid pdf files inside the directory, try to find a valid identifier for each of them, and return a list of dictionaries.

For example, the code

import pdf2doi
results = pdf2doi.pdf2doi('.\examples')
for result in results:
    print(result['identifier'])

produces the output

10.1103/PhysRevLett.116.061102
10.1103/PhysRevLett.76.1055
10.1038/s41586-019-1666-5

Additional arguments can be passed to the function pdf2doi to control its behaviour, for example to specify if web-based methods (either to find an identifier and/or to validate it) should not be used.

def pdf2doi(target, verbose=False, websearch=True, webvalidation=True,
            save_identifier_metadata = config.save_identifier_metadata,
            numb_results_google_search=config.numb_results_google_search,
            filename_identifiers = False, filename_bibtex = False):
    '''
    Parameters
    ----------
    target : string
        Relative or absolute path of the target .pdf file or directory
    verbose : boolean, optional
        Increases the output verbosity. The default is False.
    websearch : boolean, optional
        If set false, any method to find an identifier which requires a web search is disabled. The default is True.
    webvalidation : boolean, optional
        If set false, validation of identifier via internet queries (e.g. to dx.doi.org or export.arxiv.org) is disabled. 
        The default is True.
    save_identifier_metadata : boolean, optional
        If set True, when a valid identifier is found with any method different than the metadata lookup, the identifier
        is also written in the file metadata with key "/identifier". If set False, this does not happen. The default
        is True.
    numb_results_google_search : integer, optional
        It sets how many results are considered when performing a google search. The default is config.numb_results_google_search.
    filename_identifiers : string or boolean, optional
        If is set equal to a string, all identifiers found in the directory specified by target are saved into a text file 
        with a path specified by filename_identifiers. The default is False.
        It is ignored if the input parameter target is a file.
    filename_bibtex : string or boolean, optional
        If is set equal to a string, all bibtex entries obtained in the validation process for the pdf files found in the 
        directory specified by target are saved into a text file with a path specified by filename_bibtex. 
        The default is False.
        It is ignored if the input parameter target is a file.

    Returns
    -------
    results, dictionary or list of dictionaries (or None if an error occured)
        The output is a single dictionary if target is a file, or a list of dictionaries if target is a directory, 
        each element of the list describing one file. Each dictionary has the following keys

        result['identifier'] = DOI or other identifier (or None if nothing is found)
        result['identifier_type'] = string specifying the type of identifier (e.g. 'doi' or 'arxiv')
        result['validation_info'] = Additional info on the paper. If config.check_online_to_validate = True, then result['validation_info']
                                    will typically contain a bibtex entry for this paper. Otherwise it will just contain True                         
        result['path'] = path of the pdf file
        result['method'] = method used to find the identifier

    ''' 

The online validation of an identifier relies on performing queries to different online archives (e.g. dx.doi.org for DOIs or export.arxiv.org for arXiv identifiers). Using data obtained from these queries, a bibtex entry is created and stored in the 'validation_info' element of the output dictionary. By setting the input argument filename_bibtex equal to a valid filename, the bibtex entries of all papers in the target directory will be saved in a file within the same directory.

For example,

import pdf2doi
results = pdf2doi.pdf2doi('.\examples',filename_bibtex='bibtex.txt')

creates the file bibtex.txt in the 'examples' folder.

Command line usage:

The library can also be used directly from the command line, without having to open a python console.

• Find and print the identifier of a single paper, outputting all logs:

>>pdf2doi '.\examples\PhysRevLett.116.061102.pdf'

• Find and print the identifiers of all pdf files contanined in a folder:

>>pdf2doi '.\examples\' --no_verbose

• Find the identifiers of all pdf files contanined in a folder and create a text file with all the bibtex entries:

>>pdf2doi '.\examples\' -b 'bibtex.txt' --no_verbose

The syntax for the command-line invokation follows closely the arguments that can be passed to the pdf2doi python function,

>> pdf2doi --h
usage: pdf2doi [-h] [-nv] [-nws] [-nwv] [-nsim]
               [-google_results GOOGLE_RESULTS] [-s [FILENAME_IDENTIFIERS]]
               [-b [FILENAME_BIBTEX]]
               path

Retrieves the DOI or other identifiers (e.g. arXiv) from pdf files of a
publications.

positional arguments:
  path                  Relative path of the pdf file or of a folder.

optional arguments:
  -h, --help            show this help message and exit
  -nv, --no_verbose     Decrease verbosity.
  -nws, --no_web_search
                        Disable any method to find identifiers which requires
                        internet searches (e.g. queries to google).
  -nwv, --no_web_validation
                        Disable the online validation of identifiers (e.g.,
                        via queries to http://dx.doi.org/).
  -nsim, --no_store_identifier_metadata
                        By default, anytime an identifier is found it is added
                        to the metadata of the pdf file (if not present yet).
                        By setting this parameter, the identifier is not
                        stored in the file metadata.
  -google_results GOOGLE_RESULTS
                        Set how many results should be considered when doing a
                        google search for the DOI (default=6).
  -s FILENAME_IDENTIFIERS, --save_identifiers_file FILENAME_IDENTIFIERS
                        Save all the identifiers found in the target folder in
                        a text file inside the same folder with name specified
                        by FILENAME_IDENTIFIERS (only available when a folder
                        is targeted).
  -b FILENAME_BIBTEX, --make_bibtex_file FILENAME_BIBTEX
                        Create a text file inside the target directory with
                        name given by FILENAME_BIBTEX containing the bibtex
                        entry of each pdf file in the target folder (if a
                        valid identifier was found). This option is only
                        available when a folder is targeted, and when the web
                        validation is allowed.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT