microarchiver 1.12.1

Archives articles from microPublication.org

Microarchiver

A program to create archives of articles from microPublication.org for sending to Portico.

Authors: Michael Hucka, Tom Morrell
Repository: https://github.com/caltechlibrary/microarchiver
License: BSD/MIT derivative – see the LICENSE file for more information

Table of Contents

• Introduction
• Installation
• Usage
• Known issues and limitations
• Getting help and support
• Contributing
• License
• Authors and history
• Acknowledgments

☀ Introduction

The Caltech Library is the publisher of the online journal microPublication and provides services to the journal that include archiving the journal in a dark archive (specifically, Portico). The archiving process involves pulling down articles from microPublication and packaging them up in a format suitable for sending to Portico. microarchiver is a program to automate this process.

✺ Installation

On Linux, macOS, and Windows operating systems, you should be able to install Microarchiver directly from the GitHub repository using pip. If you don't have the pip package or are uncertain if you do, first run the following command in a terminal command line interpreter:

sudo python3 -m ensurepip

Then, install this software by running the following command on your computer:

python3 -m pip install git+https://github.com/caltechlibrary/microarchiver.git 

Alternatively, you can clone this GitHub repository and then run setup.py:

git clone https://github.com/caltechlibrary/microarchiver.git
cd microarchiver
python3 -m pip install .

▶︎ Usage

Microarchiver is a command-line program. The installation process should put a program named microarchiver in a location normally searched by your shell interpreter. For help with usage at any time, run microarchiver with the option -h (or /h on Windows).

microarchiver -h

Basic usage

The simplest use of microarchiver involves running it without any options. By default, it will contact microPublication.org to get a list of current articles, and create an archive of all the articles in a subdirectory of the current directory.

microarchiver

If given the option -o (or /o on Windows), the output will be written to the directory named after the -o. (If no -o is given, the output will be written to the current directory instead.) For example:

microarchiver -o /tmp/micropublication-archive

If the option -d is given, microarchiver will download only articles whose publication dates are after the given date. Valid date descriptors are those accepted by the Python dateparser library. Make sure to enclose descriptions within single or double quotes. Examples:

  microarchiver -d "2014-08-29"   ....
  microarchiver -d "12 Dec 2014"  ....
  microarchiver -d "July 4, 2013"  ....
  microarchiver -d "2 weeks ago"  ....

As it works, microarchiver writes information to the terminal about the archives it puts into the archive, including whether any problems are encountered. To save this info to a file, use the option -r (or /r on Windows), which will make microarchiver write a report file. By default, the format is CSV; to change the format of the report to HTML, use the option -f html (or /f html on Windows). The title of the report will be named after the current date, unless the option -t (or /t on Windows) is used to supply a different title.

The following is a screen recording of an actual run of microarchiver:

Previewing the list of articles

If given the option -p (or /p on Windows), microarchiver will only print a list of articles it will archive and stop short of creating the archive. This is useful to see what would be produced without actually doing it. However, note that because it does not attempt to download the articles and associated files, it will not be able to report on errors that might occur when not operating in preview mode. Consequently, do not use the output of -p as a prediction of eventual success or failure.

If given the option -g (or /g on Windows), microarchiver will only write a file named article-list.xml containing the complete current article list from the micropublication.org server, and exit without doing anything else. This is useful as a starting point for creating the file used by option -a. It's probably a good idea to redirect the output to a file; e.g.,

microarchiver -g > article-list.xml

Output

Unless given the option -g or -p, microarchiver will download articles from micropublication.org and create archive files out of them.

The value supplied after the option -s (or /s on Windows) determines the structure of the archive generated by this program. Currently, two output structures are supported: PMC, and a structure suitable for Portico. (The PMC structure follows the "naming and delivery" specifications defined at https://www.ncbi.nlm.nih.gov/pmc/pub/filespec-delivery/.) If the output will be sent to PMC, use -s pmc; else, use -s portico or leave off the option altogether (because Portico is the default).

The output will be written to the directory indicated by the value of the option -o (or /o on Windows). If no -o is given, the output will be written to the directory in which microarchiver was started. Each article will be written to a subdirectory named after the DOI of the article. The output for each article will consist of an XML metadata file describing the article, the article itself in PDF format, and a subdirectory named jats containing the article in JATS XML format along with any image that may appear in the article. The image is always converted to uncompressed TIFF format (because it is considered a good preservation format).

Unless the option -Z (or /Z on Windows) is given, the output will be archived in ZIP format. If the output structure (as determine by the -s option) is being generated for PMC, each article will be put into its own individual ZIP archive; else, the default action is to put the collected output of all articles into a single ZIP archive file. The option -Z makes microarchiver leave the output unarchived in the output directory determined by the -o option.

Additional command-line options

If given the option -a (or /a on Windows) followed by a file name, the given file will be read for the list of articles instead of getting the list from the server. The contents of the file can be either a list of DOIs, or article data in the same XML format as the list obtained from micropublication.org. (See option -g above for a way to get an article list in XML from the server.)

Microarchiver always downloads the JATS XML version of articles from micropublication.org (in addition to downloading the PDF version), and by default, microarchiver validates the XML content against the JATS DTD. To skip the XML validation step, use the option -X (/X on Windows).

microarchiver will print informational messages as it works. To reduce messages to only warnings and errors, use the option -q (or /q on Windows). Also, output is color-coded by default unless the -C option (or /C on Windows) is given; this option can be helpful if the color control sequences create problems for your terminal emulator.

If given the -@ option (/@ on Windows), this program will output a detailed real-time trace of what it is doing. The output will be written to the given destination, which can be a dash character (-) to indicate console output, or a file path.

If given the -V option (/V on Windows), this program will print version information and exit without doing anything else.

Return values

This program exits with a return code of 0 if no problems are encountered while fetching data from the server. It returns a nonzero value otherwise, following conventions for use in shells such as bash which only understand return code values of 0 to 255. If no network is detected, it returns a value of 1; if it is interrupted (e.g., using ctrl-c) it returns a value of 2; if it encounters a fatal error, it returns a value of 3. If it encounters any non-fatal problems (such as a missing PDF file or JATS validation error), it returns a nonzero value equal to 100 + the number of articles that had failures. Summarizing the possible return codes:

Return value	Meaning
0	No errors were encountered – success
1	No network detected – cannot proceed
2	The user interrupted program execution
3	An exception or fatal error occurred
100 + n	Encountered non-fatal problems on a total of n articles

Summary of command-line options

The following table summarizes all the command line options available. (Note: on Windows computers, / must be used as the prefix character instead of -):

Short	Long form opt	Meaning	Default
-aA	--articlesA	Get list of articles from file A	Get list from server
-C	--no-color	Don't color-code the output	Color the terminal output
-dD	--after-dateD	Only get articles published after date D	Get all articles	⬥
-fF	--rep-formatF	Report format, either html or csv	csv
-g	--get-xml	Print the server's article list & exit	Do other actions instead
-oO	--output-dirO	Write output in directory O	Write in current dir
-p	--preview	Preview what would be obtained	Obtain the articles
-q	--quiet	Only print important messages	Be chatty while working
-rR	--rep-fileR	Write list of article & results in file R	Don't write a report
-sS	--structureS	Structure output for Portico or PMC	Portico
-tT	--rep-titleT	Use T as the report title	Use the current date
-V	--version	Print program version info and exit	Do other actions instead
-X	--no-check	Don't validate JATS XML files	Validate JATS XML
-Z	--no-zip	Don't put output into one ZIP archive	ZIP up the output
-@OUT	--debugOUT	Debugging mode; write trace to OUT	Normal mode	⚑

⬥   Enclose the date in quotes if it contains space characters; e.g., "12 Dec 2014".
⚑   To write to the console, use the character - (a single dash) as the value of OUT; otherwise, OUT must be the name of a file where the output should be written.

⚑ Known issues and limitations

Currently, the only way to indicate that a subset of articles should be obtained from microPublication.org is to use the option -a in combination with a file that contains the list of desired articles, or the -d option to indicate a cut-off for the article publication date.

⁇ Getting help and support

If you find an issue, please submit it in the GitHub issue tracker for this repository.

♬ Contributing

We would be happy to receive your help and participation with enhancing microarchiver! Please visit the guidelines for contributing for some tips on getting started.

☥ License

Copyright © 2019-2020, Caltech. This software is freely distributed under a BSD/MIT type license. Please see the LICENSE file for more information.

❡ Authors and history

Tom Morrell developed the original algorithm for extracting metadata from DataCite and creating XML files for use with Portico submissions of microPublication.org articles. Mike Hucka created the much-expanded second version of the software, now known as Microarchiver.

♥︎ Acknowledgments

The vector artwork used as a starting point for the logo for this repository was created by Thomas Helbig for the Noun Project. It is licensed under the Creative Commons Attribution 3.0 Unported license. The vector graphics was modified by Mike Hucka to change the color.

Nick Stiffler from the microPublication.org team helped figure out the requirements for PMC output (introduced in version 1.9), helped guide development of microarchiver, and engaged in many discussions about microPublication.org's needs.

Microarchiver makes use of numerous open-source packages, without which it would have been effectively impossible to develop Microarchiver with the resources we had. We want to acknowledge this debt. In alphabetical order, the packages are:

• colorful – terminal/text string styling
• commonpy – a collection of commonly-useful Python functions
• dateparser – parser for human-readable dates
• dateutil – extensions to the Python datetime module
• humanize – make numbers more easily readable by humans
• lxml – an XML parsing library for Python
• Pillow – a fork of the Python Imaging Library
• plac – a command line argument parser
• pypubsub – a publish-and-subscribe message-passing library for Python
• recordclass – a mutable version of Python named tuples
• requests – an HTTP library for Python
• setuptools – library for setup.py
• slack-cli – a command-line interface to Slack written in Bash
• urllib3 – a powerful HTTP library for Python
• xmltodict – a module to make working with XML feel like working with JSON
• wxPython – a cross-platform GUI toolkit for the Python language

Finally, we are grateful for computing & institutional resources made available by the California Institute of Technology.