zowie 1.3.0

Write Zotero select links into article attachment files

Zowie

Zowie ("Zotero link writer") is a command-line program for macOS that writes Zotero select links into the file attachments contained in a Zotero database.

Table of contents

• Introduction
• Installation
• Usage
• Known issues and limitations
• Additional tips
• Getting help
• Contributing
• License
• Acknowledgments

Introduction

When using Zotero, you may on occasion want to work with PDF files and other attachment files outside of Zotero. For example, if you're a DEVONthink user, you will at some point discover the power of indexing your local Zotero database from DEVONthink. However, when viewing or manipulating the attachments from outside of Zotero, you may run into the following problem: when looking at a given file, how do you find out which Zotero entry it belongs to?

Enter Zowie (a loose acronym for "Zotero link writer", and pronounced like the interjection). Zowie scans through the files on your disk in a local Zotero database, looks up the Zotero bibliographic record corresponding to each file found, and writes a Zotero select link into the file and/or certain macOS Finder/Spotlight metadata fields (depending on the user's choice). A Zotero select link has the form zotero://select/... and when opened on macOS, causes the Zotero desktop application to open that item in your database. Zowie thus makes it possible to go from a file opened in an application other than Zotero (e.g., DEVONthink, Adobe Acrobat), to the Zotero record corresponding to that file.

Regretfully, Zowie can only work with Zotero libraries that use normal/local data storage; it cannot work when Zotero is configured to use linked attachments.

Installation

There are multiple ways of installing Zowie, ranging from downloading a self-contained, single-file, ready-to-run program, to installing it as a typical Python program using pip. Please choose the alternative that suits you and your Mac environment.

Alternative 1: downloading the ready-to-run program

On macOS Catalina (10.15) or later, you can use a ready-to-run version of Zowie that only needs a Python interpreter version 3.8 or higher on your computer. That's the case for macOS 10.15 and later, but before you can use it, you may need to let macOS install some additional software components from Apple. To test it, run the following command in a terminal and take note of the version of Python that it prints:

python3 --version

If this is the first time you've run python3 on your system, macOS will either ask you if you want to install certain additional software components, or it may produce an error about xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools) .... In either case, the solution is to run the following command in the terminal:

xcode-select --install

In the pop-up dialog box that this brings up, click the Install button and agree to let it install the Command Line Tools package from Apple.

Next,

1. Go to the latest release on GitHub and find the Assets
2. Download the ZIP file whose name contains the version of Python on your computer (which you determined by running python3 --version above)
3. Unzip the file (if your browser didn't unzip it)
4. Open the folder that gets created (it will have a name like zowie-1.2.0-macos-python3.8)
5. Look inside for zowie and move it to a location where you put other command-line programs (such as /usr/local/bin).

If you want to put it in /usr/local/bin but that folder does not exist on your computer yet, you can create it by opening a terminal window and running the following command (prior to moving zowie into /usr/local/bin):

sudo mkdir /usr/local/bin

The following is an example command that you can type in a terminal to move Zowie there:

sudo mv zowie /usr/local/bin

Alternative 2: installing Zowie using pipx

You can use pipx to install Zowie. Pipx will install it into a separate Python environment that isolates the dependencies needed by Zowie from other Python programs on your system, and yet the resulting zowie command wil be executable from any shell – like any normal program on your computer. If you do not already have pipx on your system, it can be installed in a variety of easy ways and it is best to consult Pipx's installation guide for instructions. Once you have pipx on your system, you can install Zowie with the following command:

pipx install zowie

Pipx can also let you run Zowie directly using pipx run zowie, although in that case, you must always prefix every Zowie command with pipx run. Consult the documentation for pipx run for more information.

Alternative 3: installing Zowie using pip

The instructions below assume you have a Python 3 interpreter installed on your computer. Note that the default on macOS at least through 10.14 (Mojave) is Python 2 – please first install Python version 3 and familiarize yourself with running Python programs on your system before proceeding further.

You should be able to install zowie with pip for Python 3. To install zowie from the Python package repository (PyPI), run the following command:

python3 -m pip install zowie

As an alternative to getting it from PyPI, you can use pip to install zowie directly from GitHub:

python3 -m pip install git+https://github.com/mhucka/zowie.git

If you already installed Zowie once before, and want to update to the latest version, add --upgrade to the end of either command line above.

Alternative 4: installing Zowie from sources

If you prefer to install Zowie directly from the source code, you can do that too. To get a copy of the files, you can clone the GitHub repository:

git clone https://github.com/mhucka/zowie

Alternatively, you can download the files as a ZIP archive using this link directly from your browser using this link: https://github.com/mhucka/zowie/archive/refs/heads/main.zip

Next, after getting a copy of the files, run setup.py inside the code directory:

cd zowie
python3 setup.py install

Usage

For help with usage at any time, run zowie with the option -h.

The zowie command-line program should end up installed in a location where software is normally installed on your computer, if the installation steps described in the previous section proceeded successfully. Running Zowie from a terminal shell then should be as simple as running any other shell command on your system:

zowie -h

If you installed it as a Python package, then an alternative method is available to run Zowie from anywhere, namely to use the normal approach for running Python modules:

python3 -m zowie -h

Credentials for Zotero access

Zowie relies on the Zotero sync API to get information about your references. This allows it to look up Zotero item URIs for files regardless of whether they belong to your personal library or shared libraries, and from there, construct the appropriate Zotero select link for the files. If you do not already have a Zotero sync account, it will be necessary to create one before going any further.

To use Zowie, you will also need both an API user identifier (also known as the userID) and an API key. To find out your Zotero userID and create a new API key, log in to your Zotero account at Zotero.org and visit the Feeds/API tab of the your Settings page. On that page you can find your userID and create a new API key for Zowie.

The first time you run Zowie, it will ask for this information and (unless the -K option is given) store it in your macOS keychain so that it does not have to ask for it again on future occasions. It is also possible to supply the identifier and API key directly on the command line using the -i and -a options, respectively; the given values will then override any values stored in the keychain and (unless the -K option is also given) will be used to update the keychain for the next time.

Basic usage

Zowie can operate on a folder, or one or more individual files, or a mix of both. Suppose your local Zotero database is located in ~/Zotero/. Perhaps the simplest way to run Zowie is the following command:

zowie ~/Zotero

If this is your first run of Zowie, it will ask you for your userID and API key, then search for files recursively under ~/Zotero/. For each file found, Zowie will contact the Zotero servers over the network and determine the Zotero select link for the bibliographic entry containing that file. Finally, it will use the default method of recording the link, which is to write it into the macOS Finder comments for the file. It will also store your Zotero userID and API key into the system keychain so that it does not have to ask for them in the future.

If you are a user of DEVONthink, you will probably want to add the -s option (see the explanation below for the details):

zowie -s ~/Zotero

Instead of a folder, you can also invoke Zowie on one or more individual files (but be careful to put quotes around pathnames with spaces in them, such as in this example):

zowie -s "~/Zotero/storage/26GS7CZL/Smith 2020 Paper.pdf"

Available methods of writing Zotero links

Zowie supports multiple methods of writing the Zotero select link. The option -l will cause Zowie to print a list of all the methods available, then exit.

The option -m can be used to select one or more methods when running Zowie. Write the method names separated with commas without spaces. For example, the following command will make Zowie write the Zotero select link into the Finder comments as well as the PDF metadata attribute Subject:

zowie -m findercomment,pdfsubject ~/Zotero/storage

At this time, the following methods are available:

• findercomment: (The default method.) Writes the Zotero select link into the Finder comments of each file, attempting to preserve other parts of the comments. If Zowie finds an existing Zotero select link in the text of the Finder comments attribute, it only updates the link portion and tries to leave the rest of the comment text untouched. Otherwise, Zowie only writes into the comments attribute if either the attribute value is empty or Zowie is given the overwrite (-o) option. (Note that updating the link text requires rewriting the entire Finder comments attribute on a given file. Finder comments have a reputation for being easy to get into inconsistent states, so if you have existing Finder comments that you absolutely don't want to lose, it may be safest to avoid this method.)

• pdfproducer: (Only applicable to PDF files.) Writes the Zotero select link into the "Producer" metadata field of each PDF file. If the "Producer" field is not empty on a given file, Zowie looks for an existing Zotero link within the value and updates the link if one is found; otherwise, Zowie leaves the field untouched unless given the overwrite flag (-o), in which case, it replaces the entire contents of the field with the Zotero select link. For some users, the "Producer" field has not utility, and thus can be usefully hijacked for the purpose of storing the Zotero select link. The value is accessible from macOS Preview, Adobe Acrobat, DEVONthink, and presumably any other application that can display the PDF metadata fields. However, note that some users (archivists, forensics investigators, possibly others) do use the "Producer" field, and overwriting it may be undesirable.

• pdfsubject: (Only applicable to PDF files.) Writes the Zotero select link into the "Subject" metadata field of each PDF file. If the "Subject" field is not empty on a given file, Zowie looks for an existing Zotero link within the value and updates the link if one is found; otherwise, Zowie leaves the field untouched unless given the overwrite flag (-o), in which case, it replaces the entire contents of the field with the Zotero select link. Note that the PDF "Subject" field is not the same as the "Title" field. For some users, the "Subject" field is not used for any purpose and thus can be usefully hijacked for storing the Zotero select link. The value is accessible from macOS Preview, Adobe Acrobat, DEVONthink, and presumably any other application that can display the PDF metadata fields.

• wherefrom: Writes the Zotero select link to the "Where from" metadata field of each file (the com.apple.metadata:kMDItemWhereFroms extended attribute). This field is displayed as "Where from" in Finder "Get Info" panels; it is typically used by web browsers to store a file's download origin. The field is a list. If Zowie finds a Zotero select link as the first item in the list, it updates that value; otherwise, Zowie prepends the Zotero select link to the list of existing values, keeping the other values unless the overwrite option (-o) is used. When the overwrite option is used, Zowie deletes the existing list of values and writes only the Zotero select link. Note that if macOS Spotlight indexing is turned on for the volume containing the file, the macOS Finder will display the updated "Where from" values in the Get Info panel of the file; if Spotlight is not turned on, the Get info panel will not be updated, but other applications will still be able to read the updated value.

Note that, depending on the attribute, it is possible that a file has an attribute value that is not visible in the Finder or other applications. This is especially true for "Where from" values and Finder comments. The implication is that it may not be apparent when a file has a value for a given attribute, which can lead to confusion if Zowie thinks there is a value and refuses to change it without the -o option.

Filtering by file type

By default, Zowie acts on all files it finds on the command line, except for certain files that it always ignores: hidden files and files with extensions .sqlite, .bak, .csl, .css, .js, .json, .pl, and a few others. If the -m option is used to select methods that only apply to specific file types, Zowie will examine each file it finds in turn and only apply the methods that match that particular file's type, but it will still consider every file it finds in the directories it scans and apply the methods that are not limited to specific types.

You can use the option -f to make Zowie filter the files it finds based on file name extensions. This is useful if you want it to concentrate only on particular file types and ignore other files it might find while scanning folders. Here is an example (this also using the -s option for reasons given below):

zowie -s -f pdf,mp4,mov ~/Zotero

will cause it to only work on PDF, MP4, and QuickTime format files. You can provide multiple file extensions separated by commas, without spaces and without the leading periods.

Filtering by date

If the -d option is given, the files will be filtered to use only those whose last-modified date/time stamp is no older than the given date/time description. Valid descriptors are those accepted by the Python dateparser library. Make sure to enclose descriptions within single or double quotes. Examples:

zowie -d "2 weeks ago" ....
zowie -d "2014-08-29" ....
zowie -d "12 Dec 2014" ....
zowie -d "July 4, 2013" ....

Special-case behavior

Although Zowie is not aimed solely at DEVONthink users, its development was motivated by the author's desire to use Zotero with that software. A complication arose due to an undocumented feature in DEVONthink: it ignores a Finder comment if it is identical to the value of the "URL" attribute (which is the name it gives to the com.apple.metadata:kMDItemWhereFroms attribute discussed above). In practical terms, if you do something like write the Zotero select link into the Finder comment of a file and then have a DEVONthink smart rule copy the value to the URL field, the Finder comment will subsequently appear blank in DEVONthink (even though it exists on the actual file). This can be unexpected and confusing, and has caught people (including the author of Zowie) unaware. To compensate, Zowie 1.2 introduced a new option: it can add a trailing space character to the end of the value it writes into the Finder comment when using the findercomment method. Since approaches to copy the Zotero link from the Finder comment to the URL field in DEVONthink will typically strip whitespace around the URL value, the net effect is to make the value in the Finder comment just different enough from the URL field value to prevent DEVONthink from ignoring the Finder comment. Use the option -s to make Zowie to add the trailing space character.

Additional command-line arguments

To make Zowie only print what it would do without actually doing it, use the -n "dry run" option.

If given the -q option, Zowie will not print its usual informational messages while it is working. It will only print messages for warnings or errors. By default, messages printed by Zowie are also color-coded. If given the option -C, Zowie will not color the text of messages it prints. (This latter option is useful when running Zowie within subshells inside other environments such as Emacs.)

If given the -V option, this program will print the version and other information, and exit without doing anything else.

If given the -@ argument, this program will output a detailed trace of what it is doing. The debug trace will be sent to the given destination, which can be - to indicate console output, or a file path to send the output to a file.

When -@ has been given, Zowie also installs a signal handler on signal SIGUSR1 that will drop Zowie into the pdb debugger if the signal is sent to the running process.

Summary of command-line options

The following table summarizes all the command line options available.

Short	Long form opt	Meaning	Default
-aA	--api-keyA	API key to access the Zotero API service
-C	--no-color	Don't color-code the output	Use colors in the terminal
-d	--after-dateD	Only act on files modified after date "D"	Act on all files found
-f	--file-extF	Only act on files with extensions in "F"	Act on all files found	⚑
-h	--help	Display help text and exit
-i	--identifierI	Zotero user ID for API calls
-K	--no-keyring	Don't use a keyring/keychain	Store login info in keyring
-l	--list	Display known services and exit
-m	--methodM	Select how Zotero select links are written	findercomment
-n	--dry-run	Say what would be done, but don't do it	Do it
-o	--overwrite	Overwrite previous metadata content	Don't write if already present
-q	--quiet	Don't print messages while working	Be chatty while working
-s	--space	Append trailing space to Finder comments	Don't add a space	★
-V	--version	Display program version info and exit
-@OUT	--debugOUT	Debugging mode; write trace to OUT	Normal mode	⬥

⚑   Certain files are always ignored: hidden files, macOS aliases, and files with extensions .sqlite, .sqlite-journal, .bak, .csl, .css, .js, .json, .pl, and .config_resp.
⬥   To write to the console, use the character - as the value of OUT; otherwise, OUT must be the name of a file where the output should be written.
★   See the explanation in the section on special-case behavior.

Return values

This program exits with a return code of 0 if no problems are encountered. It returns a nonzero value otherwise. The following table lists the possible return values:

Code	Meaning
0	success – program completed normally
1	the user interrupted the program's execution
2	encountered a bad or missing value for an option
3	no network detected – cannot proceed
4	file error – encountered a problem with a file
5	server error – encountered a problem with a server
6	an exception or fatal error occurred

Known issues and limitations

The following is a list of currently-known issues and limitations:

• Zowie can only work when Zotero is set to use direct data storage; i.e., where attached files are stored in Zotero. It cannot work if you use linked attachments, that is, if you set Linked Attachment Base Directory in your Zotero Preferences' Advanced → Files and Folders panel.

• If you use DEVONthink in a scheme in which you index your Zotero folder and use Zowie to write the Zotero select link into the Finder comments of files, beware of the following situation. If you use a DEVONthink smart rule to copy the comment string into the "URL" field, DEVONthink will (after reindexing the file) suddenly display an empty Finder comment, even though the comment is still there. This is due to a deliberate behavior in DEVONthink and not a problem with Zowie, as discussed in the section on special-case behavior. Using the -s option will avoid this, but at the cost of adding an extra character to the Finder comment, so make sure to account for the added space character in any scripts or other actions you take on the Finder comment.

• DEVONthink bases the "URL" value of a file on the file's com.apple.metadata:kMDItemWhereFroms extended attribute. The original hope behind Zowie was to make it write Zotero select links directly into that attribute value. Unfortunately, it turns out that if a file has already been indexed by DEVONthink, then it will not detect any changes to the com.apple.metadata:kMDItemWhereFroms attribute made by an external program. Thus, if you index your Zotero folder within DEVONthink, you cannot use Zowie's wherefroms method to update the "URL" field directly. You are advised instead to use Zowie's findercomment method (the default) in combination with smart rules in DEVONthink, as discussed in the wiki. I share your frustration.

• For reasons I have not had time to investigate, the binary version of zowie takes a very long time to start up on macOS 10.15 (Catalina) and 11.1 (Big Sur). On my test system inside a virtual machine running on a fast iMac, it takes 10 seconds or more before the first output from zowie appears.

Additional tips

In the wiki associated with the Zowie project in GitHub, I have started writing some notes about how I personally use Zowie to combine Zotero with DEVONthink.

Getting help

If you find an issue, please submit it in the GitHub issue tracker for this repository.

Contributing

I would be happy to receive your help and participation if you are interested. Everyone is asked to read and respect the code of conduct when participating in this project. Development generally takes place on the development branch.

License

This software is Copyright (C) 2020–2023, by Michael Hucka and the California Institute of Technology (Pasadena, California, USA). This software is freely distributed under a 3-clause BSD type license. Please see the LICENSE file for more information.

Acknowledgments

This work is a personal project developed by the author, using computing facilities and other resources of the California Institute of Technology Library.

The vector artwork of an exclamation point circled by a zigzag, used as the icon for this repository, was created by Alfredo @ IconsAlfredo.com from the Noun Project. It is licensed under the Creative Commons CC-BY 3.0 license.

Zowie makes use of numerous open-source packages, without which Zowie could not have been developed. I want to acknowledge this debt. In alphabetical order, the packages are:

• aenum – advanced enumerations for Python
• biplist – A binary plist parser/writer for Python
• boltons – package of miscellaneous Python utilities
• bun – a set of basic user interface classes and functions
• CommonPy – a collection of commonly-useful Python functions
• fastnumbers – number testing and conversion functions
• ipdb – the IPython debugger
• keyring – access the system keyring service from Python
• pdfrw – a pure Python library for reading and writing PDFs
• plac – a command line argument parser
• py-applescript – a Python interface to AppleScript
• PyInstaller – a packaging program that creates standalone applications from Python programs
• pyobjc – Python ⇌ Objective-C and macOS frameworks bridge
• pyxattr – access extended file attributes from Python
• pyzotero – a Python API client for Zotero
• setuptools – library for setup.py
• Shiv – command-line utility for creating self-contained Python zipapps
• Sidetrack – simple debug logging/tracing package
• wheel – setuptools extension for building wheels

The developers of DEVONthink, especially Jim Neumann and Christian Grunenberg, quickly and consistently replied to my many questions on the DEVONtechnologies forums.