lockss-turtles 0.3.0.dev1

A tool to manage LOCKSS plugin sets and LOCKSS plugin repositories

Turtles is a tool to manage LOCKSS plugin sets and LOCKSS plugin registries.

Turtles supports Plugin Sets using a Maven layout inheriting from org.lockss:lockss-plugins-parent-pom or the legacy Ant layout of lockss-daemon, and Plugin Registries with flat directory structures (optionally with RCS versioning) on the local file system.

Prerequisites

• Python 3.6 or greater.

Turtles’ Python dependencies are defined in its requirements.txt files.

Other prerequisites depend on the Plugin Set Builders and Plugin Registry Layouts that may be involved in your activities; see the notes about system prerequisites for each.

Quick Start

• Get Turtles from Git:

  git clone https://github.com/lockss/turtles
  cd turtles

• Create a virtual environment:

  python3 -m venv .venv
  . .venv/bin/activate

  and invoke Turtles as ./turtles from here, or put ./turtles on the PATH and invoke Turtles as turtles from anywhere.

• Install Turtles’ Python dependencies:

  pip3 install [OPTIONS...] --requirement requirements.txt

• To build plugins from plugin sets (build-plugin, release-plugin), configure one or more plugin sets in plugin-sets.yaml (for example ~/.config/turtles/plugin-sets.yaml):

  ---
  kind: Settings
  plugin-sets:
    - /home/userx/lockss-daemon/turtles.yaml

  and configure your plugin signing keystore in settings.yaml (~/.config/turtles/settings.yaml):

  ---
  kind: Settings
  plugin-signing-keystore: /home/userx/secrets/userx-lockss.keystore
  plugin-signing-alias: userx-lockss

• To deploy plugins into plugin registries (deploy-plugin, release-plugins) or manage plugin registries (analyze-registry), configure one or more plugin registries in plugin-registries.yaml (for example ~/.config/turtles/plugin-sets.yaml):

  ---
  kind: Settings
  plugin-registries:
    - /var/www/plugins/turtles.yaml

Examples

Building plugins:

# Help message:
turtles build-plugin --help

# List of plugin identifiers
turtles build-plugin edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ...
# Abbreviation
turtles bp edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ...

# Alternative invocation
turtles build-plugin --identifier=edu.myuniversity.plugin.publisherx.PublisherXPlugin --identifier=edu.myuniversity.plugin.publishery.PublisherYPlugin ...
# Abbreviation
turtles bp -i edu.myuniversity.plugin.publisherx.PublisherXPlugin -i edu.myuniversity.plugin.publishery.PublisherYPlugin ...

# Alternative invocation
# /tmp/pluginids.txt has one plugin identifier per line
turtles build-plugin --identifiers=/tmp/pluginids.txt
# Abbreviation
turtles bp -I /tmp/pluginids.txt

Deploying plugins:

# Help message:
turtles deploy-plugin --help

# List of JARs
# Deploy to 'testing' layer only
turtles deploy-plugin --testing /path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar /path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ...
# Abbreviation
turtles dp -t /path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar /path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ...

# Alternative invocation
# Deploy to 'production' layer only
turtles deploy-plugin --production --jar=/path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar --jar=/path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ...
# Abbreviation
turtles dp -p -j /path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar -j /path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ...

# Alternative invocation
# /tmp/pluginjars.txt has one JAR path per line
# Deploy to both 'testing' and 'production' layers
turtles deploy-plugin --testing --production --jars=/tmp/pluginjars.txt
# Abbreviation
turtles bp -tp -J /tmp/pluginids.txt

Releasing (building and deploying) plugins:

# Help message:
turtles release-plugin --help

# List of plugin identifiers
# Deploy to 'testing' layer only
turtles release-plugin --testing edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ...
# Abbreviation
turtles rp -t edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ...

# Alternative invocation
# Deploy to 'production' layer only
turtles release-plugin --production --identifier=edu.myuniversity.plugin.publisherx.PublisherXPlugin --identifier=edu.myuniversity.plugin.publishery.PublisherYPlugin ...
# Abbreviation
turtles rp -p -i edu.myuniversity.plugin.publisherx.PublisherXPlugin -i edu.myuniversity.plugin.publishery.PublisherYPlugin ...

# Alternative invocation
# /tmp/pluginids.txt has one plugin identifier per line
# Deploy to both 'testing' and 'production' layers
turtles release-plugin --testing --production --identifiers=/tmp/pluginids.txt
# Abbreviation
turtles rp -tp -I /tmp/pluginids.txt

Plugin Sets

A plugin set is a project containing the source code of one or more LOCKSS plugins.

Declaring a Plugin Set

A plugin set is defined in a YAML file, typically named turtles.yaml and found at the root of the project:

---
kind: PluginSet
id: ...
name: ...
builder:
  type: ...
  options: ...
main: ...
test: ...

The contents are described below.

kind

Must be set to PluginSet.

id

A short identifier for the plugin set, for example my-plugin-set.

name

A display name for the plugin set, for example My Plugin Set.

builder

A mapping defining the plugin set’s builder together with options.

type

A plugin set builder type. See Plugin Set Builders below.

options

A mapping of type-specific options for the plugin set builder, if applicable. See Plugin Set Builders below.

main

The path (relative to the root of the project) under which the source code of the plugins can be found. May have a default value for a given builder type.

test

The path (relative to the root of the project) under which the source code of the plugins’ unit tests can be found. May have a default value for a given builder type.

Plugin Set Builders

The following plugin set builder types are supported:

ant

The plugin set builder type ant designates a project using the legacy Ant layout and build file of the LOCKSS Program’s lockss-daemon project.

This builder expects ant load-plugins to compile and verify all plugins, and the scripts test/scripts/jarplugin and test/scripts/signplugin. (These could all become configurable if there are plugin projects out there generally using this builder logic but not matching these assumptions.)

For this builder type, the main and test properties of the PluginSet object default to plugins/src and plugins/test/src respectively.

System prerequisites. This builder requires:

• Java Development Kit 8 (JDK)

• Apache Ant

• The environment variable JAVA_HOME must be set.

Options. This builder does not look for optional configuration information in the options mapping.

mvn

The plugin set builder type mvn designates a project using a Maven layout and inheriting from org.lockss:lockss-plugins-parent-pom.

For this builder type, the main and test properties of the PluginSet object default to the Maven standard src/main/java and src/test/java respectively.

System prerequisites. This builder requires:

• Java Development Kit 8 (JDK)

• Apache Maven

Options. This builder does not look for optional configuration information in the options mapping.

For other types of building strategies out there, more types of builders could be supported, and/or the tool could be extended to allow for custom builder types to be registered.

Plugin Registries

A plugin registry is a structure containing LOCKSS plugins packaged as signed JAR files.

Currently the only predefined structures are directory structures local to the file system, but in the future this could also be Git trees or other structures.

Plugin Registry Layers

A plugin registry consists of one or more layers. Some plugin registries may have only one layer, in which case the LOCKSS boxes in a network using the plugin registry will get what is released to it. Some plugin registries may have two or more layers, with the additional layers used for plugin development or content processing quality assurance.

Plugin layers are sequential in nature; a new version of a plugin is released to the lowest layer first, then to the next layer (after some process), and so on until the highest layer.

Although the identifiers (see id below) and display names (see name below) of plugin registry layers are arbitrary, the highest layer is commonly referred to as the production layer, and when there are exactly two layers, the lower layer is commonly referred to as the testing layer. Turtles reflects this common idiom with built-in --production and --testing options that are shorthand for --layer=production and --layer=testing respectively.

It is possible for multiple plugin registries to have a layer path in common. An example would be a team working on several plugin registries for different purposes, having distinct (public) production layer paths, but sharing a single (internal) testing layer path, if they are the only audience for it.

Declaring a Plugin Registry

A plugin registry is defined in a YAML file:

---
kind: PluginRegistry
id: ...
name: ...
layout:
  type: ...
  options: ...
layers:
  - id: ...
    name: ...
    path: ...
  - ...
plugin-identifiers:
  - ...
  - ...
suppressed-plugin-identifiers:
  - ...
  - ...

The contents are described below.

kind

Must be set to PluginRegistry.

id

A short identifier for the plugin registry, for example my-plugin-registry.

name

A display name for the plugin registry, for example My Plugin Registry.

layout

A mapping defining the plugin registry’s layout together with options.

type

A plugin registry layout type. See Plugin Registry Layouts below.

options

A mapping of type-specific options for the plugin registry layout, if applicable. See Plugin Registry Layouts below.

layers

An ordered list of the plugin registry’s layers. Each list element consists of the following three-element mapping:

id

A short identifier for the plugin registry layer, for example production or testing.

name

A display name for the plugin regisry layer, for example My Plugin Registry Testing Layer or My Plugin Registry (Testing).

path

A directory path where the root of the plugin registry layer can be found.

plugin-identifiers

A list of plugin identifiers contained in the plugin registry.

suppressed-plugin-identifiers

A list of plugin identifiers excluded by the plugin registry.

Turtles does not currently do anything with this information but it could be used to record plugins that have been abandoned or retracted over the lifetime of the plugin registry.

Plugin Registry Layouts

The following plugin registry layout types are supported:

directory

Each layer consists of a directory on the file system where signed plugin JARs are stored, which is then typically served by a Web server. The directory for each layer is designated by the layer’s path property.

System prerequisites. This layout does not have any additional system prerequisites.

Options. This layout does not look for optional configuration information in the options mapping.

rcs

A specialization of the directory type, that also keeps successive versions of a given JAR locally in RCS. The directory for each layer is designated by the layer’s path property as in the directory type, and additionally this layout expects an RCS directory to exist in the layer directory.

System prerequisites. This layout requires:

• RCS

Options. This layout accepts the following options:

layout:
  type: rcs
  options:
    file-naming-convention: ...

file-naming-convention

A rule for what to each deployed JAR file given an original file named after the plugin’s identifier, for example edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar:

abbreviated

Shorten the file name to its last component, for example edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar is deployed as PublisherXPlugin.jar.

full

Default. Use the original file name, unchanged.

Other layout types could be defined to support other uses cases out there, and/or the tool could be extended to allow for custom layout types to be registered.

Configuring

When Turtles looks for a configuration file, it looks in the following directories in sequence until it finds the matching file:

• ${HOME}/.config/turtles

• /etc/turtles

Configuration files are YAML files containing a mapping with kind set to Settings along with whatever data is required by the given configuration file.

settings.yaml

Overview of this file:

---
kind: Settings
plugin-signing-alias: ...
plugin-signing-keystore: ...

If you are using Turtles to build or release plugins (turtles build-plugin or turtles release-plugin commands), you will need to specify the following keys:

plugin-signing-alias

The alias of your plugin signing key.

plugin-signing-keystore

The path of your plugin signing keystore.

plugin-sets.yaml

This configuration file is needed by Turtles when building or releasing plugins (turtles build-plugin or turtles release-plugin commands):

---
kind: Settings
plugin-sets:
  - ...
  - ...

Each entry in the plugin-sets list is the path to a YAML file containing one or more PluginSet definitions.

plugin-registries.yaml

This configuration file is needed by Turtles when deploying or releasing plugins (turtles deploy-plugin or turtles release-plugin commands), and when outputting reports on plugin registries (turtles analyze-registry command):

---
kind: Settings
plugin-registries:
  - ...
  - ...

Each entry in the plugin-registries list is the path to a YAML file containing one or more PluginRegistry definitions.

Using

Help message (turtles --help):

usage: turtles [-h] [--debug-cli] [--non-interactive] [--output-format FMT]
               COMMAND ...

options:
  -h, --help            show this help message and exit
  --debug-cli           print the result of parsing command line arguments
  --non-interactive, -n
                        disallow interactive prompts (default: allow)
  --output-format FMT   set tabular output format to FMT (default: simple;
                        choices: fancy_grid, fancy_outline, github, grid,
                        html, jira, latex, latex_booktabs, latex_longtable,
                        latex_raw, mediawiki, moinmoin, orgtbl, pipe, plain,
                        presto, pretty, psql, rst, simple, textile, tsv,
                        unsafehtml, youtrack)

commands:
  Add --help to see the command's own help message

  COMMAND               DESCRIPTION
    analyze-registry (ar)
                        analyze plugin registries
    build-plugin (bp)   build (package and sign) plugins
    copyright           show copyright and exit
    deploy-plugin (dp)  deploy plugins
    license             show license and exit
    release-plugin (rp)
                        release (build and deploy) plugins
    usage               show detailed usage and exit
    version             show version and exit

turtles analyze-registry

Synonym: turtles ar

Help message (turtles analyze-registry --help):

usage: turtles analyze-registry [-h] [--plugin-registries FILE]
                                [--plugin-sets FILE] [--settings FILE]

Analyze plugin registries

options:
  -h, --help            show this help message and exit
  --plugin-registries FILE
                        load plugin registries from FILE (default:
                        $HOME/.config/turtles/plugin-registries.yaml or
                        /etc/turtles/plugin-registries.yaml)
  --plugin-sets FILE    load plugin sets from FILE (default:
                        $HOME/.config/turtles/plugin-sets.yaml or
                        /etc/turtles/plugin-sets.yaml)
  --settings FILE       load settings from FILE (default:
                        $HOME/.config/turtles/settings.yaml or
                        /etc/turtles/settings.yaml)

turtles build-plugin

Synonym: turtles bp

Help message (turtles build-plugin --help):

usage: turtles build-plugin [-h] [--identifier PLUGID] [--identifiers FILE]
                            [--password PASS] [--plugin-sets FILE]
                            [--settings FILE]
                            [PLUGID ...]

Build (package and sign) plugins

positional arguments:
  PLUGID                plugin identifier to build

options:
  -h, --help            show this help message and exit
  --identifier PLUGID, -i PLUGID
                        add PLUGID to the list of plugin identifiers to build
  --identifiers FILE, -I FILE
                        add the plugin identifiers in FILE to the list of
                        plugin identifiers to build
  --password PASS       set the plugin signing password
  --plugin-sets FILE    load plugin sets from FILE (default:
                        $HOME/.config/turtles/plugin-sets.yaml or
                        /etc/turtles/plugin-sets.yaml)
  --settings FILE       load settings from FILE (default:
                        $HOME/.config/turtles/settings.yaml or
                        /etc/turtles/settings.yaml)

turtles deploy-plugin

Synonym: turtles dp

Help message (turtles deploy-plugin --help):

usage: turtles deploy-plugin [-h] [--jar PLUGJAR] [--jars FILE]
                             [--layer LAYER] [--layers FILE]
                             [--plugin-registries FILE] [--production]
                             [--testing]
                             [PLUGJAR ...]

Deploy plugins

positional arguments:
  PLUGJAR               plugin JAR to deploy

options:
  -h, --help            show this help message and exit
  --jar PLUGJAR, -j PLUGJAR
                        add PLUGJAR to the list of plugin JARs to deploy
  --jars FILE, -J FILE  add the plugin JARs in FILE to the list of plugin JARs
                        to deploy
  --layer LAYER, -l LAYER
                        add LAYER to the list of plugin registry layers to
                        process
  --layers FILE, -L FILE
                        add the layers in FILE to the list of plugin registry
                        layers to process
  --plugin-registries FILE
                        load plugin registries from FILE (default:
                        $HOME/.config/turtles/plugin-registries.yaml or
                        /etc/turtles/plugin-registries.yaml)
  --production, -p      synonym for --layer=production (i.e. add 'production'
                        to the list of plugin registry layers to process)
  --testing, -t         synonym for --layer=testing (i.e. add 'testing' to the
                        list of plugin registry layers to process)

turtles release-plugin

Synonym: turtles rp

Help message (turtles release-plugin --help):

usage: turtles release-plugin [-h] [--identifier PLUGID] [--identifiers FILE]
                              [--layer LAYER] [--layers FILE]
                              [--password PASS] [--plugin-registries FILE]
                              [--plugin-sets FILE] [--production]
                              [--settings FILE] [--testing]
                              [PLUGID ...]

Release (build and deploy) plugins

positional arguments:
  PLUGID                plugin identifier to build

options:
  -h, --help            show this help message and exit
  --identifier PLUGID, -i PLUGID
                        add PLUGID to the list of plugin identifiers to build
  --identifiers FILE, -I FILE
                        add the plugin identifiers in FILE to the list of
                        plugin identifiers to build
  --layer LAYER, -l LAYER
                        add LAYER to the list of plugin registry layers to
                        process
  --layers FILE, -L FILE
                        add the layers in FILE to the list of plugin registry
                        layers to process
  --password PASS       set the plugin signing password
  --plugin-registries FILE
                        load plugin registries from FILE (default:
                        $HOME/.config/turtles/plugin-registries.yaml or
                        /etc/turtles/plugin-registries.yaml)
  --plugin-sets FILE    load plugin sets from FILE (default:
                        $HOME/.config/turtles/plugin-sets.yaml or
                        /etc/turtles/plugin-sets.yaml)
  --production, -p      synonym for --layer=production (i.e. add 'production'
                        to the list of plugin registry layers to process)
  --settings FILE       load settings from FILE (default:
                        $HOME/.config/turtles/settings.yaml or
                        /etc/turtles/settings.yaml)
  --testing, -t         synonym for --layer=testing (i.e. add 'testing' to the
                        list of plugin registry layers to process)

Tabular Output Format

Turtles’ tabular output is performed by the tabulate library through the --output-format option. See https://github.com/astanin/python-tabulate#table-format for a visual reference of the various output formats available. The default is simple.