Tool to manage LOCKSS plugin sets and LOCKSS plugin registries
Reason this release was yanked:
0.3.1 release
Project description
Turtles is a tool to manage LOCKSS plugin sets and LOCKSS plugin registries.
Latest release: 0.3.1-dev1 (2023-03-07)
Prerequisites
Python 3.7 or greater.
Additional prerequisites depending on the types of Plugin Set Builders or Plugin Registry Layouts in use (see their respective requirements).
Prerequisites for development work only:
Poetry 1.4 or greater.
Installation
Turtles is available from the Python Package Index (PyPI) as lockss-turtles: https://pypi.org/project/lockss-turtles
You can install it with pip. To install it in a virtual environment, simply use:
pip3 install lockss-turtles
To install it in your own non-root, non-virtual environment, use the --user option:
pip3 install --user lockss-turtles
The installation process adds a turtles executable to the PATH; you can check that the installation is functional by running turtles version or turtles --help.
Invocation
Turtles is invoked at the command line as:
turtles
or as a Python module:
python3 -m lockss.turtles
Help messages and this document use turtles throughout, but the two invocation styles are interchangeable.
Turtles uses Commands, in the style of programs like git, dnf/yum, apt/apt-get, and the like. You can see the list of available Commands by invoking turtles --help.
Configuration Files
All Turtles configuration files are YAML files and have a top-level kind key that declares what kind of configuration file it is.
Some Turtles commands require a Plugin Set Catalog, Plugin Registry Catalog, or Plugin Signing Credentials. You can specify the configuration file to use via the appropriate command line option (for example --plugin-set-catalog/-s). Otherwise, Turtles will look for then appropriate configuration file (for example plugin-set-catalog.yaml) in several standard directories, in this order:
$XDG_CONFIG_HOME/lockss.turtles (by default $HOME/.config/lockss.turtles)
/usr/local/share/lockss.turtles
/etc/lockss.turtles
Building Plugins
To use Turtles to build (package and sign) LOCKSS plugins, you will need to define one or more Plugin Sets, declare them in a Plugin Set Catalog, and declare your Plugin Signing Credentials. You then use the build-plugin command, or the release-plugin command (which amounts to build-plugin followed by deploy-plugin).
Deploying Plugins
To use Turtles to deploy LOCKSS plugins to LOCKSS plugin registries, you will need to define one or more Plugin Registries and declare them in a Plugin Registry Catalog. You then use the deploy-plugin command, or the release-plugin command (which amounts to build-plugin followed by deploy-plugin).
Plugin Set Catalog
Turtles commands that are Building Plugins (build-plugin, release-plugin) need a plugin set catalog. It can be specified via the --plugin-set-catalog/-s option, otherwise Turtles looks through Configuration Files for a file named plugin-set-catalog.yaml.
A plugin set catalog is defined by a YAML document:
--- kind: PluginSetCatalog plugin-set-files: - /path/to/some/file1.yaml - /path/to/another/file2.yaml - ...
The contents are described below:
- kind
Required. Must be set to PluginSetCatalog.
- plugin-set-files
Required. A list of one or more paths to Plugin Sets.
Plugin Sets
A plugin set is a project containing the source code of one or more LOCKSS plugins.
A plugin set is defined by a YAML document:
--- kind: PluginSet id: mypluginset name: My Plugin Set builder: type: ... ...
The contents are described below:
- kind
Required. Must be set to PluginSet.
- id
Required. A short identifier for the plugin set, for example mypluginset.
- name
Required. A display name for the plugin set, for example My Plugin Set.
- builder
Required. An object defining the plugin set’s builder together with its options.
- type
Required. A plugin set builder type. See Plugin Set Builders below.
- Other
Additional properties depending on the plugin set builder type. See Plugin Set Builders below.
Plugin Set Builders
Turtles Plugin Sets support two types of plugin set builders:
Maven Plugin Set Builder
This type of plugin set builder is for a Maven project inheriting from org.lockss:lockss-plugins-parent-pom.
Prerequisites
Java Development Kit 8 (JDK).
Declaration
For this plugin set builder type, the builder object in the plugin set definition has the following structure:
- type
Required. Must be set to mvn.
- main
Optional. The path (relative to the root of the project) to the plugins’ source code. Default: src/main/java.
- test
Optional. The path (relative to the root of the project) to the plugins’ unit tests. Default: src/test/java.
Ant Plugin Set Builder
This type of plugin set builder is for the LOCKSS 1.x (https://github.com/lockss/lockss-daemon) code tree, based on Ant.
Prerequisites
Java Development Kit 8 (JDK).
JAVA_HOME must be set appropriately.
Declaration
For this plugin set builder type, the builder object in the plugin set definition has the following structure:
- type
Required. Must be set to ant.
- main
Optional. The path (relative to the root of the project) to the plugins’ source code. Default: plugins/src.
- test
Optional. The path (relative to the root of the project) to the plugins’ unit tests. Default: plugins/test/src.
Plugin Registry Catalog
Turtles commands that are Deploying Plugins (deploy-plugin, release-plugin) need a plugin registry catalog. It can be specified via the --plugin-registry-catalog/-r option, otherwise Turtles looks through Configuration Files for a file named plugin-registry-catalog.yaml.
A plugin set catalog is defined by a YAML document:
--- kind: PluginRegistryCatalog plugin-registry-files: - /path/to/some/file1.yaml - /path/to/another/file2.yaml - ...
The contents are described below:
- kind
Required. Must be set to PluginRegistryCatalog.
- plugin-registry-files
Required. A list of one or more paths to Plugin Registries.
Plugin Registries
A plugin registry is a structure containing LOCKSS plugins packaged as signed JAR files.
Currently the only predefined structures are directory structures local to the file system, which are then typically served by a Web server.
A plugin registry is defined by a YAML document:
---
kind: PluginRegistry
id: mypluginregistry
name: My Plugin Registry
layout:
type: ...
...
layers:
- id: testing
name: My Plugin Registry (Testing)
path: /path/to/testing
- id: production
name: My Plugin Registry (Production)
path: /path/to/production
- ...
plugin-identifiers:
- edu.myuniversity.plugin.publisherx.PublisherXPlugin
- edu.myuniversity.plugin.publishery.PublisherYPlugin
- ...
suppressed-plugin-identifiers:
- edu.myuniversity.plugin.old.OldPlugin
- ...
The contents are described below:
- kind
Required. Must be set to PluginRegistry.
- id
Required. A short identifier for the plugin registry, for example mypluginregistry.
- name
Required. A display name for the plugin registry, for example My Plugin Registry.
- layout
Required. An object defining the plugin registry’s layout together with its options.
- type
Required. A plugin registry layout type. See Plugin Registry Layouts below.
- Other
Additional properties depending on the plugin registry layout type. See Plugin Registry Layouts below.
- layers
Required. A list of objects describing the layers of the plugin registry. See Plugin Registry Layers below.
- plugin-identifiers
Required. Non-empty list of the plugin identifiers in this plugin registry.
- plugin-identifiers
Optional. Non-empty list of plugin identifiers that are excluded from this plugin registry.
Turtles does not currently do anything with this information, but it can be used to record plugins that have been abandoned or retracted over the lifetime of the plugin registry.
Plugin Registry Layouts
Turtles supports two kinds of plugin registry layouts:
Directory Plugin Registry Layout
In this type of plugin registry layout, each layer consists of a directory on the local file system where signed plugin JARs are stored, which is then typically served by a Web server. The directory for each layer is designated by the layer’s path property.
Prerequisites
None.
Declaration
For this plugin registry layout type, the layout object in the plugin registry definition has the following structure:
- type
Required. Must be set to directory.
Currently, this layout type does not support the file-naming-convention option of the RCS Plugin Registry Layout, but this feature will be introduced in a future release.
RCS Plugin Registry Layout
In this specialization of the Directory Plugin Registry Layout, each successive version of a given JAR is kept locally in RCS.
Prerequisites
Declaration
For this plugin registry layout type, the layout object in the plugin registry definition has the following structure:
- type
Required. Must be set to rcs.
- file-naming-convention
Optional. A rule for what to name each deployed JAR file. If unspecified, the behavior is that of identifier. Can be one of:
identifier: Use the plugin identifier and add .jar. For example edu.myuniversity.plugin.publisherx.PublisherXPlugin results in edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar.
abbreviated: Use the last component of the plugin identifier and add .jar. For example edu.myuniversity.plugin.publisherx.PublisherXPlugin results in PublisherXPlugin.jar.
Plugin Registry Layers
A plugin registry consists of one or more layers.
Some plugin registries only one layer, in which case the LOCKSS boxes in a network using the plugin registry will get what is released to it. Some plugin registries may have two or more layers, with the additional layers used for plugin development or content processing quality assurance.
Layers are sequential in nature; a new version of a plugin is released to the lowest layer first, then to the next layer (after some process), and so on until the highest layer. This sequencing is reflected in the ordering of the layers list in the plugin registry definition.
Although the identifiers (see id below) and display names (see name below) of plugin registry layers are arbitrary, the highest layer is commonly referred to as the production layer, and when there are exactly two layers, the lower layer is commonly referred to as the testing layer. Turtles reflects this common idiom with built-in --production/-p and --testing/-t options that are shorthand for --layer=production and --layer=testing respectively.
It is possible for multiple plugin registries to have a layer path in common. An example would be a team working on several plugin registries for different purposes, having distinct (public) production layer paths, but sharing a single (internal) testing layer path, if they are the only audience for it.
A plugin registry layer is defined as one of the objects in the plugin registry definition’s layers list. Each layer object has the following structure:
- id
Required. A short identifier for the plugin registry layer, for example testing.
- name
Required. A display name for the plugin registry layer, for example My Plugin Registry (Testing).
- path
Required. The local path to the root of the plugin registry layer, for example /path/to/testing.
Plugin Signing Credentials
Turtles commands that are Building Plugins (build-plugin, release-plugin) need a reference to plugin signing credentials. They can be specified via the --plugin-signing-credentials/-c option, otherwise Turtles looks through Configuration Files for a file named plugin-signing-credentials.yaml.
Plugin signing credentials are defined by a YAML document:
--- kind: PluginSigningCredentials plugin-signing-keystore: /path/to/myalias.keystore plugin-signing-alias: myalias
The contents are described below:
- kind
Required. Must be set to PluginSigningCredentials.
- plugin-signing-keystore
Required. Path to the plugin signing key (keystore).
- plugin-signing-alias
Required. The alias to use, which must be that of the plugin signing key (keystore) and also found in the LOCKSS network’s shared keystore.
Commands
The available commands are:
build-plugin (bp): build (package and sign) plugins
copyright: show copyright and exit
deploy-plugin (dp): deploy plugins
license: show license and exit
release-plugin (rp): release (build and deploy) plugins
usage: show detailed usage and exit
version: show version and exit
Top-Level Command
The top-level executable alone does not perform any action or default to a given command. It does define a few options, which you can see by invoking Turtles with the --help/-h option:
usage: turtles [-h] [--debug-cli] [--non-interactive] [--output-format FMT]
COMMAND ...
options:
-h, --help show this help message and exit
--debug-cli print the result of parsing command line arguments
--non-interactive, -n
disallow interactive prompts (default: allow)
--output-format FMT set tabular output format to FMT (default: simple;
choices: asciidoc, double_grid, double_outline,
fancy_grid, fancy_outline, github, grid, heavy_grid,
heavy_outline, html, jira, latex, latex_booktabs,
latex_longtable, latex_raw, mediawiki, mixed_grid,
mixed_outline, moinmoin, orgtbl, outline, pipe, plain,
presto, pretty, psql, rounded_grid, rounded_outline,
rst, simple, simple_grid, simple_outline, textile,
tsv, unsafehtml, youtrack)
Turtles’ tabular output is performed by the tabulate library through the --output-format option. See its PyPI page for a visual reference of the various output formats available. The default is simple.
build-plugin (bp)
The build-plugin command is used for Building Plugins. It has its own --help/-h option:
usage: turtles build-plugin [-h] [--identifier PLUGID] [--identifiers FILE]
[--password PASS] [--plugin-set-catalog FILE]
[--plugin-signing-credentials FILE]
[PLUGID ...]
Build (package and sign) plugins
positional arguments:
PLUGID plugin identifier to build
options:
-h, --help show this help message and exit
--identifier PLUGID, -i PLUGID
add PLUGID to the list of plugin identifiers to build
--identifiers FILE, -I FILE
add the plugin identifiers in FILE to the list of
plugin identifiers to build
--password PASS set the plugin signing password
--plugin-set-catalog FILE, -s FILE
load plugin set catalog from FILE (default:
$HOME/.config/lockss.turtles/plugin-set-
catalog.yaml or
/usr/local/share/lockss.turtles/plugin-set-
catalog.yaml or /etc/lockss.turtles/plugin-set-
catalog.yaml)
--plugin-signing-credentials FILE, -c FILE
load plugin signing credentials from FILE (default:
$HOME/.config/lockss.turtles/plugin-signing-
credentials.yaml or
/usr/local/share/lockss.turtles/plugin-signing-
credentials.yaml or /etc/lockss.turtles/plugin-
signing-credentials.yaml)
The command needs:
Plugin Signing Credentials, either via the --plugin-signing-credentials/-c option or from the Configuration Files.
A Plugin Set Catalog, either via the --plugin-set-catalog/-s option or from the Configuration Files.
One or more plugin identifiers. The list of plugin identifiers to process is derived from:
The identifiers listed as bare arguments to the command.
The identifiers listed as --identifier/-i options.
The identifiers found in the files listed as --identifiers/-I options.
Examples:
# Help message turtles build-plugin --help # Abbreviation turtles bp -h # List of plugin identifiers turtles build-plugin edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Abbreviation turtles bp edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Alternative invocation turtles build-plugin --identifier=edu.myuniversity.plugin.publisherx.PublisherXPlugin --identifier=edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Abbreviation turtles bp -i edu.myuniversity.plugin.publisherx.PublisherXPlugin -i edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Alternative invocation # /tmp/pluginids.txt has one plugin identifier per line turtles build-plugin --identifiers=/tmp/pluginids.txt # Abbreviation turtles bp -I /tmp/pluginids.txt
copyright
The copyright command displays the copyright notice for Turtles and exits.
deploy-plugin (dp)
The deploy-plugin command is used for Deploying Plugins. It has its own --help/-h option:
usage: turtles deploy-plugin [-h] [--jar PLUGJAR] [--jars FILE]
[--layer LAYER] [--layers FILE]
[--plugin-registry-catalog FILE] [--production]
[--testing]
[PLUGJAR ...]
Deploy plugins
positional arguments:
PLUGJAR plugin JAR to deploy
options:
-h, --help show this help message and exit
--jar PLUGJAR, -j PLUGJAR
add PLUGJAR to the list of plugin JARs to deploy
--jars FILE, -J FILE add the plugin JARs in FILE to the list of plugin JARs
to deploy
--layer LAYER, -l LAYER
add LAYER to the list of plugin registry layers to
process
--layers FILE, -L FILE
add the layers in FILE to the list of plugin registry
layers to process
--plugin-registry-catalog FILE, -r FILE
load plugin registry catalog from FILE (default:
$HOME/.config/lockss.turtles/plugin-registry-
catalog.yaml or
/usr/local/share/lockss.turtles/plugin-registry-
catalog.yaml or /etc/lockss.turtles/plugin-registry-
catalog.yaml)
--production, -p synonym for --layer=production (i.e. add 'production'
to the list of plugin registry layers to process)
--testing, -t synonym for --layer=testing (i.e. add 'testing' to the
list of plugin registry layers to process)
The command needs:
A Plugin Registry Catalog, either via the --plugin-registry-catalog/-r option or from the Configuration Files.
One or more JAR paths. The list of paths to process is derived from:
The paths listed as bare arguments to the command.
The paths listed as --jar/-j options.
The paths found in the files listed as --jars/-J options.
One or more plugin registry layer IDs. The list of layer IDs to process is derived from:
The layer IDs listed as --layer/-l options.
The layer IDs found in the files listed as --layers/-L options.
Alternatively:
--testing/-t is a synonym for --layer=testing.
--production/-p is a synonym for --layer=production.
Examples:
# Help message turtles deploy-plugin --help # Abbreviation turtles dp -h # List of JARs # Deploy to 'testing' layer only turtles deploy-plugin --testing /path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar /path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ... # Abbreviation turtles dp -t /path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar /path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ... # Alternative invocation # Deploy to 'production' layer only turtles deploy-plugin --production --jar=/path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar --jar=/path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ... # Abbreviation turtles dp -p -j /path/to/edu.myuniversity.plugin.publisherx.PublisherXPlugin.jar -j /path/to/edu.myuniversity.plugin.publishery.PublisherYPlugin.jar ... # Alternative invocation # /tmp/pluginjars.txt has one JAR path per line # Deploy to both 'testing' and 'production' layers turtles deploy-plugin --testing --production --jars=/tmp/pluginjars.txt # Abbreviation turtles bp -tp -J /tmp/pluginids.txt
license
The license command displays the license terms for Turtles and exits.
release-plugin (rp)
The release-plugin command is used for Building Plugins and Deploying Plugins, being essentially build-plugin followed by deploy-plugin. It has its own --help/-h option:
usage: turtles release-plugin [-h] [--identifier PLUGID] [--identifiers FILE]
[--layer LAYER] [--layers FILE]
[--password PASS]
[--plugin-registry-catalog FILE]
[--plugin-set-catalog FILE]
[--plugin-signing-credentials FILE]
[--production] [--testing]
[PLUGID ...]
Release (build and deploy) plugins
positional arguments:
PLUGID plugin identifier to build
options:
-h, --help show this help message and exit
--identifier PLUGID, -i PLUGID
add PLUGID to the list of plugin identifiers to build
--identifiers FILE, -I FILE
add the plugin identifiers in FILE to the list of
plugin identifiers to build
--layer LAYER, -l LAYER
add LAYER to the list of plugin registry layers to
process
--layers FILE, -L FILE
add the layers in FILE to the list of plugin registry
layers to process
--password PASS set the plugin signing password
--plugin-registry-catalog FILE, -r FILE
load plugin registry catalog from FILE (default:
$HOME/.config/lockss.turtles/plugin-registry-
catalog.yaml or
/usr/local/share/lockss.turtles/plugin-registry-
catalog.yaml or /etc/lockss.turtles/plugin-registry-
catalog.yaml)
--plugin-set-catalog FILE, -s FILE
load plugin set catalog from FILE (default:
$HOME/.config/lockss.turtles/plugin-set-
catalog.yaml or
/usr/local/share/lockss.turtles/plugin-set-
catalog.yaml or /etc/lockss.turtles/plugin-set-
catalog.yaml)
--plugin-signing-credentials FILE, -c FILE
load plugin signing credentials from FILE (default:
$HOME/.config/lockss.turtles/plugin-signing-
credentials.yaml or
/usr/local/share/lockss.turtles/plugin-signing-
credentials.yaml or /etc/lockss.turtles/plugin-
signing-credentials.yaml)
--production, -p synonym for --layer=production (i.e. add 'production'
to the list of plugin registry layers to process)
--testing, -t synonym for --layer=testing (i.e. add 'testing' to the
list of plugin registry layers to process)
The command needs:
Plugin Signing Credentials, either via the --plugin-signing-credentials/-c option or from the Configuration Files.
A Plugin Set Catalog, either via the --plugin-set-catalog/-s option or from the Configuration Files.
One or more plugin identifiers. The list of plugin identifiers to process is derived from:
The identifiers listed as bare arguments to the command.
The identifiers listed as --identifier/-i options.
The identifiers found in the files listed as --identifiers/-I options.
A Plugin Registry Catalog, either via the --plugin-registry-catalog/-r option or from the Configuration Files.
One or more plugin registry layer IDs. The list of layer IDs to process is derived from:
The layer IDs listed as --layer/-l options.
The layer IDs found in the files listed as --layers/-L options.
Alternatively:
--testing/-t is a synonym for --layer=testing.
--production/-p is a synonym for --layer=production.
Examples:
# Help message turtles release-plugin --help # Abbreviation turtles rp -h # List of plugin identifiers # Deploy to 'testing' layer only turtles release-plugin --testing edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Abbreviation turtles rp -t edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Alternative invocation # Deploy to 'production' layer only turtles release-plugin --production --identifier=edu.myuniversity.plugin.publisherx.PublisherXPlugin --identifier=edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Abbreviation turtles rp -p -i edu.myuniversity.plugin.publisherx.PublisherXPlugin -i edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Alternative invocation # /tmp/pluginids.txt has one plugin identifier per line # Deploy to both 'testing' and 'production' layers turtles release-plugin --testing --production --identifiers=/tmp/pluginids.txt # Abbreviation turtles rp -tp -I /tmp/pluginids.txt
usage
The version command displays the usage message of all the Turtles Commands.
version
The version command displays the version number of Turtles and exits.
Examples
Releasing (building and deploying) plugins:
# Help message: turtles release-plugin --help # List of plugin identifiers # Deploy to 'testing' layer only turtles release-plugin --testing edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Abbreviation turtles rp -t edu.myuniversity.plugin.publisherx.PublisherXPlugin edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Alternative invocation # Deploy to 'production' layer only turtles release-plugin --production --identifier=edu.myuniversity.plugin.publisherx.PublisherXPlugin --identifier=edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Abbreviation turtles rp -p -i edu.myuniversity.plugin.publisherx.PublisherXPlugin -i edu.myuniversity.plugin.publishery.PublisherYPlugin ... # Alternative invocation # /tmp/pluginids.txt has one plugin identifier per line # Deploy to both 'testing' and 'production' layers turtles release-plugin --testing --production --identifiers=/tmp/pluginids.txt # Abbreviation turtles rp -tp -I /tmp/pluginids.txt
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for lockss_turtles-0.3.1.dev1.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 21b3306bf661d0d5889a1c2b4625afc94f9ae69eddd2c5bc0ec554ac2475fb64 |
|
| MD5 | 7668459238592c1d17d0f1c6be97be88 |
|
| BLAKE2b-256 | 09c2dbb0e7ef9ea97764160e1f5caba1de2589b4504ee34b3257c07e3e09f193 |
Hashes for lockss_turtles-0.3.1.dev1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b2a6a69c4a2786c3eed045f3c404d0a4173635ffff92c4eb028efeaf88a3fc6e |
|
| MD5 | 24413db00d2fde044aaabacb946082d1 |
|
| BLAKE2b-256 | f2b4cdbdaa0ba52353c8b8b5f0cf2f26ccc633c97eab4f3b100c8a1ea97e4e5c |
