Creates manifests for syncd works.
Project description
volume-manifest-tool
Intent
This project originated as a script to extract image dimensions from a work, and:
- write the dimensions to a json file
- report on images which broke certain rules.
Implementation
Archival Operations determined that this would be most useful to BUDA to implement as a service which could be injected into the current sync process. To do this, the system needed to:
- be more modular
- be distributable onto an instance which could be cloned in AWS.
This branch expands the original tool by:
- Adding the ability to use the eXist db as a source for the image dimensions.
- Use a pre-built BOM Bill of Materials) to derive the files which should be included in the dimesnsions file
- Read input from an S3 device
- Create and save log files.
- Manage input files.
- Run as a service on a Linux platform
Standalone tool
Internal tool to create json manifests for volumes present in S3 for the IIIF presentation API server.
Dependencies
Language
Python 3.6 or newer. It is highly recommended to use pip
to install, to manage dependencies. If you must do it yourself, you can refer to setup.py
for the dependency list.
Environment
- Write access to
/var/log/VolumeManifestTool
which must exist. systemctl
service management, if you want to use the existing materials to install as a service.
Installation
PIP
PyPI contains volume-manifest-tool
Install is simply
pip install volume-manifest-tool
When you install volume-manifest-tool
two entry points are defined in /usr/local/bin
:
manifestforwork
the commad modemanifestFromS3
the mode which runs continuously, polling an S3 resource for a file, and processing all the files it finds.
Development
volume-manifest-tool
is hosted on BUDA Github volume-manifest-tool
- Credentials: you must have the input credentials for a specific AWS user installed to deposit into the archives.
Building a distribution
Be sure to check PyPI for current release, and update accoringly. Use PEP440 for naming releases.
python3 setup.py bdist_wheel
twine upload dist/<thing you built
Usage
volume-manifest-tool
has two modes:
- command line, which allows using a list of workRIDS on a local system
- service, which continually polls a well-known location,
s3://manifest.bdrc.org/processing/todo/
for a file.
Command line mode
$ manifestforwork -h
usage: manifestforwork sourcefile.
Prepares an inventory of image dimensions
positional arguments:
sourceFile File containing one RID per line.
optional arguments:
-h, --help show this help message and exit
-l {info,warning,error,debug,critical}, --loglevel {info,warning,error,debug,critical}
-i POLL_INTERVAL, --interval POLL_INTERVAL
Seconds between alerts for file.
Note that -i
is disregarded in this mode.
Local disk input
Prepare a file listing one RID per line (no bdr:
prefix), let's say it's on /path/to/file
and run:
manifestforwork /path/to/file
S3 input
- Upload the input list to s3://manifest.bdrc.org/processing/todo/
- run
manifestFromS3 -i n [ -l {info,debug,error}
from the command line. See above for argument explanations
manifestFromS3
does the following:
- Moves the desginated input file from
s3://manifest.bdrc.org/processing/input
to.../processing/inprocess
and changes the name from to - Runs the processing, uploading a dimensions.json file for each volume in series.
- When complete, it moves the file from
.../processing/inprocess
to../processing/done
Logging
All messages are output to /var/log/VolumeManifestTool/
Output
volume-manifest-tool
also probes images for errors, which it writes into its working directory. This functionality largely duplicates other tools, so it is not documented further.
Service
See Service Readme for details on installing manifestFromS3 as a service on systemctl
supporting platforms.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for volume_manifest_tool-1.0a3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70fa3eb77b4df7fdd0c63806a0ffe59dbc27190991d986aa31f8442396d29b95 |
|
MD5 | 32c6b992948a137260aa679ba7ecdf74 |
|
BLAKE2b-256 | 66d17d23e66427ba889fb094197fbef2149e0275b80c90f068c3c7d3af226e7b |