description-harvester 0.7.1

A tool for working with archival description for public access.

description_harvester

A tool for working with archival description for public access. description_harvester reads archival description into a minimalist data model for public-facing archival description and then converts it to the ArcLight data model and POSTs it into an ArcLight Solr index using PySolr.

description_harvester is designed to be extensible and harvest archival description from a number of sources. Currently the only available sources harvests data from the ArchivesSpace API using ArchivesSnake or EAD 2002 XML files. Its also possible to add additional output modules to serialize description to EAD or other formats in addition to or in replace of sending description to an ArcLight Solr instance. This potential opens up new possibilities of managing description using low-barrier formats and tools.

description_harvester is designed to be a drop-in replacement for the ArcLight Traject indexer. It also includes a plugin that attempts to recognized IIIF manifests included as file versions and uses manifests to fully index digital objects from digital repositories and other sources, including item-level metadata fields, embedded text, OCR text, and transcriptions.

This is still a bit drafty, as its only tested on ASpace v2.8.0 and needs tests and better error handling. Validation is also very minimal, but there is potential to add detailed validation with jsonschema .

Installation

pip install description_harvester

First, you need to configure ArchivesSnake by creating a ~/.archivessnake.ymlfile with your API credentials as detailed by the ArchivesSnake configuration docs.

Next, you also need a ~/.description_harvester/config.yml file that lists your Solr URL and the core you want to index to. These can also be overridden with args. description_harvester reads your config.yml as utf-8, so if you're creating this file in a Windows environment you should ensure its utf-8.

solr_url: http://127.0.0.1:8983/solr
solr_core: blacklight-core
last_query: 0
cache_expiration: 3600

Adding custom digital object metadata

You can also add custom digital object metadata fields by adding them to your config.yml under the Solr suffix you would like them to be indexed as. These fields must match metadata fields in your IIIF manifests.

metadata:
- ssi:
  - date_uploaded
- ssm:
  - date_digitized
  - extent
- ssim:
  - legacy_id
  - resource_type
  - coverage
  - preservation_package
  - creator
  - contributor
  - preservation_format
  - source
- tesm:
  - processing_activity
- tesim:
  - description

Repositories

By default, when reading from ArchivesSpace, description harvester will use the repository name stored there.

To enable the --repo argument, place a copy of your ArcLight repositories.yml file as ~/.description_harvester/repositories.yml. You can then use harvest --id mss001 --repo slug to index using the slug from repositories.yml. This will overrite the ArchivesSpace repository name.

There is also the option do customize this with a plugin.

Encoding note: While ArcLight does not explicitly read repositories.yml as utf-8, its Rails stack means that you're likely reading it in a utf-8 (non-Windows) environment. Since description_harvester enables you to index from a Windows machine, it expects your ~/.description_harvester/repositories.yml file to be utf-8.

Indexing from ArchivesSpace API to ArcLight

Once description_harvester is set up, you can index from the ASpace API to ArcLight using the to-ArcLight command.

Index by id_0

You can provide one or more IDs to index using a resource's id_0` field

harvest --id ua807

harvest --id mss123 apap106

Index by URI

You can also use integers from ASpace URIs for resource, such as 263 for https://my.aspace.edu/resources/263

harvest --uri 435

harvest --uri 1 755

Indexing by modified time

Index collections modified in the past hour: harvest --hour

Index collections modified in the past day: harvest --today

Index collections modified since las run: harvest --updated

Index collections not already in the index: harvest --new

Indexing from EAD 2002

harvest --ead path/to/ead.xml

You can also give it a directory and it will harvest all *.xml

harvest --ead path/to/ead_files

Verbose output

harvest --id ger071 -v

Caching

description_harvester will cache collections from the ArchivesSpace API, storing them by default to ~/.description_harvester/cache after they are converted to the description model. Cache time is set in seconds as cache_expiration in ~/.description_harvester/config.yml. Thus, cache_expiration: 3600 will use the cached data instead of the ArchivesSpace API for data less than 1 hour old.

You can override the cache path in config or turn caching off gobally with cache_dir: false.

cache_dir: "~/path/to/my_cache"
cache_dir: "C:/Users/username/my_cache"
cache_dir: false

Deleting collections

You can delete one or more collections using the --delete argument. This uses the Solr document ID, such as apap106 for https://my.ArcLight.edu/catalog/apap106.

harvest --delete apap101 apap301

Plugins

Local implementations may have to override some description_harvester logic. Indexing digital objects from local systems may be a common use case.

To create a plugin, create a plugin directory, either at ~/.description_harvester or a path you pass with a DESCRIPTION_HARVESTER_PLUGIN_DIR environment variable.

Use the example default.py and make a copy in your plugin directory.

Use custom_repository() to customize how repository names are set. This has access to an ArchivesSpace resource API object.

Use read_data() to customize DigitalObject objects.

The plugin importer will first import plugins from within the package, second it will look in ~/.description_harvester, and finally it will look in the DESCRIPTION_HARVESTER_PLUGIN_DIR path.

Use as a library

You can also use description harvester in a script

from description_harvester import harvest

harvest(["--id", "myid001"])