IIIFingest 1.0.0

A wrapper library for interacting with the MPS ingest system for IIIF resources

IIIF LTS Library

This is a Python library which facilitates interacting with the Harvard LTS (Library Technology Services) media ingest solution, which takes images and metadata and serves them via IIIF at scale. IIIFingest helps other applications manage ingest credentials, upload images to S3 for ingest, create IIIF manifests, create asset ingest requests, generate JWT tokens for signing ingest requests, and track the status of ingest jobs.

Getting Started

Installation

pip install git+https://github.com/Harvard-ATG/lts-iiif-ingest-service.git

or

pip install IIIFingest

Using the library

from IIIFingest.auth import Credentials
from IIIFingest.client import Client

# Configure ingest API auth credentials
jwt_creds = Credentials(
    issuer="atissuer",
    kid="atissuerdefault",
    private_key_path="path/to/private.key"
)

# Configure ingest API client 
client = Client(
    space="atspace",      # space registered to an MPS account
    namespace="at",       # NRS namespace for assets and manifests
    environment="qa",     # must be on VPN for non-prod (or whitelisted)
    asset_prefix="myapp", # used with asset IDs
    jwt_creds=jwt_creds,
    boto_session=None,
)

# Call client methods as needed
assets = client.upload(...) 
manifest = client.create_manifest(...)
result = client.ingest(...)
status = client.jobstatus(...)

Documentation & References

See the following Media Presentation Service (MPS) documentation for more details:

• Authentication & Authorization
• Ingest API

Development

Getting setup

Clone repo:

$ git clone [e-mail]:Harvard-ATG/lts-iiif-ingest-service.git
$ cd lts-iiif-ingest-service

Setup python environment:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

Optionally, to run linting / precommit hooks: $ pip install -r requirements-dev.txt

Install pre-commit hooks:

This installs pre-commit git hooks to automatically lint and format code before committing. See the .pre-commit-config.yaml for specifics.

$ pre-commit install

Testing

To run an end-to-end test:

$ python3 tests/test_client.py -i atmediamanager -s atmediamanager -e qa --asset-prefix mcih

To run unit tests:

$ python3 -m unittest discover -s ./tests

PyPi

$ pip install twine
$ python setup.py sdist bdist_wheel
$ tar tzf dist IIIFingest-VERSION-.tar.gz
$ twine check dist/*
$ twine upload --repository-url https://pypi.python.org/pypi dist/IIIFingest-VERSION*

Auth

For local development, you can store tokens in /auth which is ignored (). For production environments, use environment variables injected via SSM or other secret management techniques. You can pass either private_key_path (e.g. to one of the private.keys below) or private_key_string (stringified environment variable) to a Credentials class instance; Client takes a boto3 session.

File hierarchy:

• auth
  • dev
    • issuers
      • atmch.json
      • atmediamanager.json
      • atomeka.json
    • keys
      • atmch
        • atmcihdefault
          • private.key
          • public.key
      • atmediamanager
        • atmediamanagerdefault
          • private.key
          • public.key
      • atomeka
        • atomeka
          • private.key
          • public.key
  • prod ...
  • qa ...

Examples and notes

• Upload a file: python3 bucket.py --file=/Users/colecrawford/Github/lts-iiif-ingest-service/tests/images/27.586.1-cm-2016-02-09.tif --bucket=edu.harvard.huit.lts.mps.at-atdarth-dev
• Check it was uploaded: aws s3 ls edu.harvard.huit.lts.mps.at-atdarth-dev or aws s3 ls s3://edu.harvard.huit.lts.mps.at-atdarth-dev --recursive --human-readable --summarize
• Generate a JWT token: python3 iiif_jwt.py
• QA MPS endpoint: https://mps-admin-qa.lib.harvard.edu/admin/ingest/initialize

Ingesting images and simple V2 manifest via Postman and without manifest generation(iiifpres module)

Prerequisite if you are using aws cli, you must set export AWS_PROFILE=<BUCKET CREDENTIALS> to your desired bucket corresponding to your .aws/credentials permissions to upload files

• Upload a file using bucket.py via Examples and notes or the command below:

$ export IIIF_QA_BUCKET=edu.harvard.huit.lts.mps.at-atomeka-qa
$ aws s3 cp ./test_images/27.586.2A-cm-2016-02-09.tif s3://$IIIF_QA_BUCKET/iiif/

Note: you will need to set/create the folder/path inside the bucket named /iiif/

• Generate the jwt token via Examples and notes. You will need to have the correct private.key and public.key in similar folder structure /auth/qa/keys/omeka/omekadefault and modify the test function within iiif_jwt.py to the following parameters load_auth("atomeka", "qa")

• Create the ingess json using the example provided in sample-manifests/success-test-ingest-manifest.json. There is an example below with some fields ommited. You will need to update the fields listed below with <UNIQUE IDENTIFIER>(example: OMEKATEST2) and <UNIQUE ID or NAME>(corresponds with URN-3 @id, example: OMEKAMANIFEST2) in your own json file.

{ ...
    "assets": {
      ...
      "image": [
        {
          ...
          "storageSrcKey": <Name of the image e.g."27.586.2A-cm-2016-02-09.tif">,
          "identifier":"AT:<UNIQUE IDENTIFIER>",
         ...
    },
     "manifest" : {
      "@context": "http://iiif.io/api/presentation/2/context.json",
      "@id": "https://mps-qa.lib.harvard.edu/iiif/2/URN-3:AT:<UNIQUE ID or NAME>:MANIFEST:2",
      ...
      ,
      "sequences": [
              {
                  "@id": "https://mps-qa.lib.harvard.edu/URN-3:AT:<UNIQUE ID or NAME>/...
                  "canvases": [
                      {
                          "@id": "https://mps-qa.lib.harvard.edu/URN-3:AT:<UNIQUE ID or NAME>/canvas/canvas-400000206.json",
                        ...
                          "images": [
                              {
                                  "@id": "https://mps-qa.lib.harvard.edu/URN-3:AT:<UNIQUE ID or NAME>/annotation/annotation-400000206.json",
                                 ...
                                  "on": "https://mps-qa.lib.harvard.edu/URN-3:AT:<UNIQUE ID or NAME>/canvas/canvas-400000206.json",
                                  "resource": {
                                      "@id": "https://mps-qa.lib.harvard.edu/assets/images/AT:<UNIQUE IDENTIFIER>/full/full/0/default.jpg",
                                      ...
                                      "service": {
                                         ...
                                          "@id": "https://mps-qa.lib.harvard.edu/assets/images/AT:<UNIQUE IDENTIFIER>"
                                      }
                                  }
                              }
                          ],
                          "thumbnail": {
                              "@id": "https://mps-qa.lib.harvard.edu/assets/images/AT:<UNIQUE IDENTIFIER>/full/,100/0/default.jpg",
                              "@type": "dctypes:Image"
                          }
                      }
                  ]
              }
          ],
          "structures": [
              {
                  "@id": "https://mps-qa.lib.harvard.edu/URN-3:AT:<UNIQUE ID or NAME>/range/range-0-0.json",
                  ...
                  "canvases": ["https://mps-qa.lib.harvard.edu/URN-3:AT:<UNIQUE ID or NAME>/canvas/canvas-400000206.json"]
              }
          ]
      }
}

• use Postman to first set the bearer token to the generated jwt from previous steps and POST the json that you modified to the following endpoint: https://mps-admin-qa.lib.harvard.edu/admin/ingest/initialize Note: you will need to be on the #WEBS VPN for the POST to work.
• After recieving a 200 success code you should be able to access the image via https://mps-qa.lib.harvard.edu/assets/images/AT:<UNIQUE IDENTIFIER>/full/,100/0/default.jpg and manifest via https://mps-qa.lib.harvard.edu/iiif/c/URN-3:AT:<UNIQUE ID or NAME>