doipy 0.5.0

A Python wrapper for DOIP.

DOIPY

DOIPY is a Python wrapper for communication using the Digital Object Interface Protocol (DOIP) in its current specification v2.0.

It supports three main functionalities:

1. Receive the IP and port of a DOIP service by supplying the service ID (see get_connection).
2. All basic DOIP operations to handle digital objects (DOs): hello, list_operations, create, update, delete, retrieve, and search. Extended operations implemented by specific repository software are and will be included in the future.
3. Some FDO-Manager functionality to handle FAIR digital objects (FDOs): Create or delete an FDO ( see create_fdo, delete_fdo) that complies with configuration type 14, which is expressed by a combination of DOIP basic operations, validation steps and communication with the data type registry (DTR).

Install

Simply run

$ pip install doipy

Usage (Python Code)

To use the doipy package in the Python code simply import it and call the exposed methods. The package has several methods. Please use doipy --help to list all available methods.

1. Get the IP and Port of a DOIP Service

The service ID is a handle which identifies a DOIP service. For example, 21.T11967/service is the service ID identifying the Cordra instance in the FDO One Testbed. Starting with the service ID, one receives the IP and port of the DOIP service by applying the get_connection function.

from doipy import get_connection

# get the IP and port of a DOIP service
service_id, ip, port = get_connection('21.T11967/service')

2. Basic DOIP Operations

hello, list_operations, create, and search

To communicate with a DOIP service, one has to supply a target ID as well as the IP and port of the desired service. For the operations hello, create, and search, the target ID is equal to the service ID. For retrieve, delete, and update, the target ID corresponds to the PID of a DO. For list_operations, the target ID might be either a service ID or a DO PID. Below is demonstrated how to apply operations where the service ID is required.

from doipy import get_connection, hello, list_operations, create, search
from pathlib import Path

# get the IP and port of a DOIP service
service_id, ip, port = get_connection(service='21.T11967/service')

# say hello to the data service
response_hello = hello(target_id=service_id, ip=ip, port=port)

# list all available operations at the data service
response_list_operations = list_operations(target_id=service_id, ip=ip, port=port)

# create a DO at the data service
metadata = {'key1': 'value1', 'key2': 'value2'}
username = ''
password = ''
# possibilities for authentication: provide either (username, password) or (client_id, password) or (token). The 
# authentication credentials are the credentials to authenticate the user at the DOIP service.
response_create = create(target_id=service_id, ip=ip, port=port, bitsq=Path('file.txt'), metadata=metadata,
                         username=username, password=password)

# call the search operation
response_search = search(target_id=service_id, ip=ip, port=port, query='type:Document', username=username,
                         password=password)

list_operations, retrieve, and delete

For the operations list_operations, retrieve, and delete, the target ID corresponds to a PID identifying a DO. As an example, we take the PID 21.T11967/35463c4d5e1cf0449a31 which identifies a DO at the Cordra instance of the FDO One Testbed. To run those operations in the Python code, one can follow the lines below.

from doipy import get_connection, list_operations, retrieve, delete

# get the IP and port of a DOIP service
_, ip, port = get_connection('21.T11967/service')

do = '21.T11967/35463c4d5e1cf0449a31'
username = ''
password = ''

# list all available operations on the given DO
response_list_operations = list_operations(target_id=do, ip=ip, port=port, username=username, password=password)

# retrieve a DO
response_retrieve = retrieve(target_id=do, ip=ip, port=port, username=username, password=password)

# download a bit-sequence of a DO. The file must be the id of the bit-sequence to be downloaded.
response_download = retrieve(target_id=do, ip=ip, file='031c09fd-d45d-48b0-acab-57ec049bb6c8', port=port,
                             username=username, password=password)

# delete a DO
response_delete = delete(target_id=do, ip=ip, port=port, username=username, password=password)

add_metadata, update_metadata, delete_metadata, and update_all_metadata

There are several operations available to update the metadata key-value pairs in a PID record of a DO. The functions are: add_metadata to add some metadata to a PID record, delete_metadata to delete some metadata from a PID record, update_metadata to update some already existing metadata, and update_all_metadata which overrides the current metadata by a completely new list of metadata. The target ID corresponds to the PID of the DO that should be updated. One can follow the code lines below.

from doipy import get_connection, add_metadata, update_metadata, delete_metadata, update_all_metadata

# get the IP and port of a DOIP service
_, ip, port = get_connection('21.T11967/service')

# choose a DO located at the given service, which can be updated with my username and password
do = ''
metadata = {'key1': 'value1', 'key2': 'value2'}
metadata_keys = ['key1', 'key2']
username = ''
password = ''

# add values to the metadata of the DO
response_add_metadata = add_metadata(target_id=do, ip=ip, port=port, metadata=metadata, username=username,
                                     password=password)

# update existing metadata of the DO
response_update_metadata = update_metadata(target_id=do, ip=ip, port=port, metadata=metadata, username=username,
                                           password=password)

# delete values from the metadata of the DO
response_delete_metadata = delete_metadata(target_id=do, ip=ip, port=port, metadata=metadata_keys, username=username,
                                           password=password)

# override the metadata of the DO by a new list of metadata
response_update_all_metadata = update_all_metadata(target_id=do, ip=ip, port=port, metadata=metadata, username=username,
                                                   password=password)

add_bitsq, update_bitsq, and delete_bitsq

There are also several operations available to update the (unique) bit-sequence of a DO. The functions are: add_bitsq to add a bit-sequence to a DO which does not have any bit-sequence yet, delete_bitsq to delete an existing bit-sequence from a DO, and update_bitsq to update the (unique) bit-sequence of a DO by a new bit-sequence. The target ID corresponds to the PID of the DO that should be updated. One can follow the code lines below.

from doipy import get_connection, add_bitsq, update_bitsq, delete_bitsq
from pathlib import Path

# get the IP and port of a DOIP service
_, ip, port = get_connection('21.T11967/service')

# choose a DO located at the given service, which does not have any bit-sequence yet and which can be updated with my 
# username and password
do = ''
username = ''
password = ''

# add values to the metadata of the DO
response_add_bitsq = add_bitsq(target_id=do, ip=ip, port=port, bitsq=Path('file1.txt'), username=username,
                               password=password)

# update existing metadata of the DO
response_update_bitsq = update_bitsq(target_id=do, ip=ip, port=port, bitsq=Path('file2.txt'), username=username,
                                     password=password)

# delete values from the metadata of the DO
response_delete_bitsq = delete_bitsq(target_id=do, ip=ip, port=port, username=username, password=password)

3. FDO Manager

create_fdo

To create an FDO in the Python code, one needs to supply a Python dictionary which follows the structure of the schema defined at https://typeapi.lab.pidconsortium.net/v1/types/schema/21.T11969/6e36f6c0de5fcab4a425 as input to create_fdo.

The create_fdo function supports FDOs following configuration type 14, i.e., which consist of multiple data DOs and multiple metadata DOs.

Each item in FDO_Data_and_Metadata is a data bit-sequence data_bitsq and its corresponding metadata bit-sequence metadata_bitsq. One DO is generated for the data bit-sequence and one DO is generated for the metadata bit-sequence. The content of data_values is written into the PID record of the data DO. The content of metadata_values is written into the PID record of the metadata DO.

Use create_fdo to register an FDO with specified (meta)data bit-sequences. If create_fdo is successful, the PID of the new FDO is returned. If it is not successful, an error is returned.

from doipy import create_fdo

user_input = {
    "FDO_Service_Ref": "21.T11969/01370800d56a0d897dc1",
    "FDO_Profile_Ref": "21.T11969/141bf451b18a79d0fe66",
    "FDO_Authentication": {
        "username": "",
        "password": ""
    },
    "FDO_Type_Ref": "21.1/thisIsAnFdoType",
    "FDO_Rights_Ref": "21.1/thisIsAnFdoRightsSpecification",
    "FDO_Genre_Ref": "21.1/thisIsAnFdoGenre",
    "FDO_Data_and_Metadata": [
        {
            "data_bitsq": "data_bitsq_1.txt",
            "data_values": "data_values_1.json",
            "metadata_bitsq": "metadata_bitsq_1.json",
            "metadata_values": "metadata_values_1.json"
        },
        {
            "data_bitsq": "data_bitsq_2.txt",
            "data_values": "data_values_2.json",
            "metadata_bitsq": "metadata_bitsq_2.json",
            "metadata_values": "metadata_values_2.json"
        }
    ]
}

# create an FDO
result_create_fdo = create_fdo(user_input)

delete_fdo

To delete an FDO that follows configuration type 14 in the Python code, one needs to supply a Python dictionary following the structure of the schema defined at https://typeapi.lab.pidconsortium.net/v1/types/schema/21.T11969/78a7b599f1f707830402 as input to delete_fdo.

Deleting the FDO means to delete all data bit-sequences that belong to the FDO. Additionally, one can specify whether also the metadata bit-sequences should be deleted (parameter: delete_md). The FDO_Status in the FDO record is set to deleted. The DO_Status and Status_URL in all DOs whose bit-sequences are deleted are updated to deleted respectively reference a tombstone URL.

If delete_fdo is successful, 1 is returned. If it is not successful, an error is returned.

from doipy import delete_fdo

user_input = {
    "PID_FDO": "21.T11967/72c5e9b5843c7c8c0658",
    "FDO_Service_Ref": "21.T11969/01370800d56a0d897dc1",
    "delete_MD": True,
    "FDO_Authentication": {
        "username": "",
        "password": ""
    }
}

# create an FDO
result_delete_fdo = delete_fdo(user_input)

move_fdo (currently supported: move from Cordra to Cordra)

Using move_fdo, an FDO that follows configuration type 14 can be moved from a source repository R1 to a destination repository R2. The operation requires that all DOs that form the FDO and the FDO record are initially placed in the same source repository R1. Moving this FDO from R1 to R2 means:

• All DOs (i.e., data DOs and metadata DOs) are moved to R2 and receive new PIDs in R2.
• The FDO Record is moved to R2 and receives a new PID in R2. It will contain the new DO PIDs in FDO_Data_Ref and FDO_Metadata_Ref. Its FDO_Status is moved.
• All bit-sequences of the initial FDO are deleted. The FDO record of the initial FDO gets a HS_ALIAS which references the moved FDO in R2. All other metadata are removed from the initial FDO record.

To move an FDO from R1 to R2, one needs to supply a Python dictionary following the structure of the schema defined at https://typeapi.lab.pidconsortium.net/v1/types/schema/21.T11969/4677eeaea2eb2a2c2704 as input tomove_fdo. If move_fdo is successful, the PID of the moved FDO is returned. If it is not successful, an error is returned.

from doipy import move_fdo

user_input = {
    "PID_FDO": "",
    "FDO_Service_Ref_Source": "21.T11969/01370800d56a0d897dc1",
    "FDO_Authentication_Source": {
        "username": "",
        "password": ""
    },
    "FDO_Service_Ref_Destination": "21.T11969/01370800d56a0d897dc1",
    "FDO_Authentication_Destination": {
        "username": "",
        "password": ""
    }
}
# create an FDO
result_move_fdo = move_fdo(user_input)

Authentication

If authentication credentials need to be provided, then the user has several options to authenticate:

1. username and password
2. client_id and password
3. token

Authentication credentials must be provided for the DOIP functions: create, delete, and update. Depending on the rights for read access, authentication credentials might be necessary for list_operations of a DO, retrieve, and search as well. Authentication credentials must be provided for the FDO-Manager functions: create_fdo. Note that not all data services accept all three options of authentication credentials.

Usage (Command Line Interface)

1. Get the IP and port of a DOIP service

Starting with the service ID, receive the IP and port of the DOIP service by applying the get_connection function.

# get the IP and port of a DOIP service
$ doipy get_connection '21.T11967/service'

2. Basic DOIP Operations

hello, list_operations, create, and search

We demonstrate how to apply operations hello, list_operations, create, and search on the DOIP service which is identified by the service ID. Additional to the service ID, IP and port, the create operation has more parameters: The do_type refers to the type of the DO at the data service, which is Document in Cordra. bitsq is a path to a file which contains the data of the DO. metadata is a path to a JSON file containing the metadata. The key-value pairs from the JSON file are written into the handle record of the DO at generation. username and passwordare the authentication credentials of a user at the data service.

# get information from the DOIP service
$ doipy hello '21.T11967/service' '141.5.106.77' 9000

# list all available operations at the DOIP service
$ doipy list_operations '21.T11967/service' '141.5.106.77' 9000

# create a DO at the DOIP service
$ doipy create '21.T11967/service' '141.5.106.77' 9000 --do-type 'Document' --do-name 'my_DO' --bitsq 'data.txt' --metadata 'metadata.json' --username '' --password ''

# search in the DOIP service for a DO (todo)
$ doipy search '21.T11967/service' '141.5.106.77' 9000 'type:Document' --username '' --password ''

list_operations, retrieve, and delete

Apply the functions list_operations , retrieve, and delete on the DO which is identified by the PID "21.T11967/35463c4d5e1cf0449a31.

# List all available operations of a DO
$ doipy list_operations '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 --username '' --password ''

# retrieve a DO 
$ doipy retrieve '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 --username '' --password ''

# download a bit-sequence of a DO. The file must be the id of the bit-sequence to be downloaded.
$ doipy retrieve '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 --file '031c09fd-d45d-48b0-acab-57ec049bb6c8' --username '' --password ''

# delete a DO
$ doipy delete '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 --username '' --password ''

add_metadata, update_metadata, delete_metadata and update_all_metadata

To update the metadata in the PID record of a DO, apply the functions add_metadata, update_metadata, delete_metadata and update_all_metadata. The new list of metadata shall be supplied as a JSON file metadata.json.

# add values to the metadata of the DO
$ doipy add_metadata '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 'metadata.json' --username '' --password ''

# update existing metadata of the DO
$ doipy update_metadata '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 'metadata.json' --username '' --password ''

# delete values from the metadata of the DO
$ doipy delete_metadata '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 'metadata.json' --username '' --password ''

# override the metadata of the DO by a new list of metadata
$ doipy update_all_metadata '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 'metadata.json' --username '' --password ''

add_bitsq, update_bitsq, and delete_bitsq

To update the bit-sequence of a DO, apply the functions add_bitsq, update_bitsq, and delete_bitsq. The new bit-sequence of the DO shall be supplied as a file file.txt.

# add values to the metadata of the DO
$ doipy add_bitsq '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 'file.txt' --username '' --password ''

# update existing metadata of the DO
$ doipy update_bitsq '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 'file.txt' --username '' --password ''

# delete values from the metadata of the DO
$ doipy delete_bitsq '21.T11967/35463c4d5e1cf0449a31' '141.5.106.77' 9000 --username '' --password ''

3. FDO Manager

create_fdo

To create an FDO on the CLI, first create a JSON file (called input.json), whose content follows the schema defined at https://typeapi.lab.pidconsortium.net/v1/types/schema/21.T11969/6e36f6c0de5fcab4a425

An example JSON file could look like this:

{
  "FDO_Service_Ref": "21.T11969/01370800d56a0d897dc1",
  "FDO_Profile_Ref": "21.T11969/141bf451b18a79d0fe66",
  "FDO_Authentication": {
    "username": "",
    "password": ""
  },
  "FDO_Type_Ref": "21.1/thisIsAnFdoType",
  "FDO_Rights_Ref": "21.1/thisIsAnFdoRightsSpecification",
  "FDO_Genre_Ref": "21.1/thisIsAnFdoGenre",
  "FDO_Data_and_Metadata": [
    {
      "data_bitsq": "data_bitsq_1.txt",
      "data_values": "data_values_1.json",
      "metadata_bitsq": "metadata_bitsq_1.json",
      "metadata_values": "metadata_values_1.json"
    },
    {
      "data_bitsq": "data_bitsq_2.txt",
      "data_values": "data_values_2.json",
      "metadata_bitsq": "metadata_bitsq_2.json",
      "metadata_values": "metadata_values_2.json"
    }
  ]
}

Use create_fdo to register an FDO with specified (meta)data bit-sequences:

$ doipy create_fdo input.json

delete_fdo

To delete an FDO on the CLI, first create a JSON file (called input.json), whose content follows the schema defined at https://typeapi.lab.pidconsortium.net/v1/types/schema/21.T11969/78a7b599f1f707830402

An example JSON file could look like this:

{
  "PID_FDO": "21.T11967/72c5e9b5843c7c8c0658",
  "FDO_Service_Ref": "21.T11969/01370800d56a0d897dc1",
  "delete_MD": true,
  "FDO_Authentication": {
    "username": "",
    "password": ""
  }
}

Use delete_fdo to delete an FDO that follows configuration type 14:

$ doipy delete_fdo input.json

move_fdo (currently supported: move from Cordra to Cordra)

To move an FDO from R1 to R2 on the CLI, first create a JSON file (called input.json), whose content follows the schema defined at https://typeapi.lab.pidconsortium.net/v1/types/schema/21.T11969/4677eeaea2eb2a2c2704

An example JSON file could look like this:

{
  "PID_FDO": "",
  "FDO_Service_Ref_Source": "21.T11969/01370800d56a0d897dc1",
  "FDO_Authentication_Source": {
    "username": "",
    "password": ""
  },
  "FDO_Service_Ref_Destination": "21.T11969/01370800d56a0d897dc1",
  "FDO_Authentication_Destination": {
    "username": "",
    "password": ""
  }
}

Use move_fdo to move an FDO that follows configuration type 14:

$ doipy move_fdo input.json

For developer

The project is managed by Poetry. Therefore, make sure that Poetry is installed in your system. Then run

$ poetry install

to install all dependencies. With this command, Poetry also installs the package in editable mode.