Skip to main content

Python Client for European XFEL Metadata Catalogue Web App available at https://in.xfel.eu/metadata

Project description

MyMdC is the Web App design for Data Management at European XFEL.

This library (metadata_client) is a client for the RESTful APIs exposed by the European XFEL Metadata Catalogue Web Application - myMdC (https://in.xfel.eu/metadata).

Repository:

Dependencies:

Installation

Python project

  1. Install requirements, if never done before

1.1. For OS X distributions:

1.1.1. Homebrew

      brew install python3

1.1.2 Port

      sudo port install python36

      sudo port select --set python3 python36

      sudo port install py36-pip
      sudo port select --set pip pip36

1.2. For Linux distributions:

sudo apt-get update
sudo apt-get install python3.9
  1. Make metadata_client library available in your python environment

2.1. Install it via pip:

# Install dependencies from local wheels files
pip install . --no-index --find-links ./external_dependencies/

# Install dependencies from the pypi
pip install .

# Force re-installation of packages
pip install --ignore-installed

Installing it will place two folders under the current Python installation site-packages folder:

  • metadata_client with the sources;
  • metadata_client-3.8.0.dist-info/ with Wheels configuration files.

To identify your Python site-packages folder run:

python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

Usage

To use this project you need to import it:

from metadata_client import MetadataClient
  1. Connection to the MdC (Metadata Catalog):

    from metadata_client import MetadataClient
    
    # Necessary configuration variables to establish a connection
    # Go to https://in.xfel.eu/metadata/oauth/applications to make a token for
    # the metadata catalogue.
    user_id = '201ed15ff071a63e76cb0b91a1ab17b36d5f92d24b6df4497aa646e39c46a324'
    user_secret = 'a8ae80f5e96531f19bf2d2b6102f5a537196aca44a673ad36533310e07529757'
    user_email = 'luis.maia@xfel.eu'
    #
    metadata_web_app_url = 'https://in.xfel.eu/metadata'
    token_url = 'https://in.xfel.eu/metadata/oauth/token'
    refresh_url = 'https://in.xfel.eu/metadata/oauth/token'
    auth_url = 'https://in.xfel.eu/metadata/oauth/authorize'
    scope = ''
    base_api_url = 'https://in.xfel.eu/metadata/api/'
    
    # Generate the connection (example with minimum parameter options)
    client_conn = MetadataClient(client_id=user_id,
                                 client_secret=user_secret,
                                 user_email=user_email,
                                 token_url=token_url,
                                 refresh_url=refresh_url,
                                 auth_url=auth_url,
                                 scope=scope,
                                 base_api_url=base_api_url)
    
    # Generate the connection (example with all parameter options)
    client_conn = MetadataClient(client_id=user_id,
                                 client_secret=user_secret,
                                 user_email=user_email,
                                 token_url=token_url,
                                 refresh_url=refresh_url,
                                 auth_url=auth_url,
                                 scope=scope,
                                 base_api_url=base_api_url,
                                 session_token=None,
                                 max_retries=3,
                                 timeout=12,
                                 ssl_verify=True)
    
  2. Interaction with the MyMdC (Metadata Catalog):

2.1 Example data_group_types:

all_group_types = client_conn.get_all_data_group_types()

all_group_types
# >>> {'success': True,
#      'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'},
#      'data': [{'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1},
#               {'description': '', 'identifier': 'CAL', 'name': 'Calibration', 'flg_available': True, 'id': 2},
#               {'description': '', 'identifier': 'PROC', 'name': 'Processed', 'flg_available': True, 'id': 3},
#               {'description': '', 'identifier': 'REDU', 'name': 'Reduced', 'flg_available': True, 'id': 4},
#               {'description': '', 'identifier': 'SIM', 'name': 'Simulation', 'flg_available': True, 'id': 5},
#               {'description': '', 'identifier': 'UNK', 'name': 'Unknown', 'flg_available': True, 'id': 6}],
#      'app_info': {},
#      'info': 'Got data_group_type successfully'}

all_group_types['success']
# >>> True

all_group_types['pagination']
# >>> {'Date': 'Wed, 11 May 2022 09:55:34 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'}

all_group_types['data'][0]
# >>> {'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1}

all_group_types['data'][0]['name']
# >>> 'Raw'

2.2 Example instruments:

all_xfel_instruments = client_conn.get_all_xfel_instruments()

>>> for instrument in all_xfel_instruments['data']:
...   print('id = {0} | name = {1}'.format(instrument['id'], instrument['name']))
...
# id = -1 | name = test-instrument
# id = 1 | name = SPB/SFX SASE1
# id = 2 | name = FXE SASE1
# id = 3 | name = SQS SASE3
# id = 4 | name = SCS SASE3
# id = 5 | name = MID SASE2
# id = 6 | name = HED SASE2
# id = 7 | name = Hera South Detector Test Stand
# id = 8 | name = SASE1 Test Stand
# id = 9 | name = SASE2 Test Stand
# id = 10 | name = SASE3 Test Stand

all_xfel_instruments = client_conn.get_all_xfel_instruments(page=1, page_size=1)
all_xfel_instruments

# >>> {'success': True,
#      'info': 'Got instrument successfully',
#      'app_info': {},
#      'pagination': {'Date': 'Wed, 11 May 2022 09:57:45 GMT', 'X-Total-Pages': '21', 'X-Count-Per-Page': '1', 'X-Current-Page': '1', 'X-Total-Count': '21'},
#      'data': [{'id': 1, 'name': 'SPB/SFX SASE1', 'identifier': 'SPB', 'url': 'https://www.xfel.eu/facility/instruments/spb_sfx', 'leading_scientist_id': 230, 'deputy_leading_scientist_id': 1018, 'facility_id': 1, 'instrument_type_id': 2, 'repository_id': 103, 'topic_id': 1, 'dsg_host': None, 'system_user': None, 'flg_online_resource': True, 'online_script': 'make_online', 'flg_available': True, 'description': 'The Single Particles, Clusters, and Biomolecules & Serial Femtosecond Crystallography (SPB/SFX) instrument of the European XFEL is primarily concerned with three-dimensional diffractive imaging, and three-dimensional structure determination, of micrometre-scale and smaller objects, at atomic or near-atomic¿resolution.', 'doi': None, 'techniques': [{'id': 250, 'identifier': 'PaNET01168', 'name': 'serial femtosecond crystallography', 'url': 'http://purl.org/pan-science/PaNET/PaNET01168', 'flg_available': True, 'description': None}, {'id': 259, 'identifier': 'PaNET01188', 'name': 'small angle x-ray scattering', 'url': 'http://purl.org/pan-science/PaNET/PaNET01188', 'flg_available': True, 'description': None}, {'id': 364, 'identifier': 'PaNET01101', 'name': 'x-ray powder diffraction', 'url': 'http://purl.org/pan-science/PaNET/PaNET01101', 'flg_available': True, 'description': None}, {'id': 28, 'identifier': 'PaNET01174', 'name': 'coherent diffraction imaging', 'url': 'http://purl.org/pan-science/PaNET/PaNET01174', 'flg_available': True, 'description': None}]}]}

2.3 Get instrument active proposal:

active_proposal = client_conn.get_active_proposal_by_instrument(1)

2.4 Register Run replica:

# (e.g. proposal_number == 1234)
# (e.g. proposal_number == 12)
# (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

resp = client_conn.register_run_replica(
    proposal_number, run_number, repository_identifier
)
# resp = {'success': True,
#         'info': 'Run replica registered successfully',
#         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'},
#         'data': {'experiment_id': '-1',
#                  'sample_id': '-1',
#                  'run_id': '1588',
#                  'data_group_id': '777'},
#         'app_info': {}}

2.5 Unregister Run replica:

# (e.g. proposal_number == 1234)
# (e.g. proposal_number == 12)
# (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

resp = client_conn.unregister_run_replica(
    proposal_number, run_number, repository_identifier
)
# resp = {'success': True,
#         'info': 'Run replica unregistered successfully',
#         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT', 'X-Total-Pages': '1', 'X-Count-Per-Page': '100', 'X-Current-Page': '1', 'X-Total-Count': '6'},
#         'data': {'data_group_id': '-1',
#                  'repository_id': '1',
#                  'flg_available': 'false'},
#         'app_info': {}}

2.6 Get proposal’s runs:

# (e.g. proposal_number == 1234)
# (e.g. page == 1 | Default == 1)
# (e.g. page_size == 5 | Default == 100 | Limit: 500)

resp = client_conn.get_proposal_runs(proposal_number, page=1, page_size=5)
# RESPONSE example
#
# resp = {'info': 'Got proposal successfully',
#         'success': True,
#         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT',
#                        'X-Total-Pages': '1',
#                        'X-Count-Per-Page': '100',
#                        'X-Current-Page': '1',
#                        'X-Total-Count': '6'},
#         'data': {
#           'proposal': {
#               'id': -1,
#               'number': 0,
#               'title': 'Proposal Title 001'
#                  },
#           'runs': [
#               {
#               'id': -1,
#               'run_number': 1,
#               'flg_status': 1,
#               'flg_run_quality': -1,
#               'size': null,
#               'num_files': 0,
#               'repositories': {
#                   'XFEL_TESTS_REPO': {
#                       'name": 'XFEL Tests Repository',
#                       'mount_point': '/webstorage/XFEL',
#                       'data_groups': 1
#                       }
#                   }
#               }
#            ]
#          },
#         'app_info': {}}

2.7 Get proposal’s samples:

# (e.g. proposal_number == 1234)
# (e.g. page == 1 | Default == 1)
# (e.g. page_size == 50 | Default == 100 | Limit: 500)

resp = client_conn.get_proposal_samples(proposal_number, page=1, page_size=50)
#
# RESPONSE example
#
# resp = {'info': 'Got sample successfully',
#         'success': True,
#         'pagination': {'Date': 'Tue, 10 May 2022 22:48:14 GMT',
#                        'X-Total-Pages': '1',
#                        'X-Count-Per-Page': '100',
#                        'X-Current-Page': '1',
#                        'X-Total-Count': '6'},
#         'data': [{'id': -1,
#                   'name': 'TestSample DO NOT DELETE!',
#                   'proposal_id': -1,
#                   'sample_type_id': 1,
#                   'flg_available': True,
#                   'url': '',
#                   'description': ''}],
#         'app_info': {}}

For additional examples, please take a look in the tests/ folder.

Development & Testing

When developing, and before commit changes, please validate that:

  1. All tests continue passing successfully (to validate that run pytest):

    # Go to the source code directory
    cd metadata_client
    
    # Upgrade package and all its required packages
    pip install . -U --upgrade-strategy eager
    
    # Install test dependencies
    pip install '.[test]' -U --upgrade-strategy eager
    
    # Run all tests using pytest
    pytest
    
    # When running all tests against the standard http application
    OAUTHLIB_INSECURE_TRANSPORT=1 pytest
    
    # Run all tests and get information about coverage for all files inside metadata_client package
    pytest --cov metadata_client --cov-report term-missing
    
  2. Code keeps respecting pycodestyle code conventions (to validate that run pycodestyle):

    pycodestyle .
    pycodestyle . --exclude venv
    
  3. To generate all the wheels files for the dependencies, execute:

    # Generate Wheels to itself and dependencies
    pip wheel --wheel-dir=./external_dependencies .
    pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies .
    
  4. Check that you have the desired dependency versions in external_dependencies folder, since no versions are now set in setup.py.

Registering library on https://pypi.org

To register this python library, the following steps are necessary:

# Install twine
python -m pip install --upgrade twine

# Generates source distribution (.tar.gz) and wheel (.whl) files in the dist/ folder
python setup.py sdist
python setup.py bdist_wheel

# Upload new version .egg and .whl files
twine upload dist/*

# In case a test is necessary, it is possible to test it against test.pypi.org
twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

metadata_client-3.8.0.tar.gz (64.2 kB view hashes)

Uploaded source

Built Distribution

metadata_client-3.8.0-py3-none-any.whl (114.8 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page