Skip to main content

Python Client for European XFEL Metadata Catalogue Web App available at https://in.xfel.eu/metadata

Project description

MyMdC is the Web App design for Data Management at European XFEL.

This library (metadata_client) is a client for the RESTful APIs exposed by the European XFEL Metadata Catalogue Web Application - myMdC (https://in.xfel.eu/metadata).

Repository:

Dependencies:

Installation

Python project

  1. Install requirements, if never done before

1.1. For OS X distributions:

sudo port install python36
sudo port

sudo port select --set python3 python36

sudo port install py36-pip
sudo port select --set pip pip36

sudo port install py36-nose
sudo port select --set nosetests nosetests-3.6

pip install pycodestyle
pip install node

1.2. For Linux distributions:

sudo apt-get update
sudo apt-get install python3.6
  1. Make metadata_client library available in your python environment

2.1. Install it via pip:

# Install dependencies from local wheels files
pip install --no-index --upgrade --find-links ./external_dependencies/*
pip install --user --no-index --upgrade --find-links ./external_dependencies/*

# Install dependencies from the pypi
pip install -r requirements.txt

Or as a normal python project (via .egg file):

python setup.py install
python setup.py install --user

Running this command the “compiled” metadata_client-3.0.4-py3.6.egg file is generated under the current Python installation site-packages folder.

or. Install it as a normal python project (via Wheel):

python setup.py bdist_wheel

Running this command 2 folders are generated under the current Python installation site-packages folder:

  • metadata_client with the sources;
  • metadata_client-3.0.4.dist-info/ with Wheels configuration files.
  1. To identify your Python site-packages folder run:

    python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
    

Usage

To use this project you need to import it.

If you want interact directly with API methods you should import MetadataClientApi class:

from metadata_client.metadata_client_api import MetadataClientApi

If you want interact with Model classes you should import MetadataClient class:

from metadata_client.metadata_client import MetadataClient

Or import everything:

import metadata_client
  1. Connection to the MdC (Metadata Catalog):

    from metadata_client.metadata_client import MetadataClient
    
    # Necessary configuration variables to establish a connection
    user_id = '201ed15ff071a63e76cb0b91a1ab17b36d5f92d24b6df4497aa646e39c46a324'
    user_secret = 'a8ae80f5e96531f19bf2d2b6102f5a537196aca44a673ad36533310e07529757'
    user_email = 'luis.maia@xfel.eu'
    #
    metadata_web_app_url = 'https://in.xfel.eu/dev_metadata'
    token_url = 'https://in.xfel.eu/dev_metadata/oauth/token'
    refresh_url = 'https://in.xfel.eu/dev_metadata/oauth/token'
    auth_url = 'https://in.xfel.eu/dev_metadata/oauth/authorize'
    scope = ''
    base_api_url = 'https://in.xfel.eu/dev_metadata/api/'
    
    # Generate the connection
    client_conn = MetadataClient(client_id=user_id,
                                 client_secret=user_secret,
                                 user_email=user_email,
                                 token_url=token_url,
                                 refresh_url=refresh_url,
                                 auth_url=auth_url,
                                 scope=scope,
                                 base_api_url=base_api_url)
    
  2. Interaction with the MdC (Metadata Catalog):

2.1 Example data_group_types:

all_group_types = MetadataClient.get_all_data_group_types(client_conn)

all_group_types
# >>> {'success': True,
#      'data': [{'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1},
#               {'description': '', 'identifier': 'CAL', 'name': 'Calibration', 'flg_available': True, 'id': 2},
#               {'description': '', 'identifier': 'PROC', 'name': 'Processed', 'flg_available': True, 'id': 3},
#               {'description': '', 'identifier': 'REDU', 'name': 'Reduced', 'flg_available': True, 'id': 4},
#               {'description': '', 'identifier': 'SIM', 'name': 'Simulation', 'flg_available': True, 'id': 5},
#               {'description': '', 'identifier': 'UNK', 'name': 'Unknown', 'flg_available': True, 'id': 6}],
#      'app_info': {},
#      'info': 'Got data_group_type successfully'}

all_group_types['success']
# >>> True

all_group_types['data'][0]
# >>> {'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1}

all_group_types['data'][0]['name']
# >>> 'Raw'

2.2 Example instruments:

all_xfel_instruments = MetadataClient.get_all_xfel_instruments(client_conn)

>>> for instrument in all_xfel_instruments['data']:
...   print('id = {0} | name = {1}'.format(instrument['id'], instrument['name']))
...
# id = -1 | name = test-instrument
# id = 1 | name = SPB/SFX SASE1
# id = 2 | name = FXE SASE1
# id = 3 | name = SQS SASE3
# id = 4 | name = SCS SASE3
# id = 5 | name = MID SASE2
# id = 6 | name = HED SASE2
# id = 7 | name = Hera South Detector Test Stand
# id = 8 | name = SASE1 Test Stand
# id = 9 | name = SASE2 Test Stand
# id = 10 | name = SASE3 Test Stand

2.3 Get instrument active proposal:

active_proposal = MetadataClient.get_active_proposal_by_instrument(client_conn, 1)

2.4 Register Run replica:

# (e.g. proposal_number == 1234)
# (e.g. proposal_number == 12)
# (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

resp = MetadataClient.register_run_replica(client_conn,
                                           proposal_number,
                                           run_number,
                                           repository_identifier)
# resp = {'success': True,
#         'info': 'Run replica registered successfully',
#         'data': {'experiment_id': '-1',
#                  'sample_id': '-1',
#                  'run_id': '1588',
#                  'data_group_id': '777'},
#         'app_info': {}}

2.5 Unregister Run replica:

# (e.g. proposal_number == 1234)
# (e.g. proposal_number == 12)
# (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

resp = MetadataClient.unregister_run_replica(client_conn,
                                             proposal_number,
                                             run_number,
                                             repository_identifier)
# resp = {'success': True,
#         'info': 'Run replica unregistered successfully',
#         'data': {'data_group_id': '-1',
#                  'repository_id': '1',
#                  'flg_available': 'false'},
#         'app_info': {}}

For additional examples, please take a look in the tests/ folder.

Development & Testing

When developing, and before commit changes, please validate that:

  1. All tests continue passing successfully (to validate that run nosetests):

    # Go to the source code directory
    cd metadata_client
    
    # Run all tests
    nosetests .
    
    # Run all tests and get information about coverage for all files inside metadata_client package
    pip install python-dateutil
    pip install nose-cov
    nosetests --with-cov --cover-erase --cover-inclusive --cov-report term-missing --cov metadata_client
    
    # Run all tests with xunit
    nosetests --where=./metadata_client/ --with-xunit --xunit-file=pythonTest.xml
    
    # If you don't want use nosetests you can simply run the test class
    python metadata_client/tests/metadata_client_test.py
    
  2. Code keeps respecting pycodestyle code conventions (to validate that run pycodestyle):

    pycodestyle .
    
  3. To generate all the wheels files for the dependencies, execute:

    # Generate Wheels to its dependencies
    pip wheel --wheel-dir=./external_dependencies -r requirements.txt
    pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies -r requirements.txt
    
    # Generate Wheels to itself and dependencies
    pip wheel --wheel-dir=./external_dependencies .
    pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies .
    

Guarantee that you have the desired versions in requirements.txt and setup.py files.

Registering library on https://pypi.org

To register this python library, the following steps are necessary:

# Install twine
python -m pip install --upgrade twine

# Generates egg file in the dist/ folder
python setup.py install

# Upload new version .egg and .whl files
twine upload dist/*
twine upload metadata_client-3.0.4-py3-none-any.whl

# In case a teste is necessary, it is possible to test it against test.pypi.org
twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for metadata-client, version 3.0.4
Filename, size File type Python version Upload date Hashes
Filename, size metadata_client-3.0.4-py3-none-any.whl (107.6 kB) File type Wheel Python version py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page