Skip to main content

Python Client for European XFEL Metadata Catalogue Web App available at https://in.xfel.eu/metadata

Project description

MyMdC is the Web App design for Data Management at European XFEL.

This library (metadata_client) is a client for the RESTful APIs exposed by the European XFEL Metadata Catalogue Web Application - myMdC (https://in.xfel.eu/metadata).

Repository:

Dependencies:

Installation

Python project

  1. Install requirements, if never done before

1.1. For OS X distributions:

1.1.1. Homebrew

      brew install python3
      --
      pip3 install pycodestyle --user
      pip3 install node --user
      pip3 install nose-py3 --user
      pip3 install python-dateutil --user


1.1.2 Port

      sudo port install python36

      sudo port select --set python3 python36

      sudo port install py36-pip
      sudo port select --set pip pip36

      sudo port install py36-nose
      sudo port select --set nosetests nosetests-3.6

      pip install pycodestyle --user
      pip install node --user
      pip install nose-py3 --user
      pip3 install python-dateutil --user

1.2. For Linux distributions:

sudo apt-get update
sudo apt-get install python3.6
  1. Make metadata_client library available in your python environment

2.1. Install it via pip:

# Install dependencies from local wheels files
pip3 install --no-index --upgrade --find-links ./external_dependencies/*
pip3 install --user --no-index --upgrade --find-links ./external_dependencies/*

# Install dependencies from the pypi
pip3 install -r requirements.txt

Or as a normal python project (via .egg file):

python setup.py install
python3 setup.py install --user

Running this command the “compiled” metadata_client-3.0.5-py3.6.egg file is generated under the current Python installation site-packages folder.

or. Install it as a normal python project (via Wheel):

python setup.py bdist_wheel

Running this command 2 folders are generated under the current Python installation site-packages folder:

  • metadata_client with the sources;

  • metadata_client-3.0.5.dist-info/ with Wheels configuration files.

  1. To identify your Python site-packages folder run:

    python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

Usage

To use this project you need to import it.

If you want interact directly with API methods you should import MetadataClientApi class:

from metadata_client.metadata_client_api import MetadataClientApi

If you want interact with Model classes you should import MetadataClient class:

from metadata_client.metadata_client import MetadataClient

Or import everything:

import metadata_client
  1. Connection to the MdC (Metadata Catalog):

    from metadata_client.metadata_client import MetadataClient
    
    # Necessary configuration variables to establish a connection
    user_id = '201ed15ff071a63e76cb0b91a1ab17b36d5f92d24b6df4497aa646e39c46a324'
    user_secret = 'a8ae80f5e96531f19bf2d2b6102f5a537196aca44a673ad36533310e07529757'
    user_email = 'luis.maia@xfel.eu'
    #
    metadata_web_app_url = 'https://in.xfel.eu/dev_metadata'
    token_url = 'https://in.xfel.eu/dev_metadata/oauth/token'
    refresh_url = 'https://in.xfel.eu/dev_metadata/oauth/token'
    auth_url = 'https://in.xfel.eu/dev_metadata/oauth/authorize'
    scope = ''
    base_api_url = 'https://in.xfel.eu/dev_metadata/api/'
    
    # Generate the connection
    client_conn = MetadataClient(client_id=user_id,
                                 client_secret=user_secret,
                                 user_email=user_email,
                                 token_url=token_url,
                                 refresh_url=refresh_url,
                                 auth_url=auth_url,
                                 scope=scope,
                                 base_api_url=base_api_url)
  2. Interaction with the MdC (Metadata Catalog):

2.1 Example data_group_types:

all_group_types = MetadataClient.get_all_data_group_types(client_conn)

all_group_types
# >>> {'success': True,
#      'data': [{'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1},
#               {'description': '', 'identifier': 'CAL', 'name': 'Calibration', 'flg_available': True, 'id': 2},
#               {'description': '', 'identifier': 'PROC', 'name': 'Processed', 'flg_available': True, 'id': 3},
#               {'description': '', 'identifier': 'REDU', 'name': 'Reduced', 'flg_available': True, 'id': 4},
#               {'description': '', 'identifier': 'SIM', 'name': 'Simulation', 'flg_available': True, 'id': 5},
#               {'description': '', 'identifier': 'UNK', 'name': 'Unknown', 'flg_available': True, 'id': 6}],
#      'app_info': {},
#      'info': 'Got data_group_type successfully'}

all_group_types['success']
# >>> True

all_group_types['data'][0]
# >>> {'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1}

all_group_types['data'][0]['name']
# >>> 'Raw'

2.2 Example instruments:

all_xfel_instruments = MetadataClient.get_all_xfel_instruments(client_conn)

>>> for instrument in all_xfel_instruments['data']:
...   print('id = {0} | name = {1}'.format(instrument['id'], instrument['name']))
...
# id = -1 | name = test-instrument
# id = 1 | name = SPB/SFX SASE1
# id = 2 | name = FXE SASE1
# id = 3 | name = SQS SASE3
# id = 4 | name = SCS SASE3
# id = 5 | name = MID SASE2
# id = 6 | name = HED SASE2
# id = 7 | name = Hera South Detector Test Stand
# id = 8 | name = SASE1 Test Stand
# id = 9 | name = SASE2 Test Stand
# id = 10 | name = SASE3 Test Stand

2.3 Get instrument active proposal:

active_proposal = MetadataClient.get_active_proposal_by_instrument(client_conn, 1)

2.4 Register Run replica:

# (e.g. proposal_number == 1234)
# (e.g. proposal_number == 12)
# (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

resp = MetadataClient.register_run_replica(client_conn,
                                           proposal_number,
                                           run_number,
                                           repository_identifier)
# resp = {'success': True,
#         'info': 'Run replica registered successfully',
#         'data': {'experiment_id': '-1',
#                  'sample_id': '-1',
#                  'run_id': '1588',
#                  'data_group_id': '777'},
#         'app_info': {}}

2.5 Unregister Run replica:

# (e.g. proposal_number == 1234)
# (e.g. proposal_number == 12)
# (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC')

resp = MetadataClient.unregister_run_replica(client_conn,
                                             proposal_number,
                                             run_number,
                                             repository_identifier)
# resp = {'success': True,
#         'info': 'Run replica unregistered successfully',
#         'data': {'data_group_id': '-1',
#                  'repository_id': '1',
#                  'flg_available': 'false'},
#         'app_info': {}}

For additional examples, please take a look in the tests/ folder.

Development & Testing

When developing, and before commit changes, please validate that:

  1. All tests continue passing successfully (to validate that run nosetests)

1.1 Using nosetests:

# Go to the source code directory
cd metadata_client

# Run all tests using nosetests
nosetests .

# Run all tests and get information about coverage for all files inside metadata_client package
pip install nose-cov
nosetests --with-cov --cover-erase --cover-inclusive --cov-report term-missing --cov metadata_client

# Run all tests with xunit
nosetests --where=./metadata_client/ --with-xunit --xunit-file=pythonTest.xml

1.2 Using nose2:

# Go to the source code directory
cd metadata_client

# Run all tests nose2
nose2 -v --config .nose2.cfg --with-coverage

1.3 Using python:

# Go to the source code directory
cd metadata_client

# If you don't want use nosetests you can simply run the test class
python metadata_client/tests/metadata_client_test.py
  1. Code keeps respecting pycodestyle code conventions (to validate that run pycodestyle):

    pycodestyle .
  2. To generate all the wheels files for the dependencies, execute:

    # Generate Wheels to its dependencies
    pip wheel --wheel-dir=./external_dependencies -r requirements.txt
    pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies -r requirements.txt
    
    # Generate Wheels to itself and dependencies
    pip wheel --wheel-dir=./external_dependencies .
    pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies .

Guarantee that you have the desired versions in requirements.txt and setup.py files.

Registering library on https://pypi.org

To register this python library, the following steps are necessary:

# Install twine
python -m pip install --upgrade twine --user

# Generates egg file in the dist/ folder
python setup.py install

# Upload new version .egg and .whl files
twine upload dist/*
twine upload metadata_client-3.0.5-py3-none-any.whl

# In case a test is necessary, it is possible to test it against test.pypi.org
twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

metadata_client-3.0.5-py3.7.egg (242.2 kB view hashes)

Uploaded 3 7

metadata_client-3.0.5-py3-none-any.whl (104.8 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page