Python Client for European XFEL Metadata Catalogue Web App available at https://in.xfel.eu/metadata
Project description
MyMdC is the Web App design for Data Management at European XFEL.
This library (metadata_client) is a client for the RESTful APIs exposed by the European XFEL Metadata Catalogue Web Application - myMdC (https://in.xfel.eu/metadata).
Repository:
Dependencies:
oauthlib (https://pypi.python.org/pypi/oauthlib)
requests (https://github.com/kennethreitz/requests)
requests-oauthlib (https://github.com/requests/requests-oauthlib)
oauth2_xfel_client (https://git.xfel.eu/gitlab/ITDM/oauth2_xfel_client)
Installation
Python project
Install requirements, if never done before
1.1. For OS X distributions:
1.1.1. Homebrew brew install python3 1.1.2 Port sudo port install python36 sudo port select --set python3 python36 sudo port install py36-pip sudo port select --set pip pip361.2. For Linux distributions:
sudo apt-get update sudo apt-get install python3.9
Make metadata_client library available in your python environment
2.1. Install it via pip:
# Install dependencies from local wheels files pip install . --no-index --find-links ./external_dependencies/ # Install dependencies from the pypi pip install . # Force re-installation of packages pip install --ignore-installedInstalling it will place two folders under the current Python installation site-packages folder:
metadata_client with the sources;
metadata_client-3.3.0.dist-info/ with Wheels configuration files.
To identify your Python site-packages folder run:
python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
Usage
To use this project you need to import it:
from metadata_client import MetadataClient
Connection to the MdC (Metadata Catalog):
from metadata_client import MetadataClient # Necessary configuration variables to establish a connection # Go to https://in.xfel.eu/metadata/oauth/applications to make a token for # the metadata catalogue. user_id = '201ed15ff071a63e76cb0b91a1ab17b36d5f92d24b6df4497aa646e39c46a324' user_secret = 'a8ae80f5e96531f19bf2d2b6102f5a537196aca44a673ad36533310e07529757' user_email = 'luis.maia@xfel.eu' # metadata_web_app_url = 'https://in.xfel.eu/metadata' token_url = 'https://in.xfel.eu/metadata/oauth/token' refresh_url = 'https://in.xfel.eu/metadata/oauth/token' auth_url = 'https://in.xfel.eu/metadata/oauth/authorize' scope = '' base_api_url = 'https://in.xfel.eu/metadata/api/' # Generate the connection (example with minimum parameter options) client_conn = MetadataClient(client_id=user_id, client_secret=user_secret, user_email=user_email, token_url=token_url, refresh_url=refresh_url, auth_url=auth_url, scope=scope, base_api_url=base_api_url) # Generate the connection (example with all parameter options) client_conn = MetadataClient(client_id=user_id, client_secret=user_secret, user_email=user_email, token_url=token_url, refresh_url=refresh_url, auth_url=auth_url, scope=scope, base_api_url=base_api_url, session_token=None, max_retries=3, timeout=12, ssl_verify=True)
Interaction with the MdC (Metadata Catalog):
2.1 Example data_group_types:
all_group_types = client_conn.get_all_data_group_types() all_group_types # >>> {'success': True, # 'data': [{'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1}, # {'description': '', 'identifier': 'CAL', 'name': 'Calibration', 'flg_available': True, 'id': 2}, # {'description': '', 'identifier': 'PROC', 'name': 'Processed', 'flg_available': True, 'id': 3}, # {'description': '', 'identifier': 'REDU', 'name': 'Reduced', 'flg_available': True, 'id': 4}, # {'description': '', 'identifier': 'SIM', 'name': 'Simulation', 'flg_available': True, 'id': 5}, # {'description': '', 'identifier': 'UNK', 'name': 'Unknown', 'flg_available': True, 'id': 6}], # 'app_info': {}, # 'info': 'Got data_group_type successfully'} all_group_types['success'] # >>> True all_group_types['data'][0] # >>> {'description': '', 'identifier': 'RAW', 'name': 'Raw', 'flg_available': True, 'id': 1} all_group_types['data'][0]['name'] # >>> 'Raw'2.2 Example instruments:
all_xfel_instruments = client_conn.get_all_xfel_instruments() >>> for instrument in all_xfel_instruments['data']: ... print('id = {0} | name = {1}'.format(instrument['id'], instrument['name'])) ... # id = -1 | name = test-instrument # id = 1 | name = SPB/SFX SASE1 # id = 2 | name = FXE SASE1 # id = 3 | name = SQS SASE3 # id = 4 | name = SCS SASE3 # id = 5 | name = MID SASE2 # id = 6 | name = HED SASE2 # id = 7 | name = Hera South Detector Test Stand # id = 8 | name = SASE1 Test Stand # id = 9 | name = SASE2 Test Stand # id = 10 | name = SASE3 Test Stand2.3 Get instrument active proposal:
active_proposal = client_conn.get_active_proposal_by_instrument(1)2.4 Register Run replica:
# (e.g. proposal_number == 1234) # (e.g. proposal_number == 12) # (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC') resp = client_conn.register_run_replica( proposal_number, run_number, repository_identifier ) # resp = {'success': True, # 'info': 'Run replica registered successfully', # 'data': {'experiment_id': '-1', # 'sample_id': '-1', # 'run_id': '1588', # 'data_group_id': '777'}, # 'app_info': {}}2.5 Unregister Run replica:
# (e.g. proposal_number == 1234) # (e.g. proposal_number == 12) # (e.g. repository_identifier == 'XFEL_GPFS_OFFLINE_RAW_CC') resp = client_conn.unregister_run_replica( proposal_number, run_number, repository_identifier ) # resp = {'success': True, # 'info': 'Run replica unregistered successfully', # 'data': {'data_group_id': '-1', # 'repository_id': '1', # 'flg_available': 'false'}, # 'app_info': {}}2.6 Get proposal’s runs:
# (e.g. proposal_number == 1234) # (e.g. page == 1 | Default == 1) # (e.g. page_size == 5 | Default == 100 | Limit: 500) resp = client_conn.get_proposal_runs(proposal_number, page=1, page_size=5) # RESPONSE example # # resp = {'info': 'Got proposal successfully', # 'success': True, # 'data': { # 'proposal': { # 'id': -1, # 'number': 0, # 'title': 'Proposal Title 001' # }, # 'runs': [ # { # 'id': -1, # 'run_number': 1, # 'flg_status': 1, # 'flg_run_quality': -1, # 'size': null, # 'num_files': 0, # 'repositories': { # 'XFEL_TESTS_REPO': { # 'name": 'XFEL Tests Repository', # 'mount_point': '/webstorage/XFEL', # 'data_groups': 1 # } # } # } # ] # }, # 'app_info': {}}
2.7 Get proposal’s samples:
# (e.g. proposal_number == 1234) # (e.g. page == 1 | Default == 1) # (e.g. page_size == 50 | Default == 100 | Limit: 500) resp = client_conn.get_proposal_samples(proposal_number, page=1, page_size=50) # # RESPONSE example # # resp = {'info': 'Got sample successfully', # 'success': True, # 'data': [{'id': -1, # 'name': 'TestSample DO NOT DELETE!', # 'proposal_id': -1, # 'sample_type_id': 1, # 'flg_available': True, # 'url': '', # 'description': ''}], # 'app_info': {}}
For additional examples, please take a look in the tests/ folder.
Development & Testing
When developing, and before commit changes, please validate that:
All tests continue passing successfully (to validate that run pytest):
# Go to the source code directory cd metadata_client # Upgrade package and all its required packages pip install . -U --upgrade-strategy eager # Install test dependencies pip install '.[test]' -U --upgrade-strategy eager # Run all tests using pytest pytest # When running all tests against the standard http application OAUTHLIB_INSECURE_TRANSPORT=1 pytest # Run all tests and get information about coverage for all files inside metadata_client package pytest --cov metadata_client --cov-report term-missing
Code keeps respecting pycodestyle code conventions (to validate that run pycodestyle):
pycodestyle . pycodestyle . --exclude venv
To generate all the wheels files for the dependencies, execute:
# Generate Wheels to itself and dependencies pip wheel --wheel-dir=./external_dependencies . pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies .
Check that you have the desired dependency versions in external_dependencies folder, since no versions are now set in setup.py.
Registering library on https://pypi.org
To register this python library, the following steps are necessary:
# Install twine python -m pip install --upgrade twine # Generates source distribution (.tar.gz) and wheel (.whl) files in the dist/ folder python setup.py sdist python setup.py bdist_wheel # Upload new version .egg and .whl files twine upload dist/* # In case a test is necessary, it is possible to test it against test.pypi.org twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for metadata_client-3.7.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | add3afca61c5e7cb9a36117912b1219cfaa459570463c871db4fe82a956d372f |
|
MD5 | b2ebaa4b99033fd403fbac4191cec953 |
|
BLAKE2b-256 | 7fd1c25e82809947dd5757a1156022e8ea7b593b9f5bd3eef0a0e39b50cb24b8 |