calibration-client 11.3.0

Python Client for European XFEL Calibration Catalogue Web App available at https://in.xfel.eu/calibration

Calcat is the Web App design for Calibration Constants Data Management at European XFEL.

This library (calibration_client) is a client for the RESTful APIs exposed by the European XFEL Calibration Constants Catalogue Web Application - calCat (https://in.xfel.eu/calibration).

Repository:

• https://git.xfel.eu/ITDM/calibration_client

Dependencies:

• oauthlib (https://pypi.python.org/pypi/oauthlib)

• requests (https://github.com/psf/requests)

• requests-oauthlib (https://github.com/requests/requests-oauthlib)

• oauth2_xfel_client (https://git.xfel.eu/ITDM/oauth2_xfel_client)

• pytz (https://pypi.org/project/pytz/)

Installation

Python project

1. Install requirements, if never done before

1.1. For OS X distributions:

1.1.1. Homebrew

      brew install python3

1.1.2 Port

      sudo port install python36

      sudo port select --set python3 python36

      sudo port install py36-pip
      sudo port select --set pip pip36

1.2. For Linux distributions:

sudo apt-get update
sudo apt-get install python3.9

2. Make calibration_client library available in your python environment

2.1. Install it via pip:

# Install dependencies from local wheels files
pip install . --no-index --find-links ./external_dependencies/

# Install dependencies from the pypi
pip install .

# Force re-installation of packages
pip install . --ignore-installed

Installing it will place two folders under the current Python installation site-packages folder:

• calibration_client with the sources;

• calibration_client-11.3.0.dist-info/ with Wheels configuration files.

To identify your Python site-packages folder run:

python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

Usage

To use this project you need to import it:

from calibration_client import CalibrationClient

1. Connection to the MdC (Metadata Catalog):

   from calibration_client import CalibrationClient
   
   # Necessary configuration variables to establish a connection
   # Go to https://in.xfel.eu/calibration/oauth/applications to make a token for
   # the calibration catalogue.
   user_id = 'PUT_HERE_YOUR_CLIENT_KEY'
   user_secret = 'PUT_HERE_YOUR_SECRET_KEY'
   user_email = '[e-mail]'
   #
   metadata_web_app_url = 'https://in.xfel.eu/calibration'
   token_url = 'https://in.xfel.eu/calibration/oauth/token'
   refresh_url = 'https://in.xfel.eu/calibration/oauth/token'
   auth_url = 'https://in.xfel.eu/calibration/oauth/authorize'
   scope = ''
   base_api_url = 'https://in.xfel.eu/calibration/api/'
   
   # Authentication via OAUTH2
   # Generate the connection (example with minimum parameter options)
   client_conn = CalibrationClient(client_id=user_id,
                                   client_secret=user_secret,
                                   user_email=user_email,
                                   token_url=token_url,
                                   refresh_url=refresh_url,
                                   auth_url=auth_url,
                                   scope=scope,
                                   base_api_url=base_api_url)
   
   # Authentication via OAUTH2
   # Generate the connection (example with all parameter options)
   client_conn = CalibrationClient(use_oauth2=True
                                   client_id=user_id,
                                   client_secret=user_secret,
                                   user_email=user_email,
                                   token_url=token_url,
                                   refresh_url=refresh_url,
                                   auth_url=auth_url,
                                   scope=scope,
                                   base_api_url=base_api_url,
                                   session_token=None,
                                   max_retries=3,
                                   timeout=12,
                                   ssl_verify=True)
   
   # Authentication via Proxy Service
   # Generate the connection (example with minimum parameter options)
   client_conn = CalibrationClient(use_oauth2=False,
                                   base_api_url="http://HOSTNAME:PORT/api/")

2. Interaction with the CalCat (Calibration Catalog):

2.1 Example data_group_types:

params_h = { 'detector_identifier': 'TEST_DET_CI-2',
             'snapshot_at': '' }

resp = client_conn.get_all_phy_det_units_from_detector(params_h)

resp
# >>> {'success': True,
#      'data': [ { "id":-1,
#                  "physical_name":"PHYSICAL_DETECTOR_UNIT-1_DO_NOT_DELETE",
#                  "karabo_da":"TEST_DAQ_DA_01",
#                  "virtual_device_name":"Q1M1",
#                  "uuid":1000,
#                  "float_uuid":4.94e-321,
#                  "detector_type_id":-1,
#                  "detector_id":-2,
#                  "flg_available":true,
#                  "description":"None",
#                  "detector":{
#                     "id":-2,
#                     "name":"DET_TEST-2_DO_NOT_DELETE",
#                     "identifier":"TEST_DET_CI-2",
#                     "karabo_name":"TEST_DET_CI_-2",
#                     "karabo_id_control":"TEST_DET_CI-2_CTRL00",
#                     "flg_available":true,
#                     "description":"None"
#                  },
#                  "detector_type":{
#                     "id":-1,
#                     "name":"UNIT_TEST_DETECTOR_TYPE-1_DO_NOT_DELETE",
#                     "flg_available":true,
#                     "description":"None"
#                  }
#                },
#                { ... },
#                { ... } ],
#      'app_info': {},
#      'info': 'Got physical_detector_unit successfully'}

resp['success']
# >>> True

resp['data'][0]['karabo_da']
# >>> 'TEST_DAQ_DA_01'

For additional examples, please take a look in the tests/ folder.

Development & Testing

When developing, and before commit changes, please validate that:

1. All tests continue passing successfully (to validate that run pytest):

   # Go to the source code directory
   cd calibration_client
   
   # Upgrade package and all its required packages
   pip install . -U --upgrade-strategy eager
   
   # Install test dependencies
   pip install '.[test]' -U --upgrade-strategy eager
   
   # Run all tests using pytest
   pytest
   
   # When running all tests against the standard http application
   OAUTHLIB_INSECURE_TRANSPORT=1 pytest
   
   # Run all tests and get information about coverage for all files inside calibration_client package
   pytest --cov calibration_client --cov-report term-missing

2. Code keeps respecting pycodestyle code conventions (to validate that run pycodestyle):

   pycodestyle .
   pycodestyle . --exclude venv

3. To generate all the wheels files for the dependencies, execute:

   # Generate Wheels to itself and dependencies
   pip wheel --wheel-dir=./external_dependencies .
   pip wheel --wheel-dir=./external_dependencies --find-links=./external_dependencies .

4. Check that you have the desired dependency versions in external_dependencies folder, since no versions are now set in setup.py.

Registering library on https://pypi.org

To register this python library, the following steps are necessary:

# Install twine
python -m pip install --upgrade twine

# Generates source distribution (.tar.gz) and wheel (.whl) files in the dist/ folder
python setup.py sdist
python setup.py bdist_wheel

# Upload new version .egg and .whl files
twine upload dist/*

# In case a test is necessary, it is possible to test it against test.pypi.org
twine upload --repository-url https://test.pypi.org/legacy/ dist/* --verbose