d4kms 0.11

A python package for building d4k microservcies

Introduction

This read me identifies a set of microservices that implement a range of functionality that allows for the handling of clinical trial data in an automated manner. The use cases addressed are

1. Import of data held in a variety of data formats into study designs
2. Export of data in a number of formats

The microservices within the system are:

1. Registration Authority
2. Clinical Recording Model
3. Controlled Terminology
4. Biomedical Concepts
5. SDTM
6. Forms
7. Study

The services are all python services running on FastAPI and uvicorn. They can be run locally using a single shared Neo4j instance or deployed. When deployed each micorservices requires a separate Aura Neo4j instance.

Running Locally

Use a single Neo4 instance will all databases for the microservices setup within one instance. Typical names for a devlepoment environment are

• ra-service-dev
• crm-service-dev
• bc-service-dev
• sdtm-service-dev
• study-service-dev
• form-service-dev

Note: there are no separate data bases for Study Import service or the Study Data Import service

Port Numbers

User Interface Services

Microservice	Port
CT	8000
BC	8001
Study	8002
Form	8003
SDTM	8004

Data Services

Microservice	Port
RA	8010
CRM	8011
CT	8012
BC	8013
Study	8014
Study Import	8015
Form	8016
SDTM	8017
Study Data Import	8018

To Run

Use the microservice.command file to run. Note this file is mac specific. This will start all of the services and the UIs. There is also a dev_server.sh file in each git repo to run each element individually.

Deployed

Overview

To deploy each microservice the following actions should be followed.

1. Create Aura Neo4j instance.
2. Load the data using the data prep utility
3. Deploy the microservice
4. Check all running

Neo4j Aura

Use the Aura console to create the database as per the site instructions. Download the file holding the credentials (username and password) for the instance

Data Load

Overview

From the appropriate data prep project

1. Setup the virtual environment
2. Set the environment variables for the credentials of the neo4j instance
3. Set production
4. Run the load python file

Virtual Environment

Run . ./setup_env.sh

Environment Variables

Modify the .production_env file. The file will need to contain the following lines

NEO4J_URI=<database uri>
NEO4J_DB_NAME=neo4j
NEO4J_USERNAME=<username>
NEO4J_PASSWORD=<password>
GITHUB=<URL of the github repo main page>

The database URI will look something like neo4j+s://a1bc23d4.databases.neo4j.io The githb repo URL will look something like https://raw.githubusercontent.com/data4knowledge/ra_prep/main/

Set Production

We need to tell the load utility to use the production database rather than the local (development) one. This is done via setting the PYTHON_ENVIRONMENTenvironment variable. To so this run . ./set_production.sh

Load Data

There will be a python rpogram to load the data. It will be named stage_<n>_load.py. Run this program by entering python stage_2_load.py, here n is 2.

The output from the program should look somethig like

Deleting database ...
Database deleted. Load new data ...
https://raw.githubusercontent.com/data4knowledge/ra_prep/main/load_data/node-namespace-1.csv
https://raw.githubusercontent.com/data4knowledge/ra_prep/main/load_data/relationship-manages-1.csv
https://raw.githubusercontent.com/data4knowledge/ra_prep/main/load_data/node-registration_authority-1.csv
<Record file='progress.csv' source='file' format='csv' nodes=9 relationships=7 properties=58 time=4294 rows=0 batchSize=-1 batches=0 done=True data=None>
Load complete. 9 nodes and 7 relationships loaded in 4294 milliseconds.

Deploy Microservice

The microservices are deployed using the fly.io cloud service.

1. General python instructions are available here https://fly.io/docs/languages-and-frameworks/python/
2. Installing the fly command line tool is detailed here https://fly.io/docs/hands-on/install-flyctl/
3. deploy the app
4. Set environment variables

Environment Variables

Setting environment variables on the server is achieved by using th ecommand line program, either one at a time

fly secrets set SUPER_SECRET_KEY=password1234

or multiple values, note space delimited

fly secrets set NEO4J_URI=xxx NEO4J_PASSWORD=yyy

Using Auth0 by Okta

Environment Keys

AUTH0_SESSION_SECRET=<session secret key>
AUTH0_DOMAIN=<from the Auth0 configuation>
AUTH0_CLIENT_ID=<from the Auth0 configuation>
AUTH0_CLIENT_SECRET=<from the Auth0 configuation>
AUTH0_AUDIENCE=<from the Auth0 configuation>
AUTH0_MNGT_CLIENT_ID=<Management API client ID>
AUTH0_MNGT_CLIENT_SECRET=<Management API client secret>
ROOT_URL=<base URL for the app>

For the session secret run cat /dev/urandom | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 50 | head -n 1to generate a 50 character randon string

For the root URL use http:\\localhost:8000or similar for a locally running app or https:\\xxx.fly.devfor a deployed app

Code in Main Program

Use this initialisation code

authorisation = Auth0Service(app)
authorisation.register()

Need a local method to protect an endpoint

def protect_endpoint(request: Request) -> None:
  authorisation.protect_route(request, "/login")

and then protect an endpoint by using

@app.get("/index", dependencies=[Depends(protect_endpoint)])
def index(request: Request):
  ...code...

Loading Studies & Data

To be defined.

Building package

Build and Upload

Use pip to install build and twine. Use the following commands to build

python -m build

and upload to pypi.org using the command

twine upload dist/*

Upload requires a token.

Token

Generate an API token on pypi.org. Then, to use the API token:

Set your username to __token__
Set your password to the token value, including the pypi- prefix