gazoo-research-utils 2.0.1

Utilities for the Analysis of Gazoo Research Data

GazooResearchUtils

GazooResearchUtils is a python library whose purpose is to analyze medical data from GazooResearch.

Example output from GazooResearch prostate-data.csv

The data is denormalized on fields, but ultimately has this hierarchy.
id -> collection -> tag -> (field, data_type, value, phi)

Column descriptions are as follows

id: patient identification collection: A collection of related tags tag: name of tag (eg. dob, psa) tag_id: Unique identifier field: tag name (date, value, date) data_type: data type held in the value column. Options are: text, categorical, number, date). dates will be of this form YYYY-MM-DD value: value of field phi: 0 or 1, depending on if the data element is protected health information

Issues

The value field may be empty, or will contain data conforming to the data_type structure.

id,collection,tag,tag_id,field,data_type,value,phi
111111,0,id,#6bed55ed-f844-4c88-ab74-af0e74aa2547,mrn,text,111111,1
111111,0,last-name,bd7e898c-9732-4320-8227-a1b90cc8c80e,value,text,Bing,1
111111,0,first-name,#4ff0ace3-e804-4d55-97a0-22cc6f927327,value,text,Chandler,1
111111,0,dob,#5578f001-b34b-4465-b699-8c8ff6c11d89,date,date,1952-12-17,0
111111,0,document,#ce5ab218-8e98-417b-b843-965ea637ca57,date,date,2023-12-06,0
111111,0,document,#ce5ab218-8e98-417b-b843-965ea637ca57,type,categorical,clinical,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,date,date,2010-01-01,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,type,categorical,prostatectomy-lymphadenectomy,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,target-id,text,prostate,0
111111,c61,R,#358ebce7-9e7d-441c-a976-38d02d3da58e,date,date,2010-01-01,0
111111,c61,R,#358ebce7-9e7d-441c-a976-38d02d3da58e,R,categorical,0,0
111111,c61,pT,#2485755d-aa2a-4fa9-a263-4d11c9370145,date,date,2010-01-01,0
111111,c61,pT,#2485755d-aa2a-4fa9-a263-4d11c9370145,T,categorical,2c,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,date,date,2010-01-01,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,N,categorical,0,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,involved-lymph-nodes,number,,0
111111,c61,pN,#8d04921a-cee6-4c7c-9eb2-d954c1b33008,total-lymph-nodes,number,,0
111111,c61,gleason-grade-group,#55109b94-41fe-4b11-a47a-824970ab9dfb,date,date,2010-01-01,0
111111,c61,gleason-grade-group,#55109b94-41fe-4b11-a47a-824970ab9dfb,value,categorical,2,0
111111,c61,gleason-grade-group,#55109b94-41fe-4b11-a47a-824970ab9dfb,target-id,text,prostate,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,date,date,2019-02-22,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,value,number,0.28,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,units,categorical,ng/ml,0
111111,c61,psa,91a5221d-de16-4af9-acb6-82b7d37ed118,date,date,2019-06-12,0
111111,c61,psa,91a5221d-de16-4af9-acb6-82b7d37ed118,value,number,0.31,0
111111,c61,psa,91a5221d-de16-4af9-acb6-82b7d37ed118,units,categorical,ng/ml,0
111111,c61,psa,6ed2dc02-af27-4395-a73e-36af30b7fed6,date,date,2023-12-06,0
111111,c61,psa,6ed2dc02-af27-4395-a73e-36af30b7fed6,value,number,0.01,0
111111,c61,psa,6ed2dc02-af27-4395-a73e-36af30b7fed6,units,categorical,ng/ml,0
111111,c61,androgen-deprivation-therapy,#ed2d15d6-2650-41fe-af7d-e9b6ac1140d6,start-date,date,2019-07-15,0
111111,c61,androgen-deprivation-therapy,#ed2d15d6-2650-41fe-af7d-e9b6ac1140d6,end-date,date,2019-12-06,0
111111,c61,androgen-deprivation-therapy,#ed2d15d6-2650-41fe-af7d-e9b6ac1140d6,medication,categorical,leuprolide,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,start-date,date,2019-08-15,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,end-date,date,2019-10-03,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,field,categorical,prostate-bed-pelvic-lymph-nodes,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,dose,number,,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,unit,categorical,Gy,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,fractions,number,,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,technique,categorical,unknown,0
111111,c61,external-radiation,#407ad0f8-a8ef-4e21-a41c-6ba11a85d777,target-id,text,prostate,0
111111,c61,biochemical-progression,#e6458863-ceba-4194-a361-df96407d5c3a,date,date,,0
111111,c61,biochemical-progression,#e6458863-ceba-4194-a361-df96407d5c3a,type,categorical,aua,0
111111,0,last-name,#3dd60c0d-d51d-4c4a-8250-715a8f415207,value,text,Bing,1
111111,0,id,#1c9d95e8-7d8b-4102-990b-02ebea1e9a14,mrn,text,111111,1
111111,0,dob,#47a681ae-23a8-41e8-b569-32039d3f47b4,date,date,1952-12-17,0
111111,c61,psa,#b482b981-cf18-420f-b518-d3f6279f03c5,date,date,2019-02-22,0
111111,c61,psa,#b482b981-cf18-420f-b518-d3f6279f03c5,value,number,0.28,0
111111,c61,psa,#b482b981-cf18-420f-b518-d3f6279f03c5,units,categorical,ng/ml,0
333333,0,id,c918371b-01f7-49ee-acd8-b00dee6e49e9,mrn,text,333333,1
333333,0,first-name,dc110270-d5c1-431c-8f90-86459bed3a36,value,text,Joey,1
333333,0,last-name,2d9ae449-8d3f-4493-82a8-75321f4df6cd,value,text,Tribbiani,1
333333,0,dob,d1044870-b049-4465-b306-e0eb1a20b8c2,date,date,1948-01-22,0
333333,c61,surgery,#a5b2b0b0-1752-401f-a4d6-63f41dec128b,date,date,2019-08-31,0
333333,c61,surgery,#a5b2b0b0-1752-401f-a4d6-63f41dec128b,type,categorical,prostatectomy,0
333333,c61,surgery,#a5b2b0b0-1752-401f-a4d6-63f41dec128b,target-id,text,prostate,0
333333,c61,pT,#3d0cf1cf-3608-4f98-95f9-7569d0f57a63,date,date,2019-08-31,0
333333,c61,pT,#3d0cf1cf-3608-4f98-95f9-7569d0f57a63,T,categorical,3a,0

Anchors

Often when calculating time based analysis, we need a starting point. An anchor is used to define that starting point. An anchor json follow the same structure as a filter json. Anchors should have a date field.

anchor = {'collection':str, 'tag': str, 'field': str, 'exact': [str, str,...], 'between': [float, float]}

Tag Sequence

Sometimes you want to find an ordered scenario, for example you may want to find patients that first have surgery and then adjuvant radiation. Your anchor may have this structure:

anchor = [{'tag': 'surgery'},{'tag':'radiation'}]

Note that you are not interested in the type of surgery, or type of radiation.

Time Delta

time_delta represents the maximum number of days between anchor tags. This is only relavant if there is >1 anchor tag.

Instance

There may be multiple instances of an anchor sequence. 'instance' defines which one matters. instance = 0 - Means that the 1st instance of this anchor sequence will be used.

Time Delta

Represents the maximum allowable days between anchors.

anchor = [{'tag': 'surgery'},{'tag':'radiation'}]
time_delta = 60
# the maximum time between surgery and radaition is 60 days. Useful if you don't want to indentify patients with salvage radiotherapy that typically occurs many months to years later.

Data Filtering

A filter json will be used to find patients that have matching tags. The filter does not need to include all fields. When you want to find patients matching multiple critera specify 'filters' which is an array of filter json.

filter = {'collection':str, 'tag': str, 'field': str, 'exact': [str, str,...], 'between': [float, float]}

It may include:

• collection alone

filter = {'collection':'c61'}

• tag alone

filter = {'tag':'psa'}

• field alone

filter = {'tag':'psa'}

• field with a value that exactly matches

filter = {'field':'histology', 'exact': ['melanoma', 'squamous-cell-carcinoma']}

• field with a value that is a number and is between two values

filter = {'field':'psa', 'between': [0.2, 0.5]}

• or a combination of the above

filter = {'tag':'psa', 'field':'psa', 'between': [0.2, 0.5]}

When you want to find patients matching multiple critera specify 'filters' which is an array of filter json.
The below filters finds patients who have both gleason grade group 4 OR 5, AND psa between 0.2-0.5.

filters = [{'field':'gleason-grade-group', 'exact': [4,5]}, {'tag':'psa', 'field':'psa', 'between': [0.2, 0.5]}]

Search Range

The represents the time period before and after the anchor should be examined for the filters criteria (units: days).
For example, imagine you are researching prostate cancer patients getting radiation. You are interested in looking at patients with a pre-RT psa >=20 ng/mL.

anchors = [{'tag':'radiation'}]
filters = [{'tag':'psa', 'field':'psa', 'between': [20, 10000]}]
search_range = [-90, 1] # Include patients who had a PSA >20 ng/mL 90 days before starting radiation, or 1 day after radiotherapy.