gazoo-research-utils 2.1.0

Utilities for the Analysis of Gazoo Research Data

GazooResearchUtils

GazooResearchUtils is a Python library designed for analyzing medical data from GazooResearch. It provides tools for filtering, aggregating, and visualizing longitudinal patient data with support for time-based analyses anchored to specific events.

Installation

pip install gazoo-research-utils

Data Structure

Gazoo Research data follows a denormalized structure with this hierarchy:

• id: Patient identification
• collection: A collection of related tags (e.g., 'c61' for oncology data, '0' for demographics)
• tag: Name of the tag (e.g., 'dob', 'psa', 'surgery')
• tag_id: Unique identifier for the tag
• field: Field name (e.g., 'date', 'value', 'units', 'mrn')
• data_type: Data type ('text', 'categorical', 'number', 'date')
• value: The actual value
• phi: 0 or 1, indicating if the data element is protected health information

Example Data Structure

id,collection,tag,tag_id,field,data_type,value,phi
111111,0,id,#6bed55ed-f844-4c88-ab74-af0e74aa2547,mrn,text,111111,1
111111,0,last-name,bd7e898c-9732-4320-8227-a1b90cc8c80e,value,text,Bing,1
111111,0,dob,#5578f001-b34b-4465-b699-8c8ff6c11d89,date,date,1952-12-17,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,date,date,2010-01-01,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,type,categorical,prostatectomy,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,date,date,2019-02-22,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,value,number,0.28,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,units,categorical,ng/ml,0

Core Concepts

Filters

Filters define criteria for selecting patient data. A filter is specified as a dictionary with these fields:

filter = {
    'collection': str,    # Optional: collection name
    'tag': str,           # Optional: tag name
    'field': str,         # Optional: field name
    'exact': [str, ...],  # Optional: exact value matches
    'between': [float, float]  # Optional: numeric range
}

Anchors

Anchors define reference points for time-based analysis. They identify specific events (like surgery or a lab test) from which other measurements are compared.

Anchor Format

An anchor follows the same structure as a filter:

anchor = {
    'collection': str,    # Optional
    'tag': str,           # Required: tag name
    'field': str,         # Optional: field name
    'exact': [str, ...],  # Optional
    'between': [float, float]  # Optional
}

For time-based analyses, anchors should include a date field (e.g., field='date' or field='start-date').

Tag Sequences

Multiple anchors can be specified as a sequence. All anchors in the sequence must occur within the specified time_delta:

anchor_sequence = [
    {'tag': 'surgery'},
    {'tag': 'radiation'}
]

Instance Selection

When multiple instances of an anchor sequence exist for a patient, use instance to specify which occurrence to use:

• instance = 0 - First occurrence of the sequence
• instance = 1 - Second occurrence
• instance = -1 - Last occurrence

Getting Started

Loading Data

from gazoo_research_utils import load_data, validate_gazoo_research_df

# Load data from CSV
df = load_data("path/to/prostate-data.csv")

# Validate the DataFrame structure
validate_gazoo_research_df(df)

Filtering Data

Simple Filters

from gazoo_research_utils import get_tags_where_filter, filter

# Filter by collection
collection_filter = {'collection': 'c61'}
data_c61 = get_tags_where_filter(df, collection_filter)

# Filter by tag
psa_filter = {'tag': 'psa'}
psa_data = get_tags_where_filter(df, psa_filter)

# Filter by field with exact match
psa_value_filter = {'tag': 'psa', 'field': 'value'}
psa_values = get_tags_where_filter(df, psa_value_filter)

# Filter numeric values within range
psa_range_filter = {'tag': 'psa', 'field': 'value', 'between': [0.2, 5.0]}
psa_in_range = get_tags_where_filter(df, psa_range_filter)

# Multiple filters (AND logic)
multi_filters = [
    {'tag': 'gleason-grade-group', 'field': 'value', 'exact': [4, 5]},
    {'tag': 'psa', 'field': 'value', 'between': [0.2, 0.5]}
]
matched_patients = filter(df, multi_filters)

Using Filter Objects

from gazoo_research_utils.models.filter_model import Filter

# Create a Filter object
filter_obj = Filter(collection='c61', tag='psa', field='value', between=[0.2, 5.0])
result = get_tags_where_filter(df, filter_obj)

Anchor-Based Analysis

Getting Anchors

from gazoo_research_utils import get_anchors
from datetime import timedelta

# Single anchor
anchor = [{'tag': 'surgery', 'field': 'date'}]
anchors_df = get_anchors(df, anchor, time_delta=0, instance=0)

# Anchor sequence (surgery followed by radiation within 60 days)
anchor_sequence = [
    {'tag': 'surgery', 'field': 'date'},
    {'tag': 'radiation', 'field': 'date'}
]
anchors_df = get_anchors(
    df, 
    anchor_sequence, 
    time_delta=60,  # 60 days max between anchors
    instance=0      # First occurrence
)

# Using timedelta for more precision
anchors_df = get_anchors(
    df, 
    [{'tag': 'surgery'}], 
    time_delta=timedelta(days=5, hours=3),  # 5 days and 3 hours
    instance=0
)

Filtering Data Around Anchors

from gazoo_research_utils import filter_around_anchor

# Get patients with PSA value within 30 days before to 7 days after surgery
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)
filters = [{'collection': 'c61', 'tag': 'psa', 'field': 'value'}]
result = filter_around_anchor(df, anchors_df, filters, [-30.0, 7.0])

# Get patients with pT category within 1 day of surgery
filters = [
    {'collection': 'c61', 'tag': 'pT', 'field': 'T', 'exact': ['2a', '2b']}
]
result = filter_around_anchor(df, anchors_df, filters, [-1.0, 1.0])

Time Delta

The time_delta parameter sets the maximum number of days or timedelta allowed between sequential anchors. This is only relevant when specifying multiple anchors.

anchor_sequence = [
    {'tag': 'surgery'}, 
    {'tag': 'radiation'}
]

# Using float (days)
anchors_df = get_anchors(df, anchor_sequence, time_delta=60.0, instance=0)

# Using timedelta
from datetime import timedelta
anchors_df = get_anchors(df, anchor_sequence, time_delta=timedelta(days=60), instance=0)
anchors_df = get_anchors(df, anchor_sequence, time_delta=timedelta(days=5, hours=3), instance=0)

This is useful for filtering out scenarios like salvage radiotherapy, which typically occurs months or years later rather than immediate adjuvant treatment.

Time Series Analysis

Getting Tag Time Series

from gazoo_research_utils import get_tag_time_series

# Get time series data for PSA
df_plot = get_tag_time_series(df, tag='psa')

# With anchor normalization
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=90.0, instance=0)
df_plot = get_tag_time_series(df, tag='psa', anchors=anchors_df)

The returned DataFrame includes:

• id: Patient identifier
• date or start-date: The measurement date
• value: The measurement value
• Additional columns (e.g., units, mrn)
• time-delta: Days from the first measurement (or anchor date if provided)

Visualization

from gazoo_research_utils import line_plot, categorical_plot

# Line plot for numerical data
fig = line_plot(
    df_plot, 
    field='value', 
    x_scale='year',  # or 'day'
    xlabel='Years Since First Measurement', 
    ylabel='PSA (ng/mL)'
)

# Categorical plot
fig = categorical_plot(
    df_plot,
    field='T',
    y_scale=['0', '1', '2a', '2b', '3a'],
    x_scale='day',
    xlabel='Days Since First Measurement',
    ylabel='T Category'
)

Patient-Level Transformations

Pivot Data

from gazoo_research_utils import pivot

# Get PSA data
psa_df = get_tags_where_filter(df, {'tag': 'psa'})

# Pivot to human-readable format
psa_pivoted = pivot(psa_df)

# Access PSA values for all patients
psa_values = psa_pivoted["c61:psa:value"]

Calculate Age

from gazoo_research_utils import calculate_age

# Calculate age at anchor date
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)
df_with_age = calculate_age(df, anchors_df)

Multi-lesion Transformation

from gazoo_research_utils import multilesion_transforation

# Transform patient-level data to multi-target level
df_multi = multilesion_transforation(df)

Statistical Analysis

Describe Fields Around Anchors

from gazoo_research_utils import describe_fields, plot_table1

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=90.0, instance=0)

# Extract field values within 90 days before to 1 day after surgery
fields = [
    {'collection': 'c61', 'tag': 'psa', 'field': 'value'},
    {'tag': 'pT', 'field': 'T'}
]
results = describe_fields(df, anchors_df, fields, [-90.0, 1.0])

# Plot results
fig = plot_table1([results])

Kaplan-Meier Survival Analysis

from gazoo_research_utils import kaplan_meier, plot_km_curves

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)

# Define event tags
event_tags = [
    {'tag': 'progression', 'field': 'date'}
]

# Calculate Kaplan-Meier data (time in months)
km = kaplan_meier(df, anchors_df, event_tags, time_unit="months")

# Plot survival curve
fig = plot_km_curves(km, time_column="time", event_column="event")

Time in Months

from gazoo_research_utils import time_in_months
from datetime import datetime

start_date = datetime.strptime("2019-01-01", "%Y-%m-%d")
end_date = datetime.strptime("2020-06-15", "%Y-%m-%d")
months = time_in_months(start_date, end_date)  # Returns 17

Utility Functions

Data Dictionary

from gazoo_research_utils import get_data_dictionary

# Get data dictionary
dictionary = get_data_dictionary(df)

Get Parameters

from gazoo_research_utils import get_parameters

# Get parameter options
parameters = get_parameters(df)

Patient and Tag Iterators

from gazoo_research_utils import patient_iterator, tag_iterator

# Apply function to each patient
def patient_fn(patient_df):
    # Process patient data
    return patient_df

result = patient_iterator(df, patient_fn)

# Apply function to each tag
def tag_fn(tag_df):
    # Process tag data
    return tag_df

result = tag_iterator(df, tag_fn)

Start at Anchor

from gazoo_research_utils import start_at_anchor

# Remove all data before anchor date
anchors_df = get_anchors(df, [{'tag': 'surgery'}], time_delta=0, instance=0)
df_after_anchor = start_at_anchor(df, anchors_df)

Link by Field Value

from gazoo_research_utils import link_by_field_value

# Get all tags with the same field value as a reference tag
tag_link = {'collection': 'c61', 'tag': 'target', 'field': 'mrn'}
linked_data = link_by_field_value(df, tag_link, instance=0)

Complete Example

from gazoo_research_utils import (
    load_data, 
    get_anchors, 
    filter_around_anchor, 
    get_tag_time_series,
    line_plot,
    describe_fields,
    plot_table1,
    kaplan_meier,
    plot_km_curves
)
from datetime import timedelta

# Load data
df = load_data("prostate-data.csv")

# 1. Get patients who had surgery followed by radiation within 90 days
anchor_sequence = [
    {'tag': 'surgery', 'field': 'date'},
    {'tag': 'radiation', 'field': 'date'}
]
anchors_df = get_anchors(df, anchor_sequence, time_delta=90, instance=0)

# 2. Get PSA values 30 days before to 7 days after surgery
filters = [{'collection': 'c61', 'tag': 'psa', 'field': 'value'}]
psa_around_surgery = filter_around_anchor(df, anchors_df, filters, [-30.0, 7.0])

# 3. Get time series for PSA normalized to surgery date
df_time_series = get_tag_time_series(df, tag='psa', anchors=anchors_df)

# 4. Visualize PSA time series
fig = line_plot(
    df_time_series,
    field='value',
    x_scale='year',
    xlabel='Years Since Surgery',
    ylabel='PSA (ng/mL)'
)

# 5. Describe patient characteristics at surgery
fields = [
    {'tag': 'age', 'field': 'value'},
    {'tag': 'pT', 'field': 'T'},
    {'tag': 'gleason-grade-group', 'field': 'value'}
]
results = describe_fields(df, anchors_df, fields, [-1.0, 1.0])
fig_table = plot_table1([results])

# 6. Kaplan-Meier survival analysis
event_tags = [{'tag': 'progression', 'field': 'date'}]
km = kaplan_meier(df, anchors_df, event_tags, time_unit="months")
fig_km = plot_km_curves(km)

API Reference

Data Loading & Validation

Function	Description
load_data(file_path)	Load data from CSV with type conversion
validate_gazoo_research_df(df)	Validate DataFrame structure

Filtering

Function	Description
get_tags_where_filter(df, filter)	Get rows matching filter criteria
filter(df, filters)	Filter patients by multiple criteria

Anchor Functions

Function	Description
get_anchors(df, anchors, time_delta, instance)	Get anchor dates for patients
filter_around_anchor(df, anchors_df, filters, search_range)	Filter data around anchor dates
start_at_anchor(df, anchors)	Remove data before anchor dates
calculate_age(df, anchors)	Calculate patient age at anchor

Time Series

Function	Description
get_tag_time_series(df, tag, anchors)	Get time series for a tag
pivot(df)	Pivot data to wide format
line_plot(df, field, x_scale, xlabel, ylabel)	Line plot for numerical data
categorical_plot(df, field, y_scale, ...)	Plot for categorical data

Statistical Analysis

Function	Description
describe_fields(df, anchors, fields, search_range)	Extract field values around anchors
plot_table1(df_describes)	Create Table 1 visualization
kaplan_meier(df, anchors, event_tags, time_unit)	Kaplan-Meier survival analysis
plot_km_curves(km, time_column, event_column)	Plot Kaplan-Meier curves
time_in_months(start, end)	Calculate months between dates

Utility Functions

Function	Description
get_data_dictionary(df)	Get data dictionary
get_parameters(df)	Get parameter options
patient_iterator(df, patient_fn)	Apply function to each patient
tag_iterator(df, tag_fn)	Apply function to each tag
link_by_field_value(df, tag_link, instance)	Link tags by field value
multilesion_transforation(df)	Transform to multi-target analysis

Related

For more information, see the Gazoo Research documentation.