Gazoo Research Utils

A comprehensive Python library for analyzing medical data from GazooResearch. This library provides tools for filtering, aggregating, visualizing, and performing statistical analysis on longitudinal patient data with support for time-based analyses anchored to specific clinical events.

Table of Contents

• Features
• Installation
• Upgrading from 2.x
• Quick Start
• Data Structure
• Core Concepts
• API Reference
• Examples
• Development
• Contributing
• License

Features

• Data Loading & Validation: Load and validate Gazoo Research CSV data
• Flexible Filtering: Filter patient data by collection, tag, field, exact values, and numeric ranges
• Anchor-Based Analysis: Define reference points (anchors) for time-based analysis
• Time Series Analysis: Extract and normalize time series data around clinical events
• Statistical Analysis: Perform survival analysis with Kaplan-Meier curves
• Data Visualization: Create line plots and categorical plots for data exploration
• Patient-Level Transformations: Pivot data, calculate age, and transform to multi-lesion analysis

Installation

# Install from PyPI
pip install gazoo-research-utils

# Install with development dependencies
pip install gazoo-research-utils[dev]

Requirements

• Python 3.11+
• pandas >= 3.0.0
• pydantic >= 2.12.5
• matplotlib
• lifelines >= 0.28.0

Upgrading from 2.x

Version 3.0.0 changes the import path and tightens filter validation; 3.1.0 raises minimum dependency versions and adds pivot(collapse=True). There is one breaking import change and one behaviour change to be aware of.

2.x	3.x
from gazoo_research_utils import Analysis as gz	import gazoo_research_utils as gz
from gazoo_research_utils.Analysis import load_data	from gazoo_research_utils import load_data
{'tag': 'psa', 'value': ['1', '2']}	{'tag': 'psa', 'field': 'value', 'exact': ['1', '2']}

Filter validation is stricter. Unknown keys now raise a ValidationError instead of being silently ignored, between requires min <= max, and exact or between require field. This turns previously silent typos (such as 'value' instead of 'exact') into immediate errors instead of returning unfiltered data.

# String / categorical matching uses `exact`
{'collection': 'c61', 'tag': 'pT', 'field': 'T', 'exact': ['2c', '3a']}

# Numeric ranges use `between`
{'collection': 'c61', 'tag': 'psa', 'field': 'value', 'between': [0.1, 10]}

New in 3.0.0

• tag_instances(df) - one row per tag instance with its date.
• summarize_fields(df, anchors, fields, search_range) - a tidy one-row-per-variable summary instead of a list of frames.
• kaplan_meier_groups(...) - subgroup Kaplan-Meier curves with a log-rank test.
• get_tag_time_series(..., drop_before_anchor=True) - discard pre-anchor measurements.
• load_data now accepts file-like objects as well as paths.

import gazoo_research_utils as gz

df = gz.load_data("prostate-data.csv")

New in 3.1.0

• pivot(df, collapse=True) - one row per patient instead of one row per tag instance. Additive: the default is unchanged, so existing calls behave exactly as they did in 3.0.0.
• Minimum dependency versions raised to the ones the test suite runs against (pandas 3.1, numpy 2.4.2, matplotlib 3.10.8, lifelines 0.30). get_parameters no longer uses DataFrame.rename(inplace=True), which pandas 3.1 deprecates and pandas 4.0 removes.

Quick Start

from gazoo_research_utils import load_data, get_anchors, filter_around_anchor
from gazoo_research_utils import get_tag_time_series, line_plot

# Load data
df = load_data("prostate-data.csv")

# Get anchors (surgery date for each patient)
anchors_df = get_anchors(df, [{'tag': 'surgery', 'field': 'date'}], time_delta=0, instance=0)

# Get PSA values within 30 days before to 7 days after surgery
psa_data = filter_around_anchor(
    df, 
    anchors_df, 
    [{'collection': 'c61', 'tag': 'psa', 'field': 'value'}], 
    [-30.0, 7.0]
)

# Get time series for PSA
df_time_series = get_tag_time_series(df, tag='psa', anchors=anchors_df)

# Visualize
fig = line_plot(df_time_series, field='value', xlabel='Days Since Surgery', ylabel='PSA')

Data Structure

Gazoo Research data follows a denormalized structure with the following columns:

Column	Description	Example
id	Patient identification	111111
collection	Collection of related tags (e.g., 'c61' for oncology)	'c61', '0'
tag	Name of the tag (e.g., 'dob', 'psa', 'surgery')	'psa', 'surgery'
tag_id	Unique identifier for the tag	'#6082ea75-06c3-4082-9c69-918d0df8c619'
field	Field name	'date', 'value', 'units', 'mrn'
data_type	Data type	'text', 'categorical', 'number', 'date'
value	The actual value	'2010-01-01', 0.28, 'prostatectomy'
phi	Protected health information flag (0 or 1)	0, 1

Example Data

id,collection,tag,tag_id,field,data_type,value,phi
111111,0,dob,#5578f001-b34b-4465-b699-8c8ff6c11d89,date,date,1952-12-17,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,date,date,2010-01-01,0
111111,c61,surgery,#6082ea75-06c3-4082-9c69-918d0df8c619,type,categorical,prostatectomy,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,date,date,2019-02-22,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,value,number,0.28,0
111111,c61,psa,64987467-bcc2-4c23-a943-899467d93981,units,categorical,ng/ml,0

Collections

Common collection identifiers:

• '0' - Demographics and patient-level data
• 'c61' - Prostate cancer oncology data
• 'c44.92' - Multi-lesion analysis data

Core Concepts

Filters

Filters define criteria for selecting patient data. Every key is optional, and a filter only needs enough keys to say what you want: tag alone, field alone, or just a collection are all valid.

filter = {
    'collection': str,    # Optional: collection name
    'tag': str,           # Optional: restrict to one tag
    'field': str,         # Optional: restrict to one field; REQUIRED if exact/between is used
    'exact': [str, ...],  # Optional: exact value matches for 'field'
    'between': [float, float]  # Optional: numeric range for 'field' (min <= max)
}

Unknown keys raise a ValidationError rather than being silently ignored.

Filter Examples

# Filter by collection
{'collection': 'c61'}

# Filter by tag
{'tag': 'psa'}

# Filter by tag and field
{'tag': 'psa', 'field': 'value'}

# Filter by field alone (no tag) — matches that field across all tags
{'field': 'date'}

# Filter by exact value
{'tag': 'surgery', 'field': 'type', 'exact': ['prostatectomy']}

# Filter by numeric range
{'tag': 'psa', 'field': 'value', 'between': [0.2, 5.0]}

# Invalid: exact/between always needs a field to apply to
{'tag': 'surgery', 'exact': ['prostatectomy']}  # ValidationError

Filter Objects

The library also supports Filter objects using Pydantic validation:

from gazoo_research_utils.models.filter_model import Filter

filter_obj = Filter(
    collection='c61',
    tag='psa',
    field='value',
    between=[0.2, 5.0]
)

Anchors

Anchors define reference points for time-based analysis. They identify specific events from which other measurements are compared.

Single Anchor

anchor = [
    {'tag': 'surgery', 'field': 'date'}
]

Anchor Sequence

Multiple anchors can be specified in sequence. All anchors must occur within the specified time_delta:

anchor_sequence = [
    {'tag': 'surgery', 'field': 'date'},
    {'tag': 'radiation', 'field': 'date'}
]

Instance Selection

When multiple instances of an anchor sequence exist for a patient, use instance to specify which occurrence to use:

• instance = 0 - First occurrence of the sequence
• instance = 1 - Second occurrence
• instance = -1 - Last occurrence

Time Delta

The time_delta parameter sets the maximum number of days allowed between sequential anchors. This is useful for filtering out scenarios like salvage radiotherapy, which typically occurs months or years later rather than immediate adjuvant treatment.

from datetime import timedelta

# Using integer (days)
get_anchors(df, anchor_sequence, time_delta=90, instance=0)

# Using float
get_anchors(df, anchor_sequence, time_delta=90.5, instance=0)

# Using timedelta
get_anchors(df, anchor_sequence, time_delta=timedelta(days=90), instance=0)
get_anchors(df, anchor_sequence, time_delta=timedelta(days=5, hours=3), instance=0)

API Reference

Data Loading & Validation

Function	Description	Returns
load_data(file_path)	Load data from CSV with type conversion	pd.DataFrame
validate_gazoo_research_df(df)	Validate DataFrame structure	None (raises AssertionError on failure)

Filtering Functions

Function	Description	Returns
get_tags_where_filter(df, filter)	Get rows matching filter criteria	pd.DataFrame
filter(df, filters)	Filter patients by multiple criteria (AND logic)	pd.DataFrame
tag_instances(df, date_field=None)	One row per tag instance with its date (prefers date, then start-date, end-date)	pd.DataFrame

Anchor Functions

Function	Description	Returns
get_anchors(df, anchors, time_delta, instance)	Get anchor dates for patients	pd.DataFrame
filter_around_anchor(df, anchors_df, filters, search_range)	Filter data around anchor dates	pd.DataFrame
start_at_anchor(df, anchors)	Remove data before anchor dates	pd.DataFrame

Time Series & Visualization

Function	Description	Returns
get_tag_time_series(df, tag, anchors, drop_before_anchor=False)	Get time series for a tag with optional anchor normalization	pd.DataFrame
pivot(df, collapse=False)	Pivot data to wide format (human-readable)	pd.DataFrame
line_plot(df, field, x_scale, xlabel, ylabel)	Line plot for numerical data	matplotlib.figure.Figure
categorical_plot(df, field, y_scale, x_scale, xlabel, ylabel)	Plot for categorical data	matplotlib.figure.Figure

Statistical Analysis

Function	Description	Returns
describe_fields(df, anchors, fields, search_range)	Extract field values around anchors	list[pd.DataFrame]
summarize_fields(df, anchors, fields, search_range)	Tidy one-row-per-variable summary (n, mean, median, ...)	pd.DataFrame
plot_table1(describe_results)	Create Table 1 visualization	matplotlib.figure.Figure
kaplan_meier(df, anchors, event_tags, time_unit='months')	Kaplan-Meier survival analysis	pd.DataFrame
kaplan_meier_groups(df, anchors, event_tags, groups, ...)	Subgroup KM data with a log-rank test	(km, tests, figure)
plot_km_curves(km_df)	Plot Kaplan-Meier survival curves	matplotlib.figure.Figure
time_in_months(start_date, end_date)	Calculate whole months between dates	int

Patient-Level Transformations

Function	Description	Returns
calculate_age(df, anchors)	Calculate patient age at anchor date	pd.DataFrame
multilesion_transforation(df)	Transform to multi-target analysis	pd.DataFrame

Utility Functions

Function	Description	Returns
get_data_dictionary(df)	Get data dictionary	pd.DataFrame
get_parameters(df)	Get parameter options	list[dict]
patient_iterator(df, patient_fn)	Apply function to each patient	pd.DataFrame
tag_iterator(df, tag_fn)	Apply function to each tag	pd.DataFrame
link_by_field_value(df, tag_link, instance)	Link tags by field value	pd.DataFrame

Examples

Example 1: Finding Patients with Specific Criteria

from gazoo_research_utils import load_data, filter

# Load data
df = load_data("prostate-data.csv")

# Find patients with Gleason score 4 or 5 AND PSA between 0.2 and 0.5
filters = [
    {'collection': 'c61', 'tag': 'gleason-grade-group', 'field': 'value', 'exact': ['4', '5']},
    {'collection': 'c61', 'tag': 'psa', 'field': 'value', 'between': [0.2, 0.5]}
]
matched_patients = filter(df, filters, label="High-Risk Patients")

print(f"Found {len(matched_patients.id.unique())} patients")

Example 2: Time-Based Analysis with Anchors

from gazoo_research_utils import load_data, get_anchors, filter_around_anchor

# Load data
df = load_data("prostate-data.csv")

# Get patients who had surgery followed by radiation within 90 days
anchor_sequence = [
    {'tag': 'surgery', 'field': 'date'},
    {'tag': 'radiation', 'field': 'date'}
]

anchors_df = get_anchors(df, anchor_sequence, time_delta=90, instance=0)
print(f"Found {len(anchors_df)} patients with surgery -> radiation within 90 days")

# Get PSA values 30 days before to 7 days after surgery
psa_filter = [{'collection': 'c61', 'tag': 'psa', 'field': 'value'}]
psa_around_surgery = filter_around_anchor(
    df, 
    anchors_df, 
    psa_filter, 
    [-30.0, 7.0]
)

Example 3: Time Series Visualization

from gazoo_research_utils import load_data, get_anchors, get_tag_time_series, line_plot

# Load data
df = load_data("prostate-data.csv")

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery', 'field': 'date'}], time_delta=0, instance=0)

# Get time series for PSA normalized to surgery date
df_time_series = get_tag_time_series(df, tag='psa', anchors=anchors_df)

# Visualize with line plot
fig = line_plot(
    df_time_series,
    field='value',
    x_scale='year',  # or 'day'
    xlabel='Years Since Surgery',
    ylabel='PSA (ng/mL)',
    loc='upper right'
)

Example 4: Survival Analysis with Kaplan-Meier

from gazoo_research_utils import load_data, get_anchors, kaplan_meier, plot_km_curves

# Load data
df = load_data("prostate-data.csv")

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery', 'field': 'date'}], time_delta=0, instance=0)

# Define event tags (progression or death)
event_tags = [
    {'tag': 'progression', 'field': 'date'},
    {'tag': 'death', 'field': 'date'}
]

# Calculate Kaplan-Meier data (time in months)
km = kaplan_meier(df, anchors_df, event_tags)

# Plot survival curve
fig = plot_km_curves(km)

How censoring works

kaplan_meier follows these rules, which are worth knowing when interpreting survival times:

• Event - the latest date among the event_tags that falls strictly after the anchor. Using the latest (rather than the last row) matters when several events exist.
• Censoring - the latest date field at or after the anchor. Dates before the anchor (for example date of birth) are never used as follow-up.
• Excluded - patients with no anchor, or no date at or after the anchor, are dropped rather than given a negative or zero-length survival time.
• Units - time_unit="days" uses whole days; "months" keeps fractional days divided by 30.44.

Example 4b: Subgroup Survival Comparison

from gazoo_research_utils import load_data, get_anchors, kaplan_meier_groups

df = load_data("prostate-data.csv")
anchors_df = get_anchors(df, [{'tag': 'surgery', 'field': 'date'}], time_delta=90, instance=0)

# Each group is defined by filters matched around the anchor
groups = {
    'pT2': [{'tag': 'pT', 'field': 'T', 'exact': ['2a', '2b', '2c']}],
    'pT3': [{'tag': 'pT', 'field': 'T', 'exact': ['3a', '3b']}],
}

km, tests, fig = kaplan_meier_groups(
    df,
    anchors_df,
    [{'tag': 'biochemical-progression'}, {'tag': 'death'}],
    groups,
    search_range=[-90.0, 1.0],
    plot=True,
)

print(tests[['group', 'n', 'events', 'median_survival', 'logrank']])

tests reports n, events, median_survival per group and the p-value of the across-group log-rank test (NaN when fewer than two groups have data). Note that groups are not exclusive: a patient matching two groups appears in both.

Example 5: Patient Characteristics Summary

from gazoo_research_utils import (
    load_data, get_anchors, calculate_age, describe_fields, summarize_fields, plot_table1
)

# Load data
df = load_data("prostate-data.csv")

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery', 'field': 'date'}], time_delta=90, instance=0)
df = calculate_age(df, anchors_df)

# Extract field values within 90 days before to 1 day after surgery
fields = [
    {'collection': '0', 'tag': 'age', 'field': 'value'},
    {'collection': 'c61', 'tag': 'pT', 'field': 'T'},
    {'collection': 'c61', 'tag': 'gleason-grade-group', 'field': 'value'},
    {'collection': 'c61', 'tag': 'psa', 'field': 'value'}
]

# Per-field frames (what plot_table1 expects)
results = describe_fields(df, anchors_df, fields, [-90.0, 1.0])

# Or a single tidy table: one row per variable
summary = summarize_fields(df, anchors_df, fields, [-90.0, 1.0])
print(summary[['tag', 'field', 'data_type', 'n', 'mean', 'median', 'top_value']])

# Plot results as Table 1
fig_table = plot_table1([results])

Example 6: Pivot Data to Wide Format

from gazoo_research_utils import load_data, get_tags_where_filter, pivot

# Load data
df = load_data("prostate-data.csv")

# Get PSA data
psa_df = get_tags_where_filter(df, {'tag': 'psa'})

# Pivot to human-readable format
psa_pivoted = pivot(psa_df)

# Access PSA values
print(psa_pivoted["c61:psa:value"])

By default pivot returns one row per tag instance, so a patient with 87 tag instances occupies 87 rows and each row holds a single value. That layout is correct for repeated measurements such as a series of PSA results.

For one-per-patient attributes (demographics, stage, dates), pass collapse=True to get one row per patient:

baseline = pivot(df, collapse=True)   # index is unique, one row per patient
baseline.loc["MRN001", "0:dob:date"]

Example 7: Multi-Lesion Analysis

from gazoo_research_utils import load_data, multilesion_transforation, get_anchors, kaplan_meier

# Load data
df = load_data("multi-lesion-data.csv")

# Transform to multi-lesion analysis
df_lesion = multilesion_transforation(df)

# Now analyze at lesion level
# Get anchors for each lesion
anchors_df = get_anchors(df_lesion, [{'tag': 'surgery'}], time_delta=0, instance=0)

# Perform survival analysis
event_tags = [{'tag': 'local-progression'}]
km = kaplan_meier(df_lesion, anchors_df, event_tags)

Example 8: Using Patient Iterator

from gazoo_research_utils import load_data, patient_iterator

# Load data
df = load_data("prostate-data.csv")

# Define function to apply to each patient
def calculate_max_psa(patient_df):
    psa_values = patient_df[
        (patient_df['tag'] == 'psa') & 
        (patient_df['field'] == 'value') &
        (patient_df['data_type'] == 'number')
    ]['value'].astype(float)
    
    return pd.DataFrame([{
        'id': patient_df['id'].iloc[0],
        'max_psa': psa_values.max() if len(psa_values) > 0 else None
    }])

# Apply to all patients
results = patient_iterator(df, calculate_max_psa)
print(results)

Example 9: Calculate Age at Anchor

from gazoo_research_utils import load_data, get_anchors, calculate_age

# Load data
df = load_data("prostate-data.csv")

# Get anchors
anchors_df = get_anchors(df, [{'tag': 'surgery', 'field': 'date'}], time_delta=0, instance=0)

# Calculate age
df_with_age = calculate_age(df, anchors_df)

# Filter for patients over 65 at surgery
age_rows = df_with_age[
    (df_with_age['tag'] == 'age') &
    (df_with_age['field'] == 'value') &
    (df_with_age['data_type'] == 'number')
].copy()
age_rows['value'] = age_rows['value'].astype(float)
over_65 = age_rows[age_rows['value'] > 65]

Example 10: Complex Analysis Pipeline

from gazoo_research_utils import (
    load_data, 
    get_anchors, 
    filter_around_anchor, 
    get_tag_time_series,
    line_plot,
    describe_fields,
    plot_table1,
    kaplan_meier,
    plot_km_curves,
    start_at_anchor
)
from datetime import timedelta

# Load data
df = load_data("prostate-data.csv")

# 1. Get patients who had surgery followed by radiation within 90 days
anchor_sequence = [
    {'tag': 'surgery', 'field': 'date'},
    {'tag': 'radiation', 'field': 'date'}
]
anchors_df = get_anchors(df, anchor_sequence, time_delta=90, instance=0)

# 2. Remove data before surgery for remaining analysis
df_after_surgery = start_at_anchor(df, anchors_df)

# 3. Get PSA values 30 days before to 7 days after surgery
filters = [{'collection': 'c61', 'tag': 'psa', 'field': 'value'}]
psa_around_surgery = filter_around_anchor(df, anchors_df, filters, [-30.0, 7.0])

# 4. Get time series for PSA normalized to surgery date
df_time_series = get_tag_time_series(df, tag='psa', anchors=anchors_df)

# 5. Visualize PSA time series
fig_psa = line_plot(
    df_time_series,
    field='value',
    x_scale='year',
    xlabel='Years Since Surgery',
    ylabel='PSA (ng/mL)'
)

# 6. Describe patient characteristics at surgery
fields = [
    {'tag': 'age', 'field': 'value'},
    {'tag': 'pT', 'field': 'T'},
    {'tag': 'gleason-grade-group', 'field': 'value'}
]
results = describe_fields(df, anchors_df, fields, [-1.0, 1.0])
fig_table = plot_table1([results])

# 7. Kaplan-Meier survival analysis
event_tags = [{'tag': 'progression', 'field': 'date'}]
km = kaplan_meier(df, anchors_df, event_tags, time_unit="months")
fig_km = plot_km_curves(km)

Development

Setting Up the Development Environment

# Clone the repository
cd gazoo-research-utils

# Install dependencies using uv
uv sync --extra dev

# Activate the virtual environment
source .venv/bin/activate

Running Tests

# Run all tests
pytest

# Run tests with coverage
pytest --cov=gazoo_research_utils --cov-report=html

# Run specific test file
pytest tests/test_Analysis_filter.py

Building and Publishing

# Build the package
uv build

# Publish to PyPI (requires PyPI token)
uv run twine upload dist/*

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Run tests (pytest)
5. Commit your changes (git commit -m 'Add some amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Development Guidelines

• Use type hints for all functions
• Add docstrings following Google Python Style Guide
• Write unit tests for new functionality
• Run tests before committing
• Follow existing code style

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• GazooResearch for providing the data platform
• All contributors and users of this library

Support

For support, please open an issue in the GitHub repository or contact the maintainers.

---

Built with ❤️ by the Gazoo Research Team