phenopipe 0.8.0

No project description provided

Phenopipe

Python Functions for Phenotyping and Analysis

Phenopipe is a Python library to automate phenotyping and downstream analysis. Its main development target is All of Us research platform. Phenopipe is heavily inspired by and borrowed query definitions from https://github.com/annisjs/aou.phenotyper2 and https://github.com/annisjs/aou.reader

Tasks

The basic building block in phenopipe library is a task which is a class from tasks module. A task represents a single step in phenotyping and analysis. Its inputs are polars dataframes and it has a single polars dataframe output. Therefore tasks' outputs can be passed to other tasks. This can be automated using input_tasks attribute in task class. Each task in input_tasks will be completed prior to task completion and those outputs will be added to inputs. Each task need to have a complete method which contains the logic of the data retrieval and/or analysis step and completion decorator automates common operations prior and after task is completed.

Data Queries

A common task type is getting the data from the database. These tasks are collected under get_data module in tasks module. These tasks contains additional attributes such as caching and lazy dataframe evaulation. Data queries are planned by tasks but it is run inside query connection objects which are provided in query_connections module. The goal is to allow running the same data tasks on different platform by simply changing the query connection in enviroment variables. This is partially achieved for other platforms designed around OMOP Data Model like AOU, however AOU specific data structures does not fully allow this. To track if a task is compatible (or tested) with a platform state attribute is used. This attribute holds a key value pair where keys are shared with query platform attribute of query connections and values indicate if task is compatible. Value can be one of the followings:

incompatible: Indicates the task is known to be incompatible with the platform. parsed: Indicates the task is parsed from another library and it is not yet tested. untested: Indicates the task in not tested on this platform. unverified: Indicates the resulting data is not yet verified. tested: Indicates the task is tested on this platform verified: Indicates the task is verified on this platform which means the resulting data is used in an analysis and didn't show any inconsistencies.

Environment Variables

env_vars attribute in task object holds variables that is shared between different tasks. It is also used to share common variables between tasks in a analysis. An example variable is query connection stored in env_vars to delegate communication with database.

Plan

A Pipe object holds a phenotyping and analysis plan as a dictionary of tasks and a env_vars attribute. Pipe object has a run method which will complete each task and merge each result on its anchor. Only tasks outputs without any anchor is saved in outputs dictionary.

Inputs

A task can accept other tasks or their outputs as inputs. Each task may have a minimal input schema which describes the minimal column names and data types in order to task to run succesfully. Similarly every task has a minimal output schema which describes the minimal column names and data types in the output dataframe so any task can determine if it accepts the task as input. All input schemas and output schema are validated during task completion.

Anchor Input

Anchor keyword in inputs dictionary is reserved for a data frame defining a selection criteria of the output. This can be described using anchor_date, anchor_range, anchor_pid attributes in the task object. Anchor range list of two literal which can be column names in anchor input dataframe or integers determining the time window for selection around anchor_date_col column in anchor dataframe. Anchor pid is the name of the column of person ids in the anchor dataframe to be used during subsetting.

Data Aggregate

Any task can indicate an aggregate function that will be run after or alongside the anchoring. This can be first, last, closest:nearest, closest:forward, closest:backward. For closest aggregate an anchor needs to be given. Ties are broken randomly but consistently.

Templating

Phenopipe provides a templating structure to define a plan using yaml files (or strings or dictionaries in the same format). The function plan_from_yaml will accept the file name for a yaml. The plan obtained using example below will collect initial hypertension diagnosis where there is a heart failure hospitalization in one year window before or after and return with the first heart failure hospitalization date in that window. Each task is given as a absolute import such as phenopipe.tasks.get_data.hospitalization.FirstHfHospitalizationData or commonly used modules can be described using modules keyword and relative import can be given such as modules.phenotype.HypertensionPt for convenience. Query connection will be translated as the camelcase class of the underscored name given in the template. All parameters under the task id will be passed into task init method. The inputs of a task can be other tasks in the plan given by using the identifier.

env_vars:
  query_conn:
    module: BigQueryConnection
    params:
      cache: false
      cache_loc: __phenopipe
lazy: false
modules:
  activity: phenopipe.tasks.get_data.activity 
  preprocess: phenopipe.tasks.preprocess
tasks:
  fitbit:
    name: modules.activity.GetFitbit
  sleep:
    name: modules.activity.GetSleep
  daily_hr:
    name: modules.activity.GetDailyHr
  wear_time:
    name: modules.activity.GetWearTime
  demographics:
    name: phenopipe.tasks.get_data.person_info.GetDemographics
  heart_failure:
    name: phenopipe.tasks.get_data.phenotype.HeartFailurePt
  last_medical_encounter:
    name: phenopipe.tasks.get_data.GetMedicalEncounter
    select: last        
  clean_fitbit_w_ehr:
    name: modules.preprocess.CleanFitbitWithEhr
    inputs:
      fitbit: fitbit
      wear_time: wear_time
      demographics: demographics
      last_medical_encounter: last_medical_encounter
  clean_sleep_w_ehr:
    name: modules.preprocess.CleanSleepWithEhr
    inputs:
      sleep: sleep
      demographics: demographics
      last_medical_encounter: last_medical_encounter