flatiron-cleaner

flatiron-cleaner is a Python package that cleans Flatiron Health cancer datasets into analysis-ready formats, specifically designed with predictive modeling and survival analysis in mind. By automating complex and tedious data processing workflows, it helps researchers extract meaningful insights and ensure reproducible results while reducing preparation time.

Key features of the package include:

• Providing a modular architecture that allows researchers to select which Flatiron files to process
• Converting long-format dataframes into wide-format dataframes with unique PatientIDs per row
• Ensuring appropriate data types for predictive modeling and statistical analysis
• Standardizing data cleaning around a user-specified index date, such as metastatic diagnosis or treatment initiation
• Engineering clinically relevant variables for analysis

Note: This package is under active development and serves as a starting framework for cleaning Flatiron Health datasets. Flatiron data structures may vary across different datacuts, including changes in column names, coding conventions, or unexpected values. While the package attempts to handle common variations, users should review the processing logic and adapt it as needed for their specific datacut and research question.

Installation

Built and tested in python 3.13.

pip install flatiron-cleaner 

Available Processors

Cancer-Specific Processors

The following cancers have their own dedicated data processor class:

General Processor

For cancer types without a dedicated processor, DataProcessorGeneral is available with standard methods.

Processing Methods

Standard Methods

The following methods are available across all processor classes, including the general processor:

Cancer-Specific Methods

Cancer-specific classes contain additional methods (e.g., process_enhanced() and process_biomarkers()). For a complete list of available methods for each cancer type, refer to the source code or use Python's built-in help functionality:

from flatiron_cleaner import DataProcessorUrothelial

Usage Example

from flatiron_cleaner import DataProcessorUrothelial
from flatiron_cleaner import merge_dataframes

# Initialize class
processor = DataProcessorUrothelial()

# Import dataframe with PatientIDs and index date of interest
df = pd.read_csv('path/to/your/data')

# Load and clean data
cleaned_ecog_df = processor.process_ecog('path/to/your/ECOG.csv',
                                         index_date_df=df,
                                         index_date_column='AdvancedDiagnosisDate',
                                         days_before=30,
                                         days_after=0)                  

cleaned_medication_df = processor.process_medications('path/to/your/MedicationAdmninistration.csv',
                                                      index_date_df=df,
                                                      index_date_column='AdvancedDiagnosisDate',
                                                      days_before=180,
                                                      days_after=0)

# Merge dataframes 
merged_data = merge_dataframes(cleaned_ecog_df, cleaned_medication_df)

For a more detailed usage demonstration, see the notebook titled "tutorial" in the example/ directory.

Contact

Contributions and feedback are welcome. Contact: xavierorcutt@gmail.com