methylprep 1.2.1

Python-based Illumina methylation array preprocessing software

methylprep is a python package for processing Illumina methylation array data. View on ReadTheDocs.

Methylprep package

methylprep is part of a methyl-suite of python packages that provide functions to process and analyze DNA methylation data from Illumina arrays (27, 450k, and EPIC supported). The methylprep package contains functions to processing raw data files from arrays or downloading (and processing) public data sets from GEO (the NIH Gene Expression Omnibus is a database repository) or from ArrayExpress. It contains both high-level APIs for processing data from local files and low-level functionality allowing you to customize the flow of data and how it is processed.

Related packages

You should install all three components, as they work together.

• methylcheck includes quality control (QC) functions for filtering out unreliable probes, based on the published literature and outlier detection. It also includes several data visualization functions based on seaborn and matplotlib graphic libraries.
• methylize provides analysis functions
  • differentially methylated probe statistics (between treatment and control samples)
  • volcano plots (which probes are the most different)
  • manhattan plot (where in genome are the differences)

Installation

methylprep maintains configuration files for your Python package manager of choice: conda, pipenv, and pip.

pip install methylprep

---

Command line data processing

The most common use case is processing .idat files on a computer within a command line interface. This can also be done in a Jupyter notebook, but large data sets take hours to run and Jupyter will take longer to run these than command line.

process

python -m methylprep -v process -d <filepath>

Argument	Type	Default	Description
data_dir	str, Path	REQUIRED	Base directory of the sample sheet and associated IDAT files
array_type	str	None	Code of the array type being processed. Possible values are custom, 27k, 450k, epic, and epic+. If not provided, the pacakage will attempt to determine the array type based on the number of probes in the raw data. If the batch contains samples from different array types, this may not work. Our data download function attempts to split different arrays into separate batches for processing to accommodate this.
manifest_filepath	str, Path	None	File path for the array's manifest file. If not provided, this file will be downloaded from a Life Epigenetics archive.
no_sample_sheet	bool	None	pass in "--no_sample_sheet" from command line to trigger sample sheet auto-generation. Sample names will be based on idat filenames. Useful for public GEO data sets that lack sample sheets.
sample_sheet_filepath	str, Path	None	File path of the project's sample sheet. If not provided, the package will try to find one based on the supplied data directory path.
sample_name	str to list	None	List of sample names to process, in the CLI format of -n sample1 sample2 sample3 etc. If provided, only those samples specified will be processed. Otherwise all samples found in the sample sheet will be processed.
export	bool	False	Add flag to export the processed data to CSV.
betas	bool	False	Add flag to output a pickled dataframe of beta values of sample probe values.
m_value	bool	False	Add flag to output a pickled dataframe of m_values of samples probe values.
batch_size	int	None	Optional: splits the batch into smaller sized sets for processing. Useful when processing hundreds of samples that can't fit into memory. Produces multiple output files. This is also used by the package to process batches that come from different array types.

run_pipeline (within a python interpreter, such as IDLE or Jupyter)

Run the complete methylation processing pipeline for the given project directory, optionally exporting the results to file.

Returns: A collection of DataContainer objects for each processed sample

from methylprep import run_pipeline

data_containers = run_pipeline(data_dir, array_type=None, export=False, manifest_filepath=None, sample_sheet_filepath=None, sample_names=None)

Note: All the same input parameters from command line apply to run_pipeline. Type dir(methylprep.run_pipeline) in an interactive python session to see details.

Note: By default, if run_pipeline is called as a function in a script, a list of SampleDataContainer objects is returned.

getting help from command line

methylprep provides a command line interface (CLI) so the package can be used directly in bash/batchfile or windows/cmd scripts as part of building your custom processing pipeline.

All invocations of the methylprep CLI will provide contextual help, supplying the possible arguments and/or options available based on the invoked command. If you specify verbose logging the package will emit log output of DEBUG levels and above.

python -m methylprep

usage: methylprep [-h] [-v] {process,sample_sheet} ...

Utility to process methylation data from Illumina IDAT files

positional arguments:
  {process,sample_sheet}
    process             process help
    sample_sheet        sample sheet help

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Enable verbose logging

---

Commands

The methylprep cli provides two top-level commands:

• process to process methylation data
• download script to download and process public data sets in NIH GEO or ArrayExpress collections. Provide the public Accession ID and it will handle the rest.
• sample_sheet to find/read/validate a sample sheet and output its contents

process

Process the methylation data for a group of samples listed in a single sample sheet.

If you do not provide the file path for the project's sample_sheet the module will try to find one based on the supplied data directory path. You must supply either the name of the array being processed or the file path for the array's manifest file. If you only specify the array type, the array's manifest file will be downloaded from a Life Epigenetics archive.

>>> python -m methylprep process

usage: methylprep idat [-h] -d DATA_DIR [-a {custom,27k,450k,epic,epic+}]
                       [-m MANIFEST] [-s SAMPLE_SHEET] [--no_sample_sheet]
                       [-n [SAMPLE_NAME [SAMPLE_NAME ...]]] [-e] [-b]
                       [--m_value] [--batch_size BATCH_SIZE]

Process Illumina IDAT files

optional arguments:
  -h, --help            show this help message and exit
  -d DATA_DIR, --data_dir DATA_DIR
                        Base directory of the sample sheet and associated IDAT
                        files. If IDAT files are in nested directories, this
                        will discover them.
  -a {custom,27k,450k,epic,epic+}, --array_type {custom,27k,450k,epic,epic+}
                        Type of array being processed. If omitted, this will
                        autodetect it.
  -m MANIFEST, --manifest MANIFEST
                        File path of the array manifest file. If omitted, this
                        will download the appropriate file from `s3`.
  -s SAMPLE_SHEET, --sample_sheet SAMPLE_SHEET
                        File path of the sample sheet. If omitted, this will
                        discover it. There must be only one CSV file in the
                        data_dir for discovery to work.
  --no_sample_sheet     If your dataset lacks a sample sheet csv file, specify
                        --no_sample_sheet to have it create one on the fly.
                        This will read .idat file names and ensure processing
                        works. If there is a matrix file, it will add in
                        sample names too.
  -n [SAMPLE_NAME [SAMPLE_NAME ...]], --sample_name [SAMPLE_NAME [SAMPLE_NAME ...]]
                        Sample(s) to process. You can pass multiple sample
                        names with multiple -n params.
  -e, --no_export       Default is to export data to csv in same folder where
                        IDAT file resides. Pass in --no_export to suppress
                        this.
  -b, --betas           If passed, output returns a dataframe of beta values
                        for samples x probes. Local file beta_values.npy is
                        also created.
  --m_value             If passed, output returns a dataframe of M-values for
                        samples x probes. Local file m_values.npy is also
                        created.
  --batch_size BATCH_SIZE
                        If specified, samples will be processed and saved in
                        batches no greater than the specified batch size

download

There are thousands of publically accessible DNA methylation data sets available via the GEO (US NCBI NIH) https://www.ncbi.nlm.nih.gov/geo/ and ArrayExpress (UK) https://www.ebi.ac.uk/arrayexpress/ websites. This function makes it easy to import them and build a reference library of methylation data.

Argument	Type	Default	Description
-d , --data_dir	str	[required path]	path to where the data series will be saved. Folder must exist already.
-i ID, --id ID	str	[required ID]	The dataset's reference ID (Starts with GSM for GEO or E-MTAB- for ArrayExpress)
-l LIST, --list LIST	multiple strings	optional	List of series IDs (can be either GEO or ArrayExpress), for partial downloading
-o, --dict_only	True	pass flag only	If passed, will only create dictionaries and not process any samples
-b BATCH_SIZE, --batch_size BATCH_SIZE	int	optional	Number of samples to process at a time, 100 by default. Set to 0 for processing everything as one batch. Regardless of this number, the resulting file structure will be the same. But most machines cannot process more than 200 samples in memory at once, so this helps the user set the memory limits for their machine.

sample_sheet

Find and parse the sample sheet in a given directory and emit the details of each sample. This is not required for actually processing data.

>>> python -m methylprep sample_sheet

usage: methylprep sample_sheet [-h] -d DATA_DIR

Process Illumina sample sheet file

optional arguments:
  -h, --help            show this help message and exit
  -d, --data_dir        Base directory of the sample sheet and associated IDAT
                        files
  -c, --create          If specified, this creates a sample sheet from idats
                        instead of parsing an existing sample sheet. The
                        output file will be called "samplesheet.csv".
  -o OUTPUT_FILE, --output_file OUTPUT_FILE
                        If creating a sample sheet, you can provide an
                        optional output filename (CSV).                        

example of creating a sample sheet

~/methylprep$ python -m methylprep -v sample_sheet -d ~/GSE133062/GSE133062 --create
INFO:methylprep.files.sample_sheets:[!] Created sample sheet: ~/GSE133062/GSE133062/samplesheet.csv with 70 GSM_IDs
INFO:methylprep.files.sample_sheets:Searching for sample_sheet in ~/GSE133062/GSE133062
INFO:methylprep.files.sample_sheets:Found sample sheet file: ~/GSE133062/GSE133062/samplesheet.csv
INFO:methylprep.files.sample_sheets:Parsing sample_sheet
200861170112_R01C01
200882160083_R03C01
200861170067_R02C01
200498360027_R04C01
200498360027_R08C01
200861170067_R01C01
200861170072_R05C01
200498360027_R06C01
200861170072_R01C01
200861170067_R03C01
200882160070_R02C01
...

download

The CLI now includes a download option. Supply the GEO ID or ArrayExpress ID and it will locate the files, download the idats, process them, and build a dataframe of the associated meta data. This dataframe format should be compatible with methylcheck and methylize.

optional arguments:

Argument	Type	Description
-h, --help		show this help message and exit
-d DATA_DIR, --data_dir DATA_DIR	path (required)	Directory to download series to
-i ID, --id ID	string	Unique ID of the series (either GEO or ArrayExpress ID)
-l LIST, --list LIST	multiple string arguments	List of series IDs (can be either GEO or ArrayExpress)
-o, --dict_only	no args	If passed, will only create dictionaries and not process any samples
-b BATCH_SIZE, --batch_size BATCH_SIZE	number	Number of samples to process at a time, 100 by default

• When processing large batches of raw .idat files, specify --batch_size to break the processing up into smaller batches so the computer's memory won't overload. This is off by default when using process but is ON when using download and set to batch_size of 100.

---

Low-level processing functions

These are some functions that are used within the major methylprep functions listed above. run_pipeline calls them for you as needed.

get_sample_sheet

Find and parse the sample sheet for the provided project directory path.

Returns: A SampleSheet object containing the parsed sample information from the project's sample sheet file

from methylprep import get_sample_sheet

sample_sheet = get_sample_sheet(dir_path, filepath=None)

Argument	Type	Default	Description
data_dir	str, Path	-	Base directory of the sample sheet and associated IDAT files
sample_sheet_filepath	str, Path	None	File path of the project's sample sheet. If not provided, the package will try to find one based on the supplied data directory path.

get_manifest

Find and parse the manifest file for the processed array type.

Returns: A Manifest object containing the parsed probe information for the processed array type

from methylprep import get_manifest

manifest = get_manifest(raw_datasets, array_type=None, manifest_filepath=None)

Argument	Type	Default	Description
raw_datasets	RawDataset collection	-	Collection of RawDataset objects containing probe information from the raw IDAT files.
array_type	str	None	Code of the array type being processed. Possible values are custom, 450k, epic, and epic+. If not provided, the pacakage will attempt to determine the array type based on the provided RawDataset objects.
manifest_filepath	str, Path	None	File path for the array's manifest file. If not provided, this file will be downloaded from a Life Epigenetics archive.

get_raw_datasets

Find and parse the IDAT files for samples within a project's sample sheet.

Returns: A collection of RawDataset objects for each sample's IDAT file pair.

from methylprep import get_raw_datasets

raw_datasets = get_raw_datasets(sample_sheet, sample_names=None)

Argument	Type	Default	Description
sample_sheet	SampleSheet	-	A SampleSheet instance from a valid project sample sheet file.
sample_names	str collection	None	List of sample names to process. If provided, only those samples specified will be processed. Otherwise all samples found in the sample sheet will be processed.