Skip to main content

ENCODE dataframe

I wanted a better way of exploring and downloading raw data from the ENCODE project.

For example, I’d like to get the BAM files for all ChIP-seq experiments done in uninduced MEL cells (from the mm9 assembly).

One strategy would be to individually go through each track hub (e.g., histone mods from LICR, http://genome.cit.nih.gov/cgi-bin/hgFileUi?db=mm9&g=wgEncodeLicrHistone), filter data, and download files individually.

Another strategy would be to go directly to the download page (http://hgdownload.cse.ucsc.edu/goldenPath/mm9/encodeDCC/wgEncodeLicrHistone/) and extract the files that end in .bam.

This small package takes advantage of the files.txt files (here’s an example) that describe all the metadata on the download page.

The files.txt files are downloaded from each ENCODE track hub in the assembly of interest. Then these files are parsed and concatenated together into one big pandas.DataFrame that can be used to find the data you care about.

Installation

pip install encode-dataframe

Usage

Mirror the files. This may take a minute or so. If you’ve cloned the git repo, you already have a copy of the mm9 files.

>>> import encode_dataframe as edf
>>> edf.mirror_metadata_files('mm9')

Create a large DataFrame:

>>> df = edf.encode_dataframe('mm9')
>>> len(df)
5865

Armed with the dataframe, we can now slice and dice to get the data we care about. Eventually I’d like to run a ChromHMM segmentation on MEL cells, but I need to get the data first …

Choose a cell type

>>> interesting = df.cell == 'MEL'

And only BAM files

>>> interesting &= df.type == 'bam'

And only ChIP- or DNase-seq

>>> interesting &= df.dataType.isin(['ChipSeq', 'DnaseSeq'])

And only untreated (in this case, uninduced) cells:

>>> interesting &= df.treatment != 'DMSO_2.0pct'

And only one replicate (some have 2 or 3)

>>> interesting &= df.replicate == '1'

And only those that don’t have some issue with them (looks like older versions have some text in the objStatus field):

>>> interesting &= df.objStatus.isnull()

How many do we have to work with?

>>> m = df[interesting]
>>> len(m)
60

Some of these are controls (input or IgG), and there are some duplicates (looks like H3K4me3 ChIP-seq uses 2 different controls; CTCF was done by different groups). How many unique antibodies?

>>> len(m.antibody.unique())
46

So here are the files I should download:

>>> urls = m.url.values

Metadata

Release files for encode-dataframe 0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for encode-dataframe 0.1
File Size Uploaded
encode-dataframe-0.1.tar.gz 3.3 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for encode-dataframe 0.1
File Interpreter ABI Platform
encode_dataframe-0.1-py2.py3-none-any.whl Python 3, Python 2 none any Details

Total release size: 8.7 kB

Release files / encode-dataframe-0.1.tar.gz

Download URL encode-dataframe-0.1.tar.gz
Size 3.3 kB
Tags Source
SHA-256 checksum
How to use checksums
5350067688e83d3f093cb3181ab087de720aa43d6086bec3cbedfc0bdc987228
BLAKE2b-256 checksum
How to use checksums
d1b4e71596bcddc05ae1cf1dfe6dedc9f15dcb87b08f8a4e7111cdc4497590d3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / encode_dataframe-0.1-py2.py3-none-any.whl

Download URL encode_dataframe-0.1-py2.py3-none-any.whl
Size 5.5 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
35531d104c716c6f42f1569dfb2dbc0b79924802a19fe03de36a573d150238aa
BLAKE2b-256 checksum
How to use checksums
cc12ba0fd98f1843543812bb49279a1209bb4be9879db6026dd72829ca4d0cbf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

0.1 This release

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page