genomepy 0.8.3

Automatic downloading and processing of genomes and metadata in command line and Python

genomepy

Easily install and use genomes in Python and elsewhere!

The goal is to have a simple and straightforward way to download and use genomic sequences. Currently, genomepy supports UCSC, Ensembl and NCBI.

Pssst, hey there! Is genomepy not doing what you want? Does it fail? Is it clunky? Is the documentation unclear? Have any other ideas on how to improve it? Don't be shy and let us know!

Table of Contents

1. Installation
2. Quick usage
3. Plugins and indexing
4. Configuration
5. Usage
   • Command line
   • Python
6. Known Issues
7. Citation
8. Getting help
9. Contributing
10. License

Installation

Genomepy works with Python 3.6+. You can install it via bioconda:

$ conda config --set use_only_tar_bz2 True
$ conda install genomepy

Or via pip:

$ pip install genomepy

With Pip installation, you will have to install some dependencies. Make sure these dependencies are in your PATH.

To read/write bgzipped genomes you will have to install tabix.

If you want to use the annotation download feature, you will have to install the following utilities:

• genePredToBed
• genePredToGtf
• bedToGenePred
• gtfToGenePred
• gff3ToGenePred

You can find the binaries here.

Quick usage

1. Find your genome: $ genomepy search zebrafish

Console output:

name      provider    accession          species        tax_id    other_info                 
GRCz11    Ensembl     GCA_000002035.4    Danio rerio    7955      2017-08-Ensembl/2018-04    
 ^
 Use name for genomepy install

2. Install your genome (with annotation): $ genomepy install --annotation GRCz11 ensembl

Default genome directory: ~/.local/share/genomes/

Plugins and indexing

By default genomepy generates a file with chromosome sizes and a BED file with gap locations (Ns in the sequence).

For some genomes genomepy can download blacklist files (generated by the Kundaje lab). This will only work when installing these genomes from UCSC. Enable this plugin to use it.

$ genomepy plugin enable blacklist

You can also create indices for some widely using aligners. Currently, genomepy supports:

• bowtie2
• bwa
• gmap
• hisat2
• minimap2
• star

Note 1: these programs are not installed by genomepy and need to be installed separately for the indexing to work.

Note 2: splice-aware indexing is performed by Hisat2 and STAR. Splice-aware indexing requires the annotation to be downloaded as well. You will recieve a warning if indexing is performed without annotation for these aligners.

Note 3: STAR can further improve mapping to (novel) splice junctions by indexing again (2-pass mapping mode). The second pass is currently not supported by genomepy.

You can configure the index creation using the genomepy plugin command (see below)

Configuration

To change the default configuration, generate a personal config file:

$ genomepy config generate
Created config file /home/simon/.config/genomepy/genomepy.yaml

Genome location

By default genomes will be saved in ~/.local/share/genomes.

To set the default genome directory to /data/genomes for instance, edit ~/.config/genomepy/genomepy.yaml and change the following line:

genomes_dir: ~/.local/share/genomes/

to:

genomes_dir: /data/genomes

The genome directory can also be explicitly specified in both the Python API as well as on the command-line.

Compression

Optionally genome FASTA files can be saved using bgzip compression. This means that the FASTA files will take up less space on disk. To enable this use the flag --bgzip on the command line, or add the following line to your config file:

bgzip: True

Most tools are able to use bgzip-compressed genome files. One notable exception is bedtools getfasta. As an alternative, you can use the faidx command-line script from pyfaidx which comes installed with genomepy.

Usage

Command line

Usage: genomepy [OPTIONS] COMMAND [ARGS]...

Options:
  --version   Show the version and exit.
  -h, --help  Show this message and exit.

Commands:
  config     manage configuration
  genomes    list available genomes
  install    install genome
  plugin     manage plugins
  providers  list available providers
  search     search for genomes

Install a genome.

Find the name of your desired genome:

$ genomepy search xenopus_tropicalis
name                       provider    accession          species               tax_id    other_info
Xenopus_tropicalis_v9.1    Ensembl     GCA_000004195.3    Xenopus tropicalis    8364      2019-04-Ensembl/2019-12
xenTro1                    UCSC        na                 Xenopus tropicalis    8364      Oct. 2004 (JGI 3.0/xenTro1)
xenTro2                    UCSC        na                 Xenopus tropicalis    8364      Aug. 2005 (JGI 4.1/xenTro2)
xenTro3                    UCSC        GCA_000004195.1    Xenopus tropicalis    8364      Nov. 2009 (JGI 4.2/xenTro3)
xenTro7                    UCSC        GCA_000004195.2    Xenopus tropicalis    8364      Sep. 2012 (JGI 7.0/xenTro7)
xenTro9                    UCSC        GCA_000004195.3    Xenopus tropicalis    8364      Jul. 2016 (Xenopus_tropicalis_v9.1/xenTro9)
v4.2                       NCBI        GCA_000004195.1    Xenopus tropicalis    8364      DOE Joint Genome Institute
Xtropicalis_v7             NCBI        GCA_000004195.2    Xenopus tropicalis    8364      DOE Joint Genome Institute
Xenopus_tropicalis_v9.1    NCBI        GCA_000004195.3    Xenopus tropicalis    8364      DOE Joint Genome Institute
UCB_Xtro_10.0              NCBI        GCA_000004195.4    Xenopus tropicalis    8364      University of California, Berkeley
 ^
 Use name for genomepy install

Note that genomes with a space can be searched for either by using "quotation marks", or by replacing the space(s) with and underscore _. For example, we can search for Xenopus tropicalis as "Xenopus Tropicalis", xenopus_tropicalis or xenopus. The search function is case-insensitive. You can also search by taxonomy ID. For instance, to search for Xenopus tropicalis:

$ genomepy search 8364
name                       provider    accession          species               tax_id    other_info
Xenopus_tropicalis_v9.1    Ensembl     GCA_000004195.3    Xenopus tropicalis    8364      2019-04-Ensembl/2019-12
xenTro1                    UCSC        na                 Xenopus tropicalis    8364      Oct. 2004 (JGI 3.0/xenTro1)
xenTro2                    UCSC        na                 Xenopus tropicalis    8364      Aug. 2005 (JGI 4.1/xenTro2)
xenTro3                    UCSC        GCA_000004195.1    Xenopus tropicalis    8364      Nov. 2009 (JGI 4.2/xenTro3)
xenTro7                    UCSC        GCA_000004195.2    Xenopus tropicalis    8364      Sep. 2012 (JGI 7.0/xenTro7)
xenTro9                    UCSC        GCA_000004195.3    Xenopus tropicalis    8364      Jul. 2016 (Xenopus_tropicalis_v9.1/xenTro9)
v4.2                       NCBI        GCA_000004195.1    Xenopus tropicalis    8364      DOE Joint Genome Institute
Xtropicalis_v7             NCBI        GCA_000004195.2    Xenopus tropicalis    8364      DOE Joint Genome Institute
Xenopus_tropicalis_v9.1    NCBI        GCA_000004195.3    Xenopus tropicalis    8364      DOE Joint Genome Institute
UCB_Xtro_10.0              NCBI        GCA_000004195.4    Xenopus tropicalis    8364      University of California, Berkeley
 ^
 Use name for genomepy install

Lets say we want to download the Xenopus tropicalis genome from UCSC. Copy the name returned by the search function and it with the provider name to install:

$ genomepy install xenTro9 UCSC
Downloading genome from http://hgdownload.soe.ucsc.edu/goldenPath/xenTro9/bigZips/xenTro9.fa.gz...
Genome download successful, starting post processing...

name: xenTro9
local name: xenTro9
fasta: /data/genomes/xenTro9/xenTro9.fa

Here, genomes are downloaded to the directory specified in the config file. To choose a different directory, use the -g option.

$ genomepy install sacCer3 UCSC -g /path/to/my/genomes
Downloading genome from http://hgdownload.soe.ucsc.edu/goldenPath/sacCer3/bigZips/chromFa.tar.gz...
Genome download successful, starting post processing...

name: sacCer3
local name: sacCer3
fasta: /path/to/my/genomes/sacCer3/sacCer3.fa

You can use a regular expression to filter for matching sequences (or non-matching sequences by using the --no-match option). For instance, the following command downloads hg38 and saves only the major chromosomes:

$ genomepy install hg38 UCSC -r 'chr[0-9XY]+$'
downloading from http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz...
done...
name: hg38
local name: hg38
fasta: /data/genomes/hg38/hg38.fa
$ grep ">" /data/genomes/hg38/hg38.fa
>chr1
>chr10
>chr11
>chr12
>chr13
>chr14
>chr15
>chr16
>chr17
>chr18
>chr19
>chr2
>chr20
>chr21
>chr22
>chr3
>chr4
>chr5
>chr6
>chr7
>chr8
>chr9
>chrX
>chrY

By default, sequences are soft-masked. Use -m hard for hard masking, or -m none for no masking.

The chromosome sizes are saved in file called <genome_name>.fa.sizes.

You can choose to download gene annotation files with the --annotation option. These will be saved in (gzipped) BED and GTF format.

$ genomepy  install hg38 UCSC --annotation

To facilitate the downloading of genomes not supported by either NCBI, UCSC, or Ensembl, genomes can also be downloaded directly from an url:

$ genomepy install https://research.nhgri.nih.gov/hydra/download/assembly/\Hm105_Dovetail_Assembly_1.0.fa.gz url

This installs the genome under the filename of the link, but can be changed with the --localname option

Finally, in the spirit of reproducibility all selected options are stored in a README.txt. This includes the original name, download location and other genomepy operations (such as regex filtering and time).

Manage plugins.

Use genomepy plugin list to view the available plugins.

$ genomepy plugin list
plugin              enabled
bowtie2             
bwa                 
gmap                
hisat2              
minimap2            
star

Enable plugins as follows:

$ genomepy plugin enable bwa hisat2
Enabled plugins: bwa, hisat2

And disable like this:

$ genomepy plugin disable bwa
Enabled plugins: hisat2

Search for a genome.

$ genomepy search Xenopus
NCBI	Xenopus_tropicalis_v9.1	Xenopus tropicalis; DOE Joint Genome Institute
NCBI	ViralProj30173	Xenopus laevis endogenous retrovirus Xen1; 
NCBI	Xenopus_laevis_v2	Xenopus laevis; International Xenopus Sequencing Consortium
NCBI	v4.2	Xenopus tropicalis; DOE Joint Genome Institute
NCBI	Xtropicalis_v7	Xenopus tropicalis; DOE Joint Genome Institute
Ensembl	JGI 4.2	Xenopus

Only search a specific provider:

$ genomepy search tropicalis -p UCSC
UCSC	xenTro7	X. tropicalis Sep. 2012 (JGI 7.0/xenTro7) Genome at UCSC
UCSC	xenTro3	X. tropicalis Nov. 2009 (JGI 4.2/xenTro3) Genome at UCSC
UCSC	xenTro2	X. tropicalis Aug. 2005 (JGI 4.1/xenTro2) Genome at UCSC
UCSC	xenTro1	X. tropicalis Oct. 2004 (JGI 3.0/xenTro1) Genome at UCSC

Note that searching doesn't work flawlessly, so try a few variations if you don't get any results.

Note that genomes with a space can be searched for either by using "quotation marks", or by replacing the space(s) with and underscore _.

Search is case-insensitive.

List available providers

$ genomepy providers
Ensembl
UCSC
NCBI
URL

List available genomes

You can constrain the genome list by using the -p option to search only a specific provider.

$ genomepy genomes -p UCSC
UCSC	hg38	Human Dec. 2013 (GRCh38/hg38) Genome at UCSC
UCSC	hg19	Human Feb. 2009 (GRCh37/hg19) Genome at UCSC
UCSC	hg18	Human Mar. 2006 (NCBI36/hg18) Genome at UCSC
...
UCSC	danRer4	Zebrafish Mar. 2006 (Zv6/danRer4) Genome at UCSC
UCSC	danRer3	Zebrafish May 2005 (Zv5/danRer3) Genome at UCSC

Manage configuration

List the current configuration file that genomepy uses:

$ genomepy config file
/home/simon/.config/genomepy/genomepy.yaml

To show the contents of the config file:

$ genomepy config show
# Directory were downloaded genomes will be stored
genomes_dir: ~/.local/share/genomes/

plugin:
 - blacklist

To generate a personal configuration file (existing file will be overwritten):

$ genomepy config generate
Created config file /home/simon/.config/genomepy/genomepy.yaml

Local cache.

Note that the first time you run genomepy search or list the command will take a long time as the genome lists have to be downloaded. The lists are cached locally, which will save time later. The cached files are stored in ~/.cache/genomepy and expire after 7 days. You can also delete this directory to clean the cache.

Python

>>> import genomepy
>>> for row in genomepy.search("GRCh38"):
...     print("\t".join(row))
...
UCSC	hg38	Human Dec. 2013 (GRCh38/hg38) Genome at UCSC
NCBI	GRCh38.p10	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p1	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p2	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p3	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p4	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p5	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p6	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p7	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p8	Homo sapiens; Genome Reference Consortium
NCBI	GRCh38.p9	Homo sapiens; Genome Reference Consortium
Ensembl	GRCh38.p10	Human
>>> genomepy.install_genome("hg38", "UCSC", genomes_dir="/data/genomes")
downloading...
done...
name: hg38
fasta: /data/genomes/hg38/hg38.fa
>>> g = genomepy.Genome("hg38", genomes_dir="/data/genomes")
>>> g["chr6"][166502000:166503000]
tgtatggtccctagaggggccagagtcacagagatggaaagtggatggcgggtgccgggggctggggagctactgtgcagggggacagagctttagttctgcaagatgaaacagttctggagatggacggtggggatgggggcccagcaatgggaacgtgcttaatgccactgaactgggcacttaaacgtggtgaaaactgtaaaagtcatgtgtatttttctacaattaaaaaaaATCTGCCACAGAGTTAAAAAAATAACCACTATTTTCTGGAAATGGGAAGGAAAAGTTACAGCATGTAATTAAGATGACAATTTATAATGAACAAGGCAAATCTTTTCATCTTTGCCTTTTGGGCATATTCAATCTTTGCCCAGAATTAAGCACCTTTCAAGATTAATTCTCTAATAATTCTAGTTGAACAACACAACCTTTTCCTTCAAGCTTGCAATTAAATAAGGCTATTTTTAGCTGTAAGGATCACGCTGACCTTCAGGAGCAATGAGAACCGGCACTCCCGGCCTGAGTGGATGCACGGGGAGTGTGTCTAACACACAGGCGTCAACAGCCAGGGCCGCACGAGGAGGAGGAGTGGCAACGTCCACACAGACTCACAACACGGCACTCCGACTTGGAGGGTAATTAATACCAGGTTAACTTCTGGGATGACCTTGGCAACGACCCAAGGTGACAGGCCAGGCTCTGCAATCACCTCCCAATTAAGGAGAGGCGAAAGGGGACTCCCAGGGCTCAGAGCACCACGGGGTTCTAGGTCAGACCCACTTTGAAATGGAAATCTGGCCTTGTGCTGCTGCTCTTGTGGGGAGACAGCAGCTGCGGAGGCTGCTCTCTTCATGGGATTACTCTGGATAAAGTCTTTTTTGATTCTACgttgagcatcccttatctgaaatgcctgaaaccggaagtgtttaggatttggggattttgcaatatttacttatatataatgagatatcttggagatgggccacaa

The genomepy.Genome() method returns a Genome object. This has all the functionality of a pyfaidx.Fasta object, see the documentation for more examples on how to use this.

Known issues

There might be issues with specific genome sequences. Sadly, not everything (naming, structure, filenames) is always consistent on the provider end. Let me know if you encounter issues with certain downloads.

Citation

If you use genomepy in your research, please cite it: 10.21105/joss.00320.

Getting help

If you want to report a bug or issue, or have problems with installing or running the software please create a new issue. This is the preferred way of getting support. Alternatively, you can mail me.

Contributing

Contributions welcome! Send me a pull request or get in touch.

When contributing a PR, please use the develop branch.

Quick development setup:

1. Fork & download this repo.
2. cd into your local repo.
3. git checkout develop
4. conda env create python=3.6 -f environment.yaml
5. conda activate genomepy
6. python setup.py develop
7. python setup.py build
8. git checkout -b your_develop_branch

The command line and python imports will now use the code in your local repo. To test your changes locally, run the following command: pytest -vv --disable-pytest-warnings

Contributors

• Siebren Frölich - @siebrenf
• Simon van Heeringen - @simonvh
• Maarten van der Sande - @Maarten-vd-Sande
• Dohoon Lee - @dohlee
• Jie Zhu - @alienzj

License

This module is licensed under the terms of the MIT license.