BanzaiDB 0.3.0

Database for Banzai NGS pipeline tool

---

(Using landscape.io and drone.io)

News

API changes (v1 -> v2 -> v3 -> v?). The population of a mapping run into BanzaiDB was dependent on a nesoni run. Originally we (API v1) parsed the reports.txt for each strain. In API v2 we parse the nway.any (assumes you have ran nesoni nway). BanzaiDB API v3 assumes that you still have accessto the consensus.fa (called a consensus). We need this data to store information in BanzaiDB about coverage.

What is BanzaiDB?

Please use the releases (https://github.com/mscook/BanzaiDB/releases). All versions including most recent made some significant assumptions that are currently being improved.

BanzaiDB is a tool for pairing Microbial Genomics Next Generation Sequencing (NGS) analysis with a NoSQL database. We use the RethinkDB NoSQL database.

BanzaiDB:

• initialises the NoSQL database and associated tables,

• populates the database with results of NGS experiments/analysis and,

• provides a set of query functions to wrangle with the data stored within the database.

Why BanzaiDB?

The analysis (primary/secondary/tertiary) of large collections of draft microbial genomes from NGS typically generates many separate flat files.

The bioinformatician will:

• write scripts to parse and extract the important information from the results files (often trying to standardise the output from similar programs),

• store these results in further flat files,

• write scripts to link the results of one analysis step to another,

• store these results in further flat files,

• modify scripts as hypothesis is improved as a direct consequence of incorporating the knowledge from the previous steps,

• …

• …

• …

• end up with thousands of flat files, many scripts and generally get confused as to how and where everything came from.

The idea around BanzaiDB is to run once, store once analyse many times.

About BanzaiDB

BanzaiDB is geared towards outputs of Bioinformatics software employed by the Banzai NGS pipeline.

BanzaiDB is thus geared towards handling data generated from:

• Velvet and SPAdes (assembly),

• BWA and Nesoni (mapping/variant calling),

• Mugsy (whole genome alignment),

• BRATTNextGen (recombination detection) and,

• Prokka (annotation).

The present focus is on storing and manipulating the results of SNP and recombination analysis.

Banzai is not a stable API.

See the ReadTheDocs site for BanzaiDB documentation (User & API).

About RethinkDB

We choose RethinkDB as our underlying database for a few reasons:

• RethinkDB is both developer and operations friendly. This sits well with the typical bioinformatician,

• NoSQL databases allow for a flexible schema. We can store/collect now, think later. This is much like how science is performed.

• Not every bioinformatician or lab has a system administrator. RethinkDB is easy to setup and administer

• We don’t know how big our complex our datasets could get in the future. It is easy to scale RethinkDB into a cluster.

• ReQL the underlying query language is nice and simple to learn/understand. We’re also very comfortable with Python and the availability of official python drivers (also JavaScript & Ruby, and a heap for user contributed for a swag of languages) is a big bonus.

BanzaiDB Requirements

You will need:

• (probably) administrator access to your machine(s)

• a RethinkDB server/instance. This can be running locally or on a VPS,

• git (to clone this repository) and

• pip

• bedtools, samtools and tabix (for pybedtools)

You will also need a few Python modules:

• rethinkdb

• biopython

• reportlab

• fabric

• tablib

• argparse (if Python 2.6)

• pybedtools (you will probably also need to install cython)

The Python modules should/will be pulled down automatically when installing BanzaiDB.

We recommend you increase the rethinkdb python driver performance. We have found that in some cases the installation of C++ backend fails. We provide a simple protocol that we have found works.

BanzaiDB Installation

Something like this:

$ git clone https://github.com/mscook/BanzaiDB.git
$ cd BanzaiDB
$ python setup.py install

Getting BanzaiDB talking to RethinkDB

You provide information about you RethinkDB instance and database using the file ~/.BanzaiDB.cfg (~/ is shorthand for $HOME).

The configuration file supports:

db_host  =  [def = localhost]
port     =  [def = 28015]
db_name  =  [def = Banzai]
auth_key =  [def = '']

BanzaiDB usage

Note: Please refer to the BanzaiDB documentation (via ReadTheDocs) for more detailed information (under active development).

Once both RethinkDB and BanzaiDB are installed and the configuration is set:

$ python BanzaiDB.py -h
usage: BanzaiDB.py [-h] [-v] {init,populate,update,query} ...

BanzaiDB v 0.3.0 - Database for Banzai NGS pipeline tool
(http://github.com/mscook/BanzaiDB)

positional arguments:
  {init,populate,update,query}
                        Available commands:
    init                Initialise a DB
    populate            Populates a database with results of an experiment
    update              Updates a database with results from a new experiment
    query               List available or provide database query functions

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output

Licence: ECL 2.0 by Mitchell Stanton-Cook <[e-mail]>