Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Analysis of big biological data sets for distributed HPC clusters.

Project description

A commandline tool for analysis of big biological data sets for distributed HPC clusters.

About

PyBDA is a Python library and command line tool for big data analytics and machine learning scaling to tera byte sized data sets.

In order to make PyBDA scale to big data sets, we use Apache Spark’s DataFrame API which, if developed against, automatically distributes data to the nodes of a high-performance cluster and does the computation of expensive machine learning tasks in parallel. For scheduling, PyBDA uses Snakemake to automatically execute pipelines of jobs. In particular, PyBDA will first build a DAG of methods/jobs you want to execute in succession (e.g. dimensionality reduction into clustering) and then compute every method by traversing the DAG. In the case of a successful computation of a job, PyBDA will write results and plots, and create statistics. If one of the jobs fails PyBDA will report where and which method failed (owing to Snakemake’s scheduling) such that the same pipeline can effortlessly be continued from where it failed the last time.

Documentation

Check out the documentation here. The documentation will walk you though

  • the installation process,
  • setting up Apache Spark,
  • using pybda.

Author

Simon Dirmeier simon.dirmeier at bsse.ethz.ch.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pybda, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size pybda-0.1.0-py2.py3-none-any.whl (127.2 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size pybda-0.1.0.tar.gz (55.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page