Skip to main content

Big data development environments using Docker

Project description

Ferry: big data development engine
====================================

Ferry lets you provision and deploy big data stacks on your local machine using [Docker](https://www.docker.io).

Ferry currently supports Hadoop/Yarn, GlusterFS/OpenMPI, and Cassandra (with more in the future).
By using Ferry developers can get started creating their big data applications right away without
the pain of installing and configuring all the complex backend software.

Big Data in small places
========================

Big data technologies are designed to operate and scale over many machines and usually consist
of multiple functional parts. Developers interested in creating a
Hadoop application, for example, must first download the appropriate packages, configure these
systems to operate in a single-machine environment (or multiple machines for operational environments),
and configure other required services (e.g., PostgreSQL).

Fortunately for us, Ferry and Docker vastly simplify the entire process by capturing the entire process
in a set of lightweight Linux containers. This enables developers to quickly stand up a big data stack and
attach connectors/clients with zero manual configuration. Because Docker is so lightweight, you can even test
multiple big data stacks with minimal overhead.

Getting started
===============

Ferry is a Python application and runs on your local machine. All you have to do to get started is have
`docker` installed and type the following `pip install -U ferry`. Afterwards you can start creating
your big data application. Here's an example stack:

```javascript
{
"backend":[
{
"storage":
{
"personality":"gluster",
"instances":2
},
"compute":[
{
"personality":"mpi",
"instances":2
}]
}],
"connectors":[
{"personality":"mpi-client"}
]
}
```

This stack consists of two GlusterFS data nodes, and two OpenMPI compute nodes. There's also a Linux
client that automatically connects to those backend components. To create this stack, just type
`ferry start openmpi`. Once you create the stack, you can log in by typing `ferry ssh sa-0`.

More detailed installation instructions and examples can be found [here](http://ferry.opencore.io).

Under the hood
==============

Ferry leverages some awesome open source projects:

* [Docker](https://www.docker.io) simplifies the management of Linux containers
* [Python](http://www.python.org) programming language
* [Hadoop](http://hadoop.apache.org) is a general-purpose big data storage and processing framework
* [GlusterFS](http://www.gluster.org) is a parallel filesystem actively developed by Redhat
* [OpenMPI](http://www.open-mpi.org) is a scalable MPI implementation focused on modeling & simulation
* [Cassandra](http://cassandra.apache.org) is a highly scalable column store
* [PostgreSQL](http://postgresql.org) is a popular relational database

Project details


Release history Release notifications

History Node

0.3.3.4

History Node

0.3.3.3

History Node

0.3.3.2

History Node

0.3.3.1

History Node

0.3.3

History Node

0.3.2

History Node

0.3.1.3

History Node

0.3.1.2

History Node

0.3.1.1

History Node

0.3.1

History Node

0.3.0

History Node

0.2.4.1

History Node

0.2.4

History Node

0.2.3.1

History Node

0.2.3

History Node

0.2.2

History Node

0.2.1

History Node

0.2.0

History Node

0.1.28

History Node

0.1.27

History Node

0.1.26

History Node

0.1.25

This version
History Node

0.1.24

History Node

0.1.23

History Node

0.1.22

History Node

0.1.21

History Node

0.1.20

History Node

0.1.19

History Node

0.1.18

History Node

0.1.17

History Node

0.1.16

History Node

0.1.15

History Node

0.1.14

History Node

0.1.13

History Node

0.1.12

History Node

0.1.11

History Node

0.1.10

History Node

0.1.9

History Node

0.1.8

History Node

0.1.7

History Node

0.1.6

History Node

0.1.5

History Node

0.1.4

History Node

0.1.3

History Node

0.1.2

History Node

0.1.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
ferry-0.1.24.tar.gz (4.0 MB) Copy SHA256 hash SHA256 Source None Mar 28, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page