Manage a small OAR developpement cluster with docker.
Project description
Build your own OAR cluster with docker
--------------------------------------
oar-docker is a set of docker images especially configured for deploying
your own OAR cluster. The main idea is to have a mini development cluster with
a frontend, a server and some nodes that launch in just a few seconds on a
simple laptop.
Why use oar-docker ?
--------------------
Various case scenarios may affect you:
- Quickly test OAR on a cluster
- Gain time: a ten-node cluster (or more) is launched in just a
few seconds and is cleaned in less than a second.
- Save resources: docker allows the user to pool the node between
various systems, resource utilization is thus considerably reduced.
- Synced volume : allowing you to continue working on your host machine, but
use the resources in the guest machine to compile or run your project.
Installation
------------
Requirements:
- python 2.7
- docker >= 1.3
You can install, upgrade, uninstall oar-docker with these commands::
$ pip install oar-docker
$ pip install --upgrade oar-docker
$ pip uninstall oar-docker
Or from git (last development version)::
$ pip install git+https://github.com/oar-team/oar-docker.git
Or if you already pulled the sources::
$ pip install path/to/sources
Or if you don't have pip::
$ easy_install oar-docker
Usage
-----
::
Usage: oardocker [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Manage a small OAR developpement cluster with docker.
Options:
--workdir DIRECTORY Changes the folder to operate on.
--docker-host TEXT The docker socket [default:unix://var/run/docker.sock]
--cgroup-path TEXT The cgroup file system path [default: /sys/fs/cgroup]
--docker-binary TEXT The docker client binary [default: docker]
--version Show the version and exit.
-h, --help Show this message and exit.
Commands:
build Build base images
clean Remove all stopped containers and untagged...
connect Connect to a node.
destroy Stop containers and remove all images
init Initialize a new environment.
install Install and configure OAR from src
logs Fetch the logs of all containers.
ssh Connect to machine via SSH [deprecated]
ssh-config Output OpenSSH valid ssh config [deprecated]
start Create and start the containers
status Output status of the cluster
stop Stop and remove all containers
Getting started
---------------
To get started with oar-docker, the first thing to do is to initialize a
project::
$ oardocker init
If you already have OAR sources, the best is to initialize directly the
oardocker project in the OAR sources directory::
$ cd path/to/oar/src
$ oardocker init
You have to do this only once. It allows you to import the Dockerfiles
and other configuration files.
We then launch the base image build::
$ oardocker build
Now, we have to install OAR. To do this, several options are available.
If you already have the OAR sources::
$ oardocker install . ## ou . est le chemin vers les sources de OAR
Or if you want to install from tarball::
$ oardocker install http://oar-ftp.imag.fr/oar/2.5/sources/testing/oar-2.5.4+rc4.tar.gz
You can also launch the installation from a git repository::
$ oardocker install git+https://github.com/oar-team/oar.git
We start a OAR cluster with 5 nodes::
$ oardocker start -n 5
It is possible to share directories between host machines and
all containers with the ``-v`` option::
$ oardocker start -v $PWD:/oar_src -v /my/custom/lib:/usr/local/ma_lib
To manage the cluster::
$ oardocker connect frontend|server|nodeXX
$ oardocker logs [frontend|server|nodeXX]
To clean::
$ oardocker stop ## stops and removes all containers
$ oardocker clean ## removes all stopped containers (failed) and the untagged images <none:none>
$ oardocker destroy ## removes all images and containers
With oar-docker, it is possible to chain all commands to go faster::
$ oardocker init -f build install oar-2.5.4+rc4.tar.gz start -n 4 connect -l root frontend
For instance, to develop on OAR, we often need to install OAR,
start the cluster and connect to it::
$ oardocker install $PWD start -n 10 -v $PWD:/home/docker/oar_src connect frontend
One last thing to know. The ``stop`` command is automatically launched before
every ``start``, ``install`` and ``build`` ... If we launch multiple times the
last command, we will always obtain the same result. It can be useful to
experiment and develop (even) faster.
Security
--------
oar-docker is a development project and a testing one. It is in no way secure.
Besides, the private ssh key used is also insecured since it is public (you can
find it in the sources).
Related resources
-----------------
- `A minimal Ubuntu base image modified for Docker-friendliness`_
- `Got a Minute? Spin up a Spark cluster on your laptop with Docker`_
oar-docker CHANGELOG
====================
version 0.3.0
=============
Released on Nov 27th 2014
**Features**:
- Added ``oardocker exec`` command
- Manage multiple environment variants with ``oardocker init``: added wheezy|jessie|python bases images
**Bug fixes**:
- Revert default environment to Debian Wheezy due to breaking OAR API in Jessie
- Fixed locales issue
**Improvements**:
- better synchronisation between oar-server and postgresql services
version 0.2.0
=============
Released on Nov 5th 2014
**Features**:
- Updated base images to debian jessie
- Added ``oardocker connect`` to connect to the nodes without ssh
- The commands ``oardocker ssh`` and ``oardocker ssh-config`` are deprecated from now
**Improvements**:
- Removed supervisor and make init process less complex by only using my_init.d statup scripts
- Customized help parameter to accept ``-h`` and ``--help``
- Used docker client binary for some task instead of the API
**Bug fixes**:
- Make sure that /etc/hosts file contain the localhost entry
version 0.1.4
=============
Released on Oct 28th 2014
- Ignored my-init scripts if filename ends by "~"
- Added wait_pgsql script to wait postgresql to be available
- Fixed monika config (db server hostname is server)
- Removed old code
- Adapt cgroup mount script to job_resource_manager_cgroup.pl and remove old cpuset workaround
- Fixed cpu/core/thread affinity
version 0.1.3
=============
Released on Sep 10th 2014
- Added `oar reset` cmd to restart containers
- Added a better comments about oardocker images with git information
- Used default job_resource_manager script (from oar sources)
- Mount the host cgroup path in the containers (default path is /sys/fs/cgroup)
- Removed stopped containers from ssh_config
- Remove dnsmasq and mount a custom /etc/hosts for the nodes (need docker >= 1.2.0)
version 0.1.2
=============
Released on Sep 16th 2014
- Keep compatible with older versions of git
- Don't name the containers
- Mounting OAR src as Copy-on-Write directory with unionfs-fuse
- Stopped installation when container failed during ``oardocker install``
- Added option to print version
- Allow ssh connection with different user
version 0.1.1
=============
Released on Sep 11th 2014
- Minor bug fixes
version 0.1
===========
Released on Sep 11th 2014
Initial public release of oar-docker
--------------------------------------
oar-docker is a set of docker images especially configured for deploying
your own OAR cluster. The main idea is to have a mini development cluster with
a frontend, a server and some nodes that launch in just a few seconds on a
simple laptop.
Why use oar-docker ?
--------------------
Various case scenarios may affect you:
- Quickly test OAR on a cluster
- Gain time: a ten-node cluster (or more) is launched in just a
few seconds and is cleaned in less than a second.
- Save resources: docker allows the user to pool the node between
various systems, resource utilization is thus considerably reduced.
- Synced volume : allowing you to continue working on your host machine, but
use the resources in the guest machine to compile or run your project.
Installation
------------
Requirements:
- python 2.7
- docker >= 1.3
You can install, upgrade, uninstall oar-docker with these commands::
$ pip install oar-docker
$ pip install --upgrade oar-docker
$ pip uninstall oar-docker
Or from git (last development version)::
$ pip install git+https://github.com/oar-team/oar-docker.git
Or if you already pulled the sources::
$ pip install path/to/sources
Or if you don't have pip::
$ easy_install oar-docker
Usage
-----
::
Usage: oardocker [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Manage a small OAR developpement cluster with docker.
Options:
--workdir DIRECTORY Changes the folder to operate on.
--docker-host TEXT The docker socket [default:unix://var/run/docker.sock]
--cgroup-path TEXT The cgroup file system path [default: /sys/fs/cgroup]
--docker-binary TEXT The docker client binary [default: docker]
--version Show the version and exit.
-h, --help Show this message and exit.
Commands:
build Build base images
clean Remove all stopped containers and untagged...
connect Connect to a node.
destroy Stop containers and remove all images
init Initialize a new environment.
install Install and configure OAR from src
logs Fetch the logs of all containers.
ssh Connect to machine via SSH [deprecated]
ssh-config Output OpenSSH valid ssh config [deprecated]
start Create and start the containers
status Output status of the cluster
stop Stop and remove all containers
Getting started
---------------
To get started with oar-docker, the first thing to do is to initialize a
project::
$ oardocker init
If you already have OAR sources, the best is to initialize directly the
oardocker project in the OAR sources directory::
$ cd path/to/oar/src
$ oardocker init
You have to do this only once. It allows you to import the Dockerfiles
and other configuration files.
We then launch the base image build::
$ oardocker build
Now, we have to install OAR. To do this, several options are available.
If you already have the OAR sources::
$ oardocker install . ## ou . est le chemin vers les sources de OAR
Or if you want to install from tarball::
$ oardocker install http://oar-ftp.imag.fr/oar/2.5/sources/testing/oar-2.5.4+rc4.tar.gz
You can also launch the installation from a git repository::
$ oardocker install git+https://github.com/oar-team/oar.git
We start a OAR cluster with 5 nodes::
$ oardocker start -n 5
It is possible to share directories between host machines and
all containers with the ``-v`` option::
$ oardocker start -v $PWD:/oar_src -v /my/custom/lib:/usr/local/ma_lib
To manage the cluster::
$ oardocker connect frontend|server|nodeXX
$ oardocker logs [frontend|server|nodeXX]
To clean::
$ oardocker stop ## stops and removes all containers
$ oardocker clean ## removes all stopped containers (failed) and the untagged images <none:none>
$ oardocker destroy ## removes all images and containers
With oar-docker, it is possible to chain all commands to go faster::
$ oardocker init -f build install oar-2.5.4+rc4.tar.gz start -n 4 connect -l root frontend
For instance, to develop on OAR, we often need to install OAR,
start the cluster and connect to it::
$ oardocker install $PWD start -n 10 -v $PWD:/home/docker/oar_src connect frontend
One last thing to know. The ``stop`` command is automatically launched before
every ``start``, ``install`` and ``build`` ... If we launch multiple times the
last command, we will always obtain the same result. It can be useful to
experiment and develop (even) faster.
Security
--------
oar-docker is a development project and a testing one. It is in no way secure.
Besides, the private ssh key used is also insecured since it is public (you can
find it in the sources).
Related resources
-----------------
- `A minimal Ubuntu base image modified for Docker-friendliness`_
- `Got a Minute? Spin up a Spark cluster on your laptop with Docker`_
oar-docker CHANGELOG
====================
version 0.3.0
=============
Released on Nov 27th 2014
**Features**:
- Added ``oardocker exec`` command
- Manage multiple environment variants with ``oardocker init``: added wheezy|jessie|python bases images
**Bug fixes**:
- Revert default environment to Debian Wheezy due to breaking OAR API in Jessie
- Fixed locales issue
**Improvements**:
- better synchronisation between oar-server and postgresql services
version 0.2.0
=============
Released on Nov 5th 2014
**Features**:
- Updated base images to debian jessie
- Added ``oardocker connect`` to connect to the nodes without ssh
- The commands ``oardocker ssh`` and ``oardocker ssh-config`` are deprecated from now
**Improvements**:
- Removed supervisor and make init process less complex by only using my_init.d statup scripts
- Customized help parameter to accept ``-h`` and ``--help``
- Used docker client binary for some task instead of the API
**Bug fixes**:
- Make sure that /etc/hosts file contain the localhost entry
version 0.1.4
=============
Released on Oct 28th 2014
- Ignored my-init scripts if filename ends by "~"
- Added wait_pgsql script to wait postgresql to be available
- Fixed monika config (db server hostname is server)
- Removed old code
- Adapt cgroup mount script to job_resource_manager_cgroup.pl and remove old cpuset workaround
- Fixed cpu/core/thread affinity
version 0.1.3
=============
Released on Sep 10th 2014
- Added `oar reset` cmd to restart containers
- Added a better comments about oardocker images with git information
- Used default job_resource_manager script (from oar sources)
- Mount the host cgroup path in the containers (default path is /sys/fs/cgroup)
- Removed stopped containers from ssh_config
- Remove dnsmasq and mount a custom /etc/hosts for the nodes (need docker >= 1.2.0)
version 0.1.2
=============
Released on Sep 16th 2014
- Keep compatible with older versions of git
- Don't name the containers
- Mounting OAR src as Copy-on-Write directory with unionfs-fuse
- Stopped installation when container failed during ``oardocker install``
- Added option to print version
- Allow ssh connection with different user
version 0.1.1
=============
Released on Sep 11th 2014
- Minor bug fixes
version 0.1
===========
Released on Sep 11th 2014
Initial public release of oar-docker
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
oar-docker-0.3.0.tar.gz
(141.8 kB
view hashes)