slapos.core 0.32.2

SlapOS core.

slapos.core

The core of SlapOS. Contains the SLAP library, and the slapgrid, slapformat, slapproxy tools. For more information, see http://www.slapos.org.

Changes

0.32.2 (2012-10-11)

• slapgrid: Remove default delay, now that SlapOS Master is Fast as Light (tm). (commit 03a85d6b8) [Cedric de Saint Martin]

• Fix watchdog entry point name, introduced in v0.31. (commit a8651ba12) [Cedric de Saint Martin]

• slapgrid: Better filter of instances, won’t process false positives anymore (hopefully). (commit ce0a73b41) [Cedric de Saint Martin]

• Various output improvements. [Cedric de Saint Martin]

0.32.1 (2012-10-09)

• slapgrid: Make sure error logs are sent to SlapOS master. Finish implementation began in 0.32. [Cedric de Saint Martin]

• slapgrid: Fix Usage Report in case of not empty partition with no SR. [Cedric de Saint Martin]

0.32 (2012-10-04)

• Introduce new, simpler “slapos” entry point. See documentation for more informations. Note: some functionnalities of this new entry point don’t work yet or is not as simple as it should be. [Cedric de Saint Martin, Cedric Le Ninivin]

• Revamped “slapos request” to work like described in documentation. [Cédric Le Ninivin, Cédric de Saint Martin]

• Rewrote slapgrid logger to always log into stdout. (commits a4d277c881, 5440626dea)[Cédric de Saint Martin]

0.31.2 (2012-10-02)

• Update slapproxy behavior: when instance already exist, only update partition_parameter_kw. (commit 317d5c8e0aee) [Cedric de Saint Martin]

0.31.1 (2012-10-02)

• Fixed Watchdog call in slapgrid. [Cédric Le Ninivin]

0.31 (2012-10-02)

• Added slapos-watchdog to bang exited and failing serices in instance in supervisord. (commits 16b2e8b8, 1dade5cd7) [Cédric Le Ninivin]

• Add safety checks before calling SlapOS Master if mandatory instance members of SLAP classes are not properly set. Will result in less calls to SlapOS Master in dirty cases. (commits 5097e87c9763, 5fad6316a0f6d, f2cd014ea8aa) [Cedric de Saint Martin]

• Add “periodicty” functionnality support for instances: if an instance has not been processed by slapgrid after defined time, process it. (commits 7609fc7a3d, 56e1c7bfbd) [Cedric Le Ninivin]

• slapproxy: Various improvements in slave support (commits 96c6b78b67, bcac5a397d, fbb680f53b)[Cedric Le Ninivin]

• slapgrid: bulletproof slapgrid-cp: in case one instance is bad, still processes all other ones. (commits bac94cdb56, 77bc6c75b3d, bd68b88cc3) [Cedric de Saint Martin]

• Add support for “upload to binary cache” URL blacklist [Cedric de Saint Martin]

• Request on proxy are identified by requester and name (commit 0c739c3) [Cedric Le Ninivin]

0.30 (2012-09-19)

• Add initial “slave instances” support in slapproxy. [Cedric Le Ninivin]

• slapgrid-ur fix: check for partition informations only if we have to destroy it. [Cedric de Saint Martin]

0.29 (2012-09-18)

• buildout: Migrate slap_connection magic instance profile part to slap-connection, and use variables names separated with ‘-’. [Cedric de Saint Martin]

• slapgrid: Add support for instance.cfg instance profiles [Cedric de Saint Martin]

• slapgrid-ur: much less calls to master. [Cedric de Saint Martin]

0.28.9 (2012-09-18)

• slapgrid: Don’t process not updated partitions (regression introduced in 0.28.7). [Cedric de Saint Martin]

0.28.8 (2012-09-18)

• slapgrid: Don’t process free partitions (regression introduced in 0.28.7). [Cedric de Saint Martin]

0.28.7 (2012-09-14)

• slapgrid: –maximal_delay reappeared to be used in special cases. [Cedric de Saint Martin]

0.28.6 (2012-09-10)

• register now use slapos.cfg.example from master. [Cédric Le Ninivin]

0.28.5 (2012-08-23)

• Updated slapos.cfg for register [Cédric Le Ninivin]

0.28.4 (2012-08-22)

• Fixed egg building.

0.28.3 (2012-08-22)

• Avoid artificial tap creation on system check. [Łukasz Nowak]

0.28.2 (2012-08-17)

• Resolved path problem in register [Cédric Le Ninivin]

0.28.1 (2012-08-17)

• Resolved critical naming conflict

0.28 (2012-08-17)

• Introduce “slapos node register” command, that will register computer to SlapOS Master (vifib.net by default) for you. [Cédric Le Ninivin]

• Set .timestamp in partitions ONLY after slapgrid thinks it’s okay (promises, …). [Cedric de Saint Martin]

• slapgrid-ur: when destroying (not reporting), only care about instances to destroy, completely ignore others. [Cedric de Saint Martin]

0.27 (2012-08-08)

• slapformat: Raise correct error when no IPv6 is available on selected interface. [Cedric de Saint Martin]

• slapgrid: Introduce –only_sr and –only_cp.

  • only_sr filter and force the run of a single SR, and uses url_md5 (folder_id)

  • only_cp filter which computer patition, will be runned. it can be a list, splited by comman (slappartX,slappartY …) [Rafael Monnerat]

• slapgrid: Cleanup unused option (–usage-report-periodicity). [Cedric de Saint Martin]

• slapgrid: –develop will work also for Computer Partitions. [Cedric de Saint Martin]

• slaplib: setConnectionDict won’t call Master if parameters haven’t changed. [Cedric de Saint Martin]

0.26.2 (2012-07-09)

• Define UTF-8 encoding in SlapOS Node codebase, as defined in PEP-263.

0.26.1 (2012-07-06)

• slapgrid-sr: Add –develop option to make it ignore .completed files.

• SLAP library: it is now possible to fetch whole dict of connection parameters.

• SLAP library: it is now possible to fetch single instance parameter.

• SLAP library: change Computer and ComputerPartition behavior to have proper caching of computer partition parameters.

0.26 (2012-07-05)

• slapformat: no_bridge option becomes ‘not create_tap’. create_tap is true by default. So a bridge is used and tap will be created by default. [Cedric de Saint Martin]

• Add delay for slapformat. [Cedric Le Ninivin]

• If no software_type is given, use default one (i.e fix “error 500” when requesting new instance). [Cedric de Saint Martin]

• slapgrid: promise based software release, new api to fetch full computer information from server. [Yingjie Xu]

• slapproxy: new api to mock full computer information [Yingjie Xu]

• slapgrid: minor fix randomise delay feature. [Yingjie Xu]

• slapgrid: optimise slapgrid-cp, run buildout only if there is an update on server side. [Yingjie Xu]

• libslap: Allow accessing ServerError. [Vincent Pelletier]

0.25 (2012-05-16)

• Fix support for no_bridge option in configuration files for some values: no_bridge = false was stated as true. [Cedric de Saint Martin]

• Delay a randomized period of time before calling slapgrid. [Yingjie Xu]

• slapformat: Don’t require tunctl if no_bridge is set [Leonardo Rochael]

• slapformat: remove monkey patching when creating address so that it doesn’t return false positive. [Cedric de Saint Martin]

• Various: clearer error messages.

0.24 (2012-03-29)

• Handles different errors in a user friendly way [Cedric de Saint Martin]

• slapgrid: Supports software destruction. [Łukasz Nowak]

• slap: added support to Supply.supply state parameter (available, destroyed) [Łukasz Nowak]

0.23 (2012-02-29)

• slapgrid : Don’t create tarball of sofwtare release when shacache is not configured. [Yingjie Xu]

0.22 (2012-02-09)

• slapformat : Add no-bridge feature. [Cedric de Saint Martin]

• slapgrid : Add binary cache support. [Yingjie Xu]

0.21 (2011-12-23)

• slap: Add renaming API. [Antoine Catton]

0.20 (2011-11-24)

• slapgrid: Support service-less parttions. [Antoine Catton]

• slapgrid: Avoid gid collision while dropping privileges. [Antoine Catton]

• slapgrid: Drop down network usage during usage reporting. [Łukasz Nowak]

• general: Add sphinx documentation. [Romain Courteaud]

0.19 (2011-11-07)

• bang: Executable to be called by being banged computer. [Łukasz Nowak]

0.18 (2011-10-18)

• Fix 0.17 release: missing change for slap library. [Łukasz Nowak]

0.17 (2011-10-18)

• slap: Avoid request under the hood. [Łukasz Nowak]

• slap: ComputerPartition.bang provided. It allows to update all instances in tree. [Łukasz Nowak]

• slap: Computer.bang provided. It allows to bang all instances on computer. [Łukasz Nowak]

0.16 (2011-10-03)

• slapgrid: Bugfix for slapgrid introduced in 0.15. [Łukasz Nowak]

0.15 (2011-09-27)

• slapgrid: Sanitize environment variables as early as possible. [Arnaud Fontaine]

• slap: Docstring bugfix. [Sebastien Robin]

• slap: Make request asynchronous call. [Łukasz Nowak]

0.14 (2011-08-31)

• slapgrid: Implement SSL based authentication to shadir and shacache. [Łukasz Nowak]

• slapgrid, slap: Fix usage report packing list generation. [Nicolas Godbert]

0.13 (2011-08-25)

• slapgrid: Implement software signing and shacache upload. [Lucas Carvalho]

• slap: Support slave instances [Gabriel Monnerat]

• slapformat: Generate always address for computer [Łukasz Nowak]

• slapgrid: Support promises scripts [Antoine Catton]

• general: slapos.core gets tests. [many contributors]

0.12 (2011-07-15)

• Include modifications that should have been included in 0.11.

0.11 (2011-07-15)

• Bug fix : slapconsole : shorthand methods request and supply now correctly return an object. [Cedric de Saint Martin]

0.10 (2011-07-13)

• Fix a bug in slapconsole where request and supply shorthand methods don’t accept all needed parameters. [Cedric de Saint Martin]

0.9 (2011-07-11)

• slapconsole: Simplify usage and use configuration file. You can now just run slapconsole and type things like “request(kvm, ‘mykvm’)”. [Cedric de Saint Martin]

• slapformat: Fix issue of bridge not connected with real interface on Linux >= 2.6.39 [Arnaud Fontaine]

• slapformat: Allow to have IPv6 only interface, with bridge still supporting local IPv4 stack. [Łukasz Nowak]

0.8 (2011-06-27)

• slapgrid: Bugfix for temporary extends cache permissions. [Łukasz Nowak]

0.7 (2011-06-27)

• slapgrid: Fallback to buildout in own search path. [Łukasz Nowak]

0.6 (2011-06-27)

• slap: Fix bug: state shall be XML encapsulated. [Łukasz Nowak]

0.5 (2011-06-24)

• slapgrid: Use temporary extends-cache directory in order to make faster remote profile refresh. [Łukasz Nowak]

0.4 (2011-06-24)

• general: Polish requirement versions. [Arnaud Fontaine]

• general: Remove libnetworkcache. [Lucas Carvalho]

• slap: Remove not needed method from interface. [Romain Courteaud]

• slap: state parameter is accepted and transmitted to SlapOS master [Łukasz Nowak]

• slapformat: Implement dry run. [Vincent Pelletier]

• slapgrid: Allow to select any buildout binary used to bootstrap environment. [Łukasz Nowak]

0.3 (2011-06-14)

• slap: Implement SLA by filter_kw in OpenOrder.request. [Łukasz Nowak]

• slap: Timeout network operations. [Łukasz Nowak]

• slapformat: Make slapsoft and slapuser* system users. [Kazuhiko Shiozaki]

• slapgrid: Add more tolerance with supervisord. [Łukasz Nowak]

0.2 (2011-06-01)

• Include required files in distribution [Łukasz Nowak]

0.1 (2011-05-27)

• Merged slapos.slap, slapos.tool.console, slapos.tool.format, slapos.tool.grid, slapos.tool.libnetworkcache and slapos.tool.proxy into one package: slapos.core

console

The slapconsole tool allows to interact with a SlapOS Master throught the SLAP library.

For more information about SlapOS or slapconsole usages, please go to http://community.slapos.org.

The slapconsole tool is only a bare Python console with several global variables defined and initialized.

Initialization and configuration file

Slapconsole allows to automatically connect to a Master using URL and SSL certificate from given slapos.cfg. Certificate has to be USER certificate, manually obtained from SlapOS master web interface.

Slapconsole tools reads the given slapos.cfg configuration file and use the following informations :

• Master URL is read from [slapos] in the “master_url” parameter.

• SSL Certificate is read from [slapconsole] in the “cert_file” parameter.

• SSL Key is read from [slapconsole] in the “key_file” parameter.

See slapos.cfg.example for examples.

Global functions

• request() is a shorthand for slap.registerOpenOrder().request() allowing to request instances.

• supply() is a shorthand for slap.registerSupply().supply() allowing to request software installation.

For more information about those methods, please read the SLAP library documentation.

Global aliases

“software_list” is a list containing all the Software Release URLs defined in client slapos.cfg configuration file.

Also, each Software Release defined in this configuration file is translated into a global variable to ease the request of those Sofware Releases.

This allows to request instances in a few words, i.e:

request("http://www.url.com/path/to/kvm/software.cfg", "mykvm")

can be simplified into

request(kvm, "mykvm")

If the slapos.cfg file contains

alias =
  kvm http://www.url.com/path/to/kvm/software.cfg

Global objects

“slap” is an instance of the SLAP library. It is only used for advanced usages. “slap” instance is obtained by doing

slap = slapos.slap.slap()
slap.initializeConnection(config.master_url,
  key_file=config.key_file, cert_file=config.cert_file)

Examples

>>> # Request instance
>>> request(kvm, "myuniquekvm")

>>> # Request instance on specific computer
>>> request(kvm, "myotheruniquekvm",
  filter_kw={ "computer_guid": "COMP-12345" })

>>> # Request instance, specifying parameters (here nbd_ip and nbd_port)
>>> request(kvm, "mythirduniquekvm",
  partition_parameter_kw={"nbd_ip":"2a01:e35:2e27:460:e2cb:4eff:fed9:48dc",
  "nbd_port":"1024"})

>>> # Request software installation on owned computer
>>> supply(kvm, "mycomputer")

>>> # Fetch existing instance status
>>> request(kvm, "myuniquekvm").getState()

>>> # Fetch instance informations on already launched instance
>>> request(kvm, "myuniquekvm").getConnectionParameter("url")

format

slapformat is an application to prepare SlapOS ready node (machine).

It “formats” the machine by:

• creating users and groups

• creating bridge interface

• creating needed tap interfaces

• creating needed directories with proper ownership and permissions

In the end special report is generated and information are posted to configured SlapOS server.

This program shall be only run by root.

Requirements

Linux with IPv6, bridging and tap interface support.

Binaries:

• brctl

• groupadd

• ip

• tunctl

• useradd

grid

slapgrid is a client of SLAPos. SLAPos provides support for deploying a SaaS system in a minute. Slapgrid allows you to easily deploy instances of softwares based on buildout profiles. For more informations about SLAP and SLAPos, please see the SLAP documentation.

Requirements

A working SLAP server with informations about your computer, in order to retrieve them.

As Vifib servers use IPv6 only, we strongly recommend an IPv6 enabled UNIX box.

For the same reasons, Python >= 2.6 with development headers is also strongly recommended (IPv6 support is not complete in previous releases).

For now, gcc and glibc development headers are required to build most software releases.

Concepts

Here are the fundamental concepts of slapgrid : A Software Release (SR) is just a software. A Computer Partition (CP) is an instance of a Software Release. Imagine you want to install with slapgrid some software and run it. You will have to install the software as a Software Release, and then instantiate it, i.e configuring it for your needs, as a Computer Partition.

How it works

When run, slapgrid will authenticate to the SLAP library with a computer_id and fetch the list of Software Releases to install or remove and Computer Partitions to start or stop. Then, it will process each Software Release, and each Computer Partition. It will also periodically send to SLAP the usage report of each Computer Partition.

Installation

With easy_install:

$ easy_install slapgrid

slapgrid needs several directories to be created and configured before being able to run : a software releases directory, and an instances directory with configured computer partition directory(ies). You should create for each Computer Partition directory created a specific user and associate it with its Computer Partition directory. Each Computer Partition directory should belongs to this specific user, with permissions of 0750.

Usage

slapgrid needs several informations in order to run. You can specify them by adding arguments to the slapgrid command line, or by putting then in a configuration file. Beware : you need a valid computer resource on server side.

Examples

simple example : Just run slapgrid:

$ slapgrid –instance-root /path/to/instance/root –software-root /path/to/software_root –master-url https://some.server/some.resource –computer-id my.computer.id

configuration file example:

[slapgrid]
instance_root = /path/to/instance/root
software_root = /path/to/software/root
master_url = https://slapos.server/slap_service
computer_id = my.computer.id

then run slapgrid:

$ slapgrid --configuration-file = path/to/configuration/file

proxy

slap

Simple Language for Accounting and Provisioning python library.

Developer note - python version

This library is used on client (slapgrid) and server side. Server is using python2.4 and client is using python2.6 Having this in mind, code of this library have to work on python2.4

How it works

The SLAP main server which is in charge of service coordination receives from participating servers the number of computer paritions which are available, the type of resource which a party is ready provide, and request from parties for resources which are needed.

Each participating server is identified by a unique ID and runs a slap-server daemon. This dameon collects from the main server the installation tasks and does the installation of resources, then notifies the main server of completion whenever a resource is configured, installed and available.

The data structure on the main server is the following:

A - Action: an action which can happen to provide a resource or account its usage CP - Computer Partition: provides a URL to Access a Cloud Resource RI - Resource Item: describes a resource CI - Contract Item: describes the contract to attach the DL to (This is unclear still) R - Resource: describes a type of cloud resource (ex. MySQL Table) is published on slapgrid.org DL - Delivery Line: Describes an action happening on a resource item on a computer partition D - Delivery: groups multiple Delivery Lines