slapos.cookbook 1.0.121

SlapOS recipes.

slapos.cookbook

Cookbook of SlapOS recipes.

testing

Unit tests for recipes can be found under slapos/test/recipe. To run the tests use provided unittest.defaultTestLoader inside slapos/test/test_recipe by invoking

python setup.py test –test-suite slapos.test.test_recipe.additional_tests

Changes

1.0.121 (2019-09-12)

• generic.mysql.wrap_update_mysql: prepare for MariaDB 10.4

• publish-early: process -init entries in specified order

• Partial support of Python 3

• Remove unused generic.mysql recipe

1.0.119 (2019-08-14)

• publish_early: rework API

1.0.118 (2019-08-13)

• NEO: new recipe to fix/optimize propagation of the ‘masters’ parameter

• publish_early: new ‘-update’ option, keep published values out of buildout installed file

• publish: new -publish option to list explicitly options to publish

• re6stnet: Fix typo

• librecipe: Try to reuse existing file to avoid excessive IO on update and other minor optimisations

• certificate_authority: unique_subject = no

• wrapper: handle “=” in environment variables’ content

1.0.92 (2019-02-21)

• plugin recipe: improve recipe to correctly generate promise with parameters which contain control characters

1.0.85 (2018-12-28)

• Drop slapos.recipe:xvfb, use simple slapos.recipe:wrapper instead.

• Drop slapos.recipe:seleniumrunner and slapos.recipe:firefox, they were not used.

• Encode unicode to UTF-8 on slapos.recipe:request and slapos.recipe:slapconfiguration

1.0.75 (2018-09-04)

• erp5_test: stop using erp5_test recipe

• random: fix password generation with newlines

• erp5testnode: enable password authentication for scalability test system

• pbs: Ignore numerical IDs (UID/GID) when push

• request: add requestoptional.serialised

1.0.65 (2018-06-22)

• Automatic restart of services when configuration changes

• erp5_test: define cloudooo-retry-count value in test

• userinfo: expose values as string

1.0.62 (2018-04-10)

• promise.plugin: new recipe for python promises plugin script generation

1.0.59 (2018-03-15)

• librecipe.execute: fix convert process arguments to string formatting.

1.0.58 (2018-03-14)

• generic.mysql: unregister UDFs before (re)adding UDFs

• Remove obsolete/unused recipes.

• neoppod: add support for new –dedup storage option.

• Use inotify-simple instead of inotifyx.

• erp5.test: remove duplicated code.

• librecipe: bugfixes found by pylint, performance improvements, and major refactoring of executable wrappers.

• GenericBaseRecipe.createWrapper: remove ‘comments’ parameter.

• Drop the ‘parameters-extra’ option and always forward extra parameters.

• wrapper: new ‘private-dev-shm’ option (useful for wendelin.core).

• generic.cloudooo: OnlyOffice converter support odf.

• erp5testnode: don’t tell git to ignore SSL errors.

1.0.53 (2017-09-13)

• check_port_listening: workaround for shebang limitation, reduce to a single file

• erp5.test: pass new –conversion_server_url option to runUnitTest

1.0.52 (2017-07-04)

• wrapper: Add option to reserve CPU core

• slapconfiguration: Recipe reads partitions resource file

• neoppod: add support for new –disable-drop-partitions storage option

• random: Fix the monkeypatch in random.py to incorporate the recent changes in buildout ‘get’ function

• random: Add Integer recipe.

• librecipe.execute: Notify on file moved

• zero_knowledge: allow to set destination folder of configuration file

1.0.50 (2017-04-18)

• pbs: Do not parallelize calculus when the heaviest task is IO

• re6st-registry: Refactor integration with re6st registry

• erp5testnode: make shellinabox reusing password file of pwgen

1.0.48 (2017-01-31)

• random-recipe: add option create-once to prevent storage file deletion by buildout

1.0.45 (2017-01-09)

• recipe: set default timeout of check url promise to 20 seconds

1.0.44 (2016-12-30)

• pbs: handles the fact that some parameters are not present when slaves are down

• recipe: allow usage of pidfile in wrapper recipe

• sshd: fix generation of authorized_keys

1.0.43 (2016-11-24)

• pbs: fixes trap command for dash intepreter

• pbs: remove infinite loops from pbs scripts.

• random.py: new file containing recipes generating random values.

• testnode: disallow frontend access to all folders, avoiding publishing private repositories

1.0.41 (2016-10-26)

• dcron: new parameter to get a random time, with a frequency of once a day

• softwaretype: fix parse error on ‘+ =’ when using buildout 2

• pbs: General Improvement and fixes.

1.0.35 (2016-09-19)

• pbs: fix/accelerates deployment of resilient instances

• recipe: new recipe to get a free network port

• Remove url-list parameter to download fonts from fontconfig instance

1.0.31 (2016-05-30)

• Implement cross recipe cache for registerComputerPartition

• Fix workaround for long shebang (place script on bin)

1.0.30 (2016-05-23)

• Implement a workarround for long shebang

• Implement Validation for user inputs ssl certificates

1.0.25 (2016-04-15)

• fixup slap configuration: provide instance and root instance title

1.0.22 (2016-04-01)

• slap configuration: provide instance and root instance title

1.0.16 (2015-10.27)

• kvm recipe: fix bugs dowload image and disk creation

1.0.14 (2015-10.26)

• kvm recipe: Allow to set keyboard layout language used by qemu and VNC

• simplehttpserver-recipe: fix encoding error

For older entries, see https://lab.nexedi.com/nexedi/slapos/blob/a662db75cc840df9d4664a9d048ef28ebfff4d50/CHANGES.rst

generic_cloudooo

The generic_cloudooo recipe helps you to deploy cloudooo services with their configuration files.

How to use generic_cloudooo?

Here is an example of a section to add in your software.cfg :

[cloudooo-configuration]
recipe = slapos.cookbook:generic_cloudooo
configuration-file = ${directory:etc}/cloudooo.cfg
wrapper = ${directory:services}/cloudooo
data-directory = ${directory:srv}/cloudooo
ip = 0.0.0.0
port = 1234
ooo-paster = ${directory:bin}/cloudooo_paster
mimetype_entry_addition =
  text/html application/pdf wkhtmltopdf
openoffice-port = 1235
ooo-binary-path = ${directory:libreoffice-bin}/program
environment =
  FONTCONFIG_FILE = ${fontconfig-instance:conf-path}
  PATH = ${binary-link:target-directory}
ooo-uno-path = ${directory:libreoffice-bin}/basis-link/program

Where :

• configuration-file is the path where the put the configuration file;

• wrapper is the path where the put the final executable file;

• data-directory is the folder where cloudooo would put it’s temporary files;

• ip and port is where cloudooo will listen to;

• ooo-paster is the path of the program that will load cloudooo configuration and start the application;

• mimetype_entry_addition is additional entries to give to the default mimetype registry. (see section below.) The mimetype entry list is sorted in order to make the global mimetype at the bottom of the list. (i.e. * * ooo > text/* * ooo)

  mimetype_entry_addition =
    <input_format> <output_format> <handler>

• openoffice-port is the port where the internal OpenOffice.org service will listen to;

• ooo-binary-path is the path of the openoffice service executable file;

• environment are environment vars to use with the openoffice binary;

• ooo-uno-path is the path where UNO library is installed.

Default mimetype registry

application/vnd.oasis.opendocument* * ooo
application/vnd.sun.xml* * ooo
application/pdf text/* pdf
application/pdf * ooo
video/* * ffmpeg
audio/* * ffmpeg
application/x-shockwave-flash * ffmpeg
application/ogg * ffmpeg
application/ogv * ffmpeg
image/png image/jpeg imagemagick
image/png * ooo
image/* image/* imagemagick
text/* * ooo
application/zip * ooo
application/msword * ooo
application/vnd* * ooo
application/x-vnd* * ooo
application/postscript * ooo
application/wmf * ooo
application/csv * ooo
application/x-openoffice-gdimetafile * ooo
application/x-emf * ooo
application/emf * ooo
application/octet* * ooo
* application/vnd.oasis.opendocument* ooo

generic_varnish

This recipe creates a varnish instance dedicated for ERP5 with a web checker[1] set up.

How to Use generic_varnish ?

On slap console, you can instanciate varnish like this:

instance = request(
  software_type='varnish',
  partition_parameter_kw={
     'backend-url':'https://[your_backend_address]:your_backend_port',
     'web-checker-frontend-url':'http://www.example.com',
     'web-checker-mail-address':'[e-mail]',
     'web-checker-smtp-host':'mail.example.com',
  }
)

backend-url is the backend url that varnish will cache.

web-checker-frontend-url is the entry-point-url that web checker will check the HTTP headers of all the pages in the web site.

web-checker-mail-address is the email address where web checker will send the HTTP Cache cheking result.

web-checker-smtp-host is the smtp server to be used to send the web checker result.

[Note] When web-checker-* parameters are not given, web_checker will be disabled.

References

[1] web_checker (it is a part of erp5.util) http://pypi.python.org/pypi/erp5.util web_checker: Web site HTTP Cache header checking tool

kvm

kvm: Introduction

The erp5.recipe.kvm aims to integrate KVM setups and buildout. This recipe is able to download one remote image and setup a KVM environment to use it.

This recipe is also capable to reuse images or partitions already present on disk to create the setup.

Examples

The follow examples lists different kind of configurations.

KVM with Remote and gzipped Image

[kvm-testing-with-remote-gzip-image]
image = http://URL/public.mdv2010.0_x86_64.qcow2.img.gz

# md5 checks are optional
md5_download = adcff8adcff8adcff8adcff8
md5_image = 1a4e371a4e371a4e371a4e371a4e37

gzip = true

# Use -hda instead -drive arg
# Default is drive (see Options below)
image_type = hda

### Common Configuration below. ###

# VNC is optional
kvm_vnc = <SOME-IP>:<VNC-DISPLAY>

# Graphic is optional
kvm_graphic = std


# Define list of redirections.
kvm_redir =
  tcp:13480::80

kvm_net =
  user,hostname=publicmandriva
  nic,model=ne2k_pci

# This automatically create a redirection for 13456 -> 22
ssh_port = 13456
ssh_key_path = /path/to/mykey.key

KVM with Remote and raw Image

[kvm-testing-with-remote-raw-image]
image = http://URL/public.mdv2010.0_x86_64.qcow2.img
md5_download = 1a4e371a4e371a4e371a4e371a4e37
md5_image = 1a4e371a4e371a4e371a4e371a4e37

gzip = false

### The Rest Same configuration as previous ###

KVM with direct local Image file

This does not copy and/or download the image:

[kvm-testing-with-local-image]

file = /home/you/public.mdv2010.0_x86_64.qcow2.img
md5_image = 1a4e371a4e371a4e371a4e371a4e37

### The Rest Same configuration as previous ###

KVM with a linux partition

This does not copy and/or download the image:

[kvm-testing-a-linux-partition]

file = /dev/sdb

### The Rest Same configuration as previous ###

Options

location

When define, it does not use buildout parts directory to allocate the image.

image

URL to a remote image.

file

Use file makes recipe ignore image option. You can define a path to a image or partion to use.

image_type

You can define how the KVM will use the image with “-hdx” or “-drive”. By default it uses drive and the result is:

“kvm -drive file=IMAGE,if=virtio,cache=none,boot=on …”

if you use image_type=hda:

“kvm -hda IMAGE …”

gzip

If true means the image is compressed (gzipped), and the recipe will uncompress the image before copy it.

md5_download

When defined, this values is used to verify the file downloaded.

md5_image

When defined, this values is used to verify the image generated, don’t use it when a partition is used in file parameter.

kvm_vnc

Define the ip-address:display used by KVM to start the VNC server. If not defined, no VNC port is created.

kvm_redir

Define port redirections to the buildout.

kvm_graphic

If defined it adds the “-vga value” at the KVM command.

kvm_net

Define the net definitions, each value defines one “-net” in kvm command. Example:

kvm_net =

user,hostname=publicmandriva nic,model=ne2k_pci

It generates:

“kvm -net user,hostname=publicmandriva -net nic,model=ne2k_pci …”

kvm_snapshot

Use “-snapshot” when run a KVM. This not write the changes direct into the image. Default value is False.

ssh_port

If defined creates a new redirection for port 22 and creates few script to connect to the instance.

ssh_hostname

By default it uses localhost. You don’t need to define this.

ssh_key_path

Path to the ssh key used to connect without password to the image running.

ssh_user

Define the server that will be used to connect to the instance.

kvm_bin_directory

Place where the scripts will be created. By default it uses bin-directory from buildout.

kvm_run_directory

Place where the pid file will be created, by default it uses var-directory from buildout.

Generated Commands

Few scripts are generated to you manage your KVM instance. The scripts names are created with the followed standard:

KVM-PARTS-NAME-ctl

Commands usage

KVM-PARTS-NAME-ctl (start|stop|status|restart)

This script is used to manage the KVM instance.

KVM-PARTS-NAME-sendfile REMOTEFILE LOCALFILE

Copy the local file to a remote place.

KVM-PARTS-NAME-getfile REMOTEFILE LOCALFILE

Copy the remote file to a local place.

KVM-PARTS-NAME-runscript COMMAND

Run a command into remote KVM computer.

kvm_frontend

kvm_frontend: Introduction

The slapos.recipe.kvm_frontend aims to provide proxy server to KVM instances.

It allows HTTPS IPv4/IPv6 proxying (with or without domain name), and supports the WebSocket technology needed for VNC-in-webapplication noVNC.

It works following the master/slave instances system. A master instance is created, containing all what is needed to run the proxy. Slave instances are later created, adding one line in the master instance’s proxy configuration that specify the IP/port to proxy to the KVM. The slave instance (kvm) is then accessible from http://[masterinstanceIPorhostname]/[randomgeneratednumber]

Instance parameters

Incoming master instance parameters

• port - Port of server, optional, defaults to 4443.

• domain - domain name to use, optional, default to host.vifib.net.

• redirect_plain_http - if value is one of ['y', 'yes', '1', 'true'], will try to create a simple http server on port 80 redirecting to the proxy. Optional.

Incoming slave instance parameters

• host - KVM instance IP or hostname. Mandatory.

• port - KVM instance port, Mandatory.

• https - if value is one of ['n', 'no', '0', 'false'], will try to connect to target in plain http. Optional.

Connection parameters

Outgoing master connection parameters

• domain_ipv6_address - Proxy IP

• site_url - Proxy URL

Outgoing slave connection parameters

• site_url - URL of instance

• domain_name - Domain name of proxy

• port - Port of proxy

lamp

The lamp recipe helps you to deploy simply a php based application on slapos. This recipe is able to setup mariadb, apache and apache-php for your php application, and is also capable of configuring your software during installation to ensure a full compatibility.

How to use lamp ?

just add this part in your software.cfg to use the lamp.simple module

[instance-recipe]
egg = slapos.cookbook
module = lamp.simple

you also need to extend lamp.cfg

extends =
  https://lab.nexedi.com/nexedi/slapos/raw/slapos-0.50/stack/lamp.cfg

lamp.runner

When you install some software (such as prestashop) you need to remove or rename folder, with slapos you can not access to the www-data directory. to do this, you need to tell to lamp recipe to remove or/and it when software will be instantiated. Some software requires more than rename or delete a folder (manualy create database etc…) in this case you need to write a python script and lamp recipe must run it when installing your software.

How to use lamp.runner ?

this part of lamp recipe work with slapos.toolbox, Therefore you must add it to your recipe. in software.cfg, replace instance-recipe-egg part by

[instance-recipe-egg]
recipe = zc.recipe.egg
eggs =
    ${python-mysqlclient:egg}
    ${instance-recipe:egg}
    slapos.toolbox[lampconfigure]

and add into your instance.cfg

lampconfigure_directory = ${buildout:bin-directory}/lampconfigure

CONDITION

the action (delete, rename, script, chmod) only starts when the condition is filled. in instance.cfg, add:

file_token = path_of_file

and the action will begin when path_of_www-data/path_of_file will be created you can also use database to check condition. add

table_name = name_of_table
constraint = sql_where_condition

name_of_table is the full or partial name(in some cases we can not know the prefix used to create tables) of table into mariadb databse for example table_name = admin. if you use name_of_table = **, the action will begin when database is ready. constraint is the sql_condition to use when search entry into name_of_table for example constraint = admin_id=1

you can’t use file_token and table_name at the same time, otherwise file_token will be used in priority. Beware of conditions that will never be satisfied.

ACTION

The action starts when condition is true

1- delete file or folder into instance.cfg, use:

delete = file_or_folder1, file_or_folder2, file_or_folder3 ...

for example delete = admin

2- rename file or folder into instance.cfg, use

rename_chmod = mode (optional) rename = old_name1 => new_name1, old_name2 => new_name2, …

you can also use rename = old_name1, old_name2 => new_name2, … in this case old_name1 will be rename and the new name will be chose by joining old_name1 and mysql_user: this should give rename = old_name1 => old_name1-mysql_user, old_name2 => new_name2, … use rename_chmod to apply a mode to new_name1, new_name2, …

3- Change mode of file or directory

this is usefull if you want to change the mode of files into your www-data, use

chmod = file1, file2, …. mode = mode_to_apply (ex= 0644)

4- Launch python script

use script = ${configure-script:location}/${configure-script:filename} into instance.cfg, add part configure-script into software.cfg

parts = configure-script

[configure-script]
recipe = hexagonit.recipe.download
location = ${buildout:parts-directory}/${:_buildout_section_name_}
url = url_of_script_name.py
filename = script_name.py
download-only = True

The script_name.py should contain a main function, sys.argv is given to the main. you can write script_name.py like this:

...
def setup(args):
  mysql_port, mysql_host, mysql_user, mysql_password, mysql_database, base_url, htdocs = args
  .......

if __name__ == '__main__':
    setup(sys.argv[1:])

• base_url: is the url of php software

• htdocs: is the path of www-data directory

• mysql_user, mysql_password, mysql_database, mysql_host: is the mariadb parameters

you can also use “import MySQLdb” if you want to access to database via your python script

libcloud

Slapified recipe to interact with any libcloud supported IaaS system

librecipe

Thanks to using slapos.cookbook:librecipe it is easier to create zc.buildout recipes in SlapOS environment.

How to use?

In setup.py of recipe add only one install requires to slap.lib.recipe.

In code itself subclass from slap.lib.recipe.BaseSlapRecipe.BaseSlapRecipe.

Use _install hook:

from slap.lib.recipe.BaseSlapRecipe import BaseSlapRecipe

class Recipe(BaseSlapRecipe):
  ...
  def _install(self):
    # refer below for list of available objects
    specific code
    of recipe

Available variables self.:

• name and options passed by zc.buildout during init

• work_directory – buildout’s directory

• bin_directory – places for generated binaries

• running_wrapper_location – filename of wrapper to create

• data_root_directory – directory container for data – inside this directory it is advised to create named directories for provided servers which needs data

• backup_directory – directory container for backups – inside this directory it is advised o created named directories for backups, with same structure as in data_root_directory

• var_directory – container for various, unix following things:

  • log_directory – container for logs

  • run_directory – container for pidfiles and sockets

• etc_directory – place to put named files and directories of configuration for provided servers

• computer_id – id of computer

• computer_partition_id – if of computer partition

• server_url - url of Vifib server

• software_release_url – url of software release being instantiated

• slap – initialised connection to Vifib server

• computer_partition – initialised connection to computer partition

• request – shortcut to computer partition request method

By default all directories are created before calling _install hook.

_install method shall return list of paths which are safe to be removed by buildout during part uninstallation.

Important assumptions

Because in SlapOS environment zc.buildout does not know when data are changed, recipes shall be always uninstalled/installed. This is done during constructing recipe instance which subclasses from BaseSlapRecipe.

mkdirectory

mkdirectory loops on its options and create the directory joined

Note

Use a slash / as directory separator. Don’t use system dependent separator. The slash will be parsed and replace by the operating system right separator.

Only use relative directory to the buildout root directory.

The created directory won’t be added to path list.

These recipes provide the ability to save some buildout parameters and their value in a custom file, inside the instance folder.

In both recipes, you HAVE TO give a filename, which will be stored at the root of the instance folder

WriteRecipe :

• Is used to create a section (named according to the buildout section_name).

• You can give then as much parameters you wish, with their default values.

• Whenever you run buildout, if the parameter has yet been saved in the config file, it will do nothing.

• If the parameter’s value has changed in the config file, it won’t be overwritten

• /!If you decide to change the default value of one parameter, ALL other parameters will be reseted in the config file, even if you changed it manually. Explanation : The default values aren’t expected to change, except while development purposes.

ReadRecipe :

• It fills its own section with all the options in all the sections of the config file.