python-grid5000 0.0.5

No project description provided

===============
python-grid5000
===============


``python-grid5000`` is a python package wrapping the Grid’5000 REST API.

.. warning::

The code is currently being developed heavily. Jump to the contributing section
if you want to be involved.

1 Thanks
--------

The core code is borrowed from `python-gitlab <https://github.com/python-gitlab/python-gitlab>`_ with small adaptations to
conform with the Grid5000 API models (with an ’s’!)

2 Contributing
--------------

- To contribute, you can drop me an email or open an issue for a bug report, or feature request.

- There are many areas where this can be improved some of them are listed here:

- The complete coverage of the API isn’t finished (yet) but this should be fairly easy to reach.
Most of the logic go in ```grid5000.objects`` <https://gitlab.inria.fr/msimonin/python-grid5000/blob/master/grid5000/objects.py>`_. And to be honnest I only
implemented the feature that I needed the most.

- Returned `status code <https://www.grid5000.fr/mediawiki/index.php/API#Status_Codes>`_ aren’t yet well treated.

3 Comparison with ...
---------------------

- `RESTfully <https://api.grid5000.fr/doc/4.0/tools/restfully.html>`_:
It consumes REST API following the `HATEOAS <https://en.m.wikipedia.org/wiki/HATEOAS>`_ principles. This allows the client
to fully discover the resources and actions available. Most of the G5K API
follow theses principles but, for instance the `Storage API <https://www.grid5000.fr/mediawiki/index.php/Storage_Manager>`_ don’t. Thus
RESTfully isn’t compatible with all the features offered by the Grid’5000 API.
It’s a ruby library. Python-grid5000 borrows the friendly syntax for resource
browsing, but in python.

- `Execo <http://execo.gforge.inria.fr>`_:
Written in Python. The api module gathers a lot of utils functions leveraging
the Grid’5000 API. Resources aren’t exposed in a syntax friendly manner,
instead functions for some classical operations are exposed (mainly getters).
It has a convenient way of caching the reference API. Python-grid5000 is a
wrapper around the Grid’5000 that seeks 100% coverage. Python-grid5000 is
resource oriented.

- `Raw requests <http://docs.python-requests.org>`_:
**The** reference for HTTP library in python. Good for prototyping but low-level.
python-grid5000 encapsulates this library.

4 Installation and examples
---------------------------

- Please refer to `https://api.grid5000.fr/doc/4.0/reference/spec.html <https://api.grid5000.fr/doc/4.0/reference/spec.html>`_ for
the complete specification.

- All the examples are exported in the examples subdirectory so you can
easily test and adapt them.

- The configuration is read from a configuration file located in the home
directory (should be compatible with the restfully one).
It can be created with the following:

::

echo '
username: MYLOGIN
password: MYPASSWORD
' > ~/.python-grid5000.yaml

.. info::

Inside Grid’5000, you’l probably need to set ``verify_ssl: False``

- Using a virtualenv is recommended (python 3.5+ is required)

::

virtualenv -p python3 venv
source venv/bin/activate
pip install python-grid5000

4.1 Get node information
~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: python

import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

node_info = gk.sites["nancy"].clusters["grisou"].nodes["grisou-1"]
print("grisou-1 has {threads} threads and has {ram} bytes of RAM".format(
threads=node_info.architecture["nb_threads"],
ram=node_info.main_memory["ram_size"]))

4.2 Job filtering
~~~~~~~~~~~~~~~~~

.. code:: python

import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

# state=running will be placed in the query params
running_jobs = gk.sites["rennes"].jobs.list(state="running")
print(running_jobs)

# get a specific job by its uid
job = gk.sites["rennes"].jobs.get("424242")
print(job)
# or using the bracket notation
job = gk.sites["rennes"].jobs["424242"]
print(job)

4.3 Submit a job
~~~~~~~~~~~~~~~~

.. code:: python

import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

# This is equivalent to gk.sites.get("rennes")
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
"command": "sleep 3600"})

while job.state != "running":
job.refresh()
print("Waiting for the job [%s] to be running" % job.uid)
time.sleep(10)

print(job)
print("Assigned nodes : %s" % job.assigned_nodes)

4.4 Deploy an environment
~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: python

import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

# This is equivalent to gk.sites.get("rennes")
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
"command": "sleep 3600",
"types": ["deploy"]})

while job.state != "running":
job.refresh()
print("Waiting the job [%s] to be running" % job.uid)
time.sleep(10)

print("Assigned nodes : %s" % job.assigned_nodes)

deployment = site.deployments.create({"nodes": job.assigned_nodes,
"environment": "debian9-x64-min"})

while deployment.status != "terminated":
deployment.refresh()
print("Waiting for the deployment [%s] to be finished" % deployment.uid)
time.sleep(10)

print(deployment.result)

4.5 Get Storage accesses
~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: python

import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

print(gk.sites["rennes"].storage["msimonin"].access.list())

4.6 Set storage accesses (e.g for vms)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: python

from netaddr import IPNetwork
import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
"command": "sleep 3600",
"resources": "slash_22=1+nodes=1"})

while job.state != "running":
job.refresh()
print("Waiting the job [%s] to be running" % job.uid)
time.sleep(5)

subnet = job.resources_by_type['subnets'][0]
ip_network = [str(ip) for ip in IPNetwork(subnet)]

# create acces for all ips in the subnet
access = site.storage["msimonin"].access.create({"ipv4": ip_network,
"termination": {"job": job.uid,
"site": site.uid}})

4.7 Get vlan(s)
~~~~~~~~~~~~~~~

.. code:: python

import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

site = gk.sites["rennes"]

# Get all vlans
vlans = site.vlans.list()
print(vlans)

# Get on specific
vlan = site.vlans.get("4")
print(vlan)

vlan = site.vlans["4"]
print(vlan)

# Get vlan of some nodes
print(site.vlansnodes.submit({"nodes": ["paravance-1.rennes.grid5000.fr", "paravance-2.rennes.grid5000.fr"]}))


# Get nodes in vlan
print(site.vlans["4"].nodes.list())

4.8 Set nodes in vlan
~~~~~~~~~~~~~~~~~~~~~

- Putting primary interface in a vlan

.. code:: python

import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)
site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
"command": "sleep 3600",
"resources": "{type='kavlan'}/vlan=1+nodes=1",
"types": ["deploy"]})

while job.state != "running":
job.refresh()
print("Waiting the job [%s] to be runnning" % job.uid)
time.sleep(5)

deployment = site.deployments.create({"nodes": job.assigned_nodes,
"environment": "debian9-x64-min",
"vlan": job.resources_by_type["vlans"][0]})

while deployment.status != "terminated":
deployment.refresh()
print("Waiting for the deployment [%s] to be finished" % deployment.uid)
time.sleep(10)

print(deployment.result)

- Putting the secondary interface in a vlan

.. code:: python

import logging
import os
import time

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)


def _to_network_address(host, interface):
"""Translate a host to a network address
e.g:
paranoia-20.rennes.grid5000.fr -> paranoia-20-eth2.rennes.grid5000.fr
"""
splitted = host.split('.')
splitted[0] = splitted[0] + "-" + interface

return ".".join(splitted)


conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

site = gk.sites["rennes"]

job = site.jobs.create({"name": "pyg5k",
"command": "sleep 3600",
"resources": "{type='kavlan'}/vlan=1+{cluster='paranoia'}nodes=1",
"types": ["deploy"]
})

while job.state != "running":
job.refresh()
print("Waiting the job [%s] to be runnning" % job.uid)
time.sleep(5)

vlanid = job.resources_by_type["vlans"][0]

# we hard code the interface but this can be discovered in the node info
# TODO: write the code here to discover
nodes = [_to_network_address(n, "eth2") for n in job.assigned_nodes]
print(nodes)

# set in vlan
site.vlans[vlanid].submit({"nodes": nodes})

4.9 Site of a cluster
~~~~~~~~~~~~~~~~~~~~~

.. code:: python

import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

clusters = ["dahu", "parasilo", "chetemi"]

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

sites = gk.sites.list()
matches = []
for site in sites:
candidates = site.clusters.list()
matching = [c.uid for c in candidates if c.uid in clusters]
if len(matching) == 1:
matches.append((site, matching[0]))
clusters.remove(matching[0])
print("We found the following matches %s" % matches)

4.10 Get all job with a given name on all the sites
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: python


import logging
import os

from grid5000 import Grid5000


logging.basicConfig(level=logging.DEBUG)

NAME = "pyg5k"

conf_file = os.path.join(os.environ.get("HOME"), ".python-grid5000.yaml")
gk = Grid5000.from_yaml(conf_file)

sites = gk.sites.list()
site = gk.sites["rennes"]
sites = [gk.sites["rennes"], gk.sites["nancy"], gk.sites["grenoble"]]

# creates some jobs
jobs = []
for site in sites:
job = site.jobs.create({"name": "pyg5k",
"command": "sleep 3600"})
jobs.append(job)

_jobs = []
for site in sites:
_jobs.append((site.uid, site.jobs.list(name=NAME,
state="waiting,launching,running")))

print("We found %s" % _jobs)

# deleting the jobs
for job in jobs:
job.delete()