Skip to main content

Internal tools to use the SGE grid at Idiap Research Institute

Project description

GridSearch-like on the grid

Searching good combinaisons of parameters happen quite ofter in ML-related task. I wrote a tool that help to achieve that. If you worked with the unix system call fork, you will see a similar approach with the API.

The only requirement is jinja2

Hello World example

In this example we wrote all combinaisons of hello/hi world/Idiap.

from easy_grid.experiences_manager import ExperienceBase

if __name__ == '__main__':
    if ExperienceBase.is_in_SGE(): # Write your task
        parameters = ExperienceBase.load_only_current()
        first = parameters['first']
        second = parameters['second']
        print('{} {}'.format(first, second))
    else: # Configure your task
        my_experience = ExperienceBase(python_executable='python3')
        my_experience.add_experience_key_values('first', ['hello', 'hi'])
        my_experience.add_experience_key_values('second', ['world', 'idiap'])

        my_experience.add_grid_parameter('-cwd', '')
        my_experience.add_grid_parameter('-N', 'hello_world')

        my_experience.add_export_parameter('PYTHONPATH', '.:$PYTHONPATH')

        my_experience.run()

Run the example python3 -m examples.hello_world then check the results after running the script (and waiting a little bit) ls -1 output/hello_world.o*

output/hello_world.o5439319.1
output/hello_world.o5439319.2
output/hello_world.o5439319.3
output/hello_world.o5439319.4

Each file contains the result of a combinaison. Let's check the content of the first one cat output/hello_world.o5439319.1

hello world

Get all results at once cat output/hello_world.o*

hello world
hi world
hello idiap
hi idiap

FizzBuzz example

We can also use task over a single parameter, like in fizzbuzz, which depend of n. Here an example with fizzbuzz from [1, 100]

from easy_grid.experiences_manager import ExperienceBase

if __name__ == '__main__':
    if ExperienceBase.is_in_SGE(): # Write your task
        parameters = ExperienceBase.load_only_current()
        n = parameters['n']
        my_str = ''
        if n % 3 == 0: 
            my_str += 'Fizz'
        if n % 5 == 0: 
            my_str += 'Buzz'
        if my_str == '': 
            my_str = str(n)
        print('(n={:04d}) -> {}'.format(n, my_str))
    else: # Configure your task
        my_experience = ExperienceBase(python_executable='python3')
        my_experience.add_experience_key_values('n', list(range(1, 101)))

        my_experience.add_grid_parameter('-cwd', '')
        my_experience.add_grid_parameter('-N', 'fizzbuzz')

        my_experience.add_export_parameter('PYTHONPATH', '.:$PYTHONPATH')

        my_experience.run()

Run the example python3 -m examples.fizzbuzz

Get all results at once and sort by n cat output/fizzbuzz.o* | sort

(n=0001) -> 1
(n=0002) -> 2
(n=0003) -> Fizz
(n=0004) -> 4
(n=0005) -> Buzz
(n=0006) -> Fizz
(n=0007) -> 7
(n=0008) -> 8
(n=0009) -> Fizz
(n=0010) -> Buzz
(n=0011) -> 11
...
(n=0090) -> FizzBuzz
(n=0091) -> 91
(n=0092) -> 92
(n=0093) -> Fizz
(n=0094) -> 94
(n=0095) -> Buzz
(n=0096) -> Fizz
(n=0097) -> 97
(n=0098) -> 98
(n=0099) -> Fizz
(n=0100) -> Buzz

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for easy-grid-wdroz, version 0.0.5
Filename, size File type Python version Upload date Hashes
Filename, size easy_grid_wdroz-0.0.5-py3-none-any.whl (5.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size easy_grid-wdroz-0.0.5.tar.gz (5.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page