Skip to main content

A Django application to manage long running job submission, including starting the job, saving logs, and storing results.

Project description

Django Remote Submission

https://badge.fury.io/py/django-remote-submission.png https://travis-ci.org/ornl-ndav/django-remote-submission.png?branch=master https://codecov.io/gh/ornl-ndav/django-remote-submission/branch/master/graph/badge.svg

A Django application to manage long running job submission, including starting the job, saving logs, and storing results.

Documentation

The full documentation is at https://django-remote-submission.readthedocs.org.

Quickstart

Install Django Remote Submission:

pip install django-remote-submission

Then use it in a project:

from django_remote_submission.models import Server, Job
from django_remote_submission.tasks import submit_job_to_server

server = Server.objects.get_or_create(
    title='My Server Title',
    hostname='example.com',
    port=22,
)[0]

python2_interpreter = Interpreter.objects.get_or_create(
    name = 'python2',
    path = '/usr/bin/python2.7 -u',
)[0]

python3_interpreter = Interpreter.objects.get_or_create(
    name = 'python3',
    path = '/usr/bin/python3.5 -u',
)[0]

server.interpreters.set([python2_interpreter,
                         python3_interpreter])

job = Job.objects.get_or_create(
    title='My Job Title',
    program='print("hello world")',
    remote_directory='/tmp/',
    remote_filename='test.py',
    owner=request.user,
    server=server,
    interpreter=python2_interpreter,
)[0]

# Using delay calls celery:
modified_files = submit_job_to_server.delay(
    job_pk=job.pk,
    password=request.POST.get('password'),
)

To avoid storing the password one can deploy the client public key in the server.

from django_remote_submission.tasks import copy_key_to_server

copy_key_to_server(
    username=env.remote_user,
    password=env.remote_password,
    hostname=env.server_hostname,
    port=env.server_port,
    public_key_filename=None, # finds it automaticaly
)

And it can be deleted once the session is finished:

from django_remote_submission.tasks import delete_key_from_server

delete_key_from_server(
    username=env.remote_user,
    password=env.remote_password,
    hostname=env.server_hostname,
    port=env.server_port,
    public_key_filename=None,
)

Features

  • Able to connect to any server via password-authenticated SSH.

  • Able to receive logs and write them to a database in realtime.

  • Able to return any modified files from the remote server.

  • Uses Server Side Events (SSE) to notify the Web Client the Job status

  • Uses WebSockets / SSE to provide Job Log in real time to a Web Client

Running Tests

Does the code actually work?

source <YOURVIRTUALENV>/bin/activate
(myenv) $ pip install -r requirements_test.txt
(myenv) $ make test

Some of the tests use a test server to check the functional aspects of the library. Specifically, it will try to connect to the server multiple times, run some programs, and check that their output is correct.

To run those tests as well, copy the .env.base file to .env and modify the variables as needed. If this file has not been set up, then those tests will be skipped, but it won’t affect the success or failure of the tests.

Running tests independtely, e.g.:

pytest -v tests/test_models.py
pytest -v tests/test_models.py::test_server_string_representation

Running the Example

Launch Redis:

redis-server

Launch Celery:

cd example
celery -A server.celery worker --loglevel=info

Launch Django:

cd example
# Note that there's a requirements file in this folder!
# Install it in a virtual environment!
./manage.py makemigrations
./manage.py migrate
./manage.py loaddata fixtures/initial_data.json
# You may want to create a user:
# python manage.py createsuperuser
./manage.py runserver

Open in the browser one of the links below. The password for admin is admin123:

# For the Admin Interface
http://localhost:8000/admin/
# For the REST API
http://localhost:8000/
# To test Job creation with live status update
http://127.0.0.1:8000/example/

Web Interface

The app provides two web sockets (SSE) to see in real time the Job Status and the Log associated to a Job.

Those are defined in routing.py:

path=r'^/job-user/$'
path=r'^/job-log/(?P<job_pk>[0-9]+)/$'

The example app comes with the Live Job Status and Live Log examples. See:

# Jobs
http://127.0.0.1:8000/example/
# Job 123 Log
http://127.0.0.1:8000/logs/123/

Both files:

django-remote-submission/example/templates/example_job_status.html
django-remote-submission/example/templates/example_job_log.html

Have the client side web socket code to interact with the django-remote-submission app. Also to include the Live information on a web app it is worth looking at the celery configuration: /home/rhf/git/django-remote-submission/example/server/celery.py and the WebSockets rooting: /home/rhf/git/django-remote-submission/example/server/routing.py

Useful notes

The Results files are stored in MEDIA. So add to your setings something similar to:

MEDIA_URL = '/media/'
MEDIA_ROOT = '../dist/media'

To make media available in DEBUG mode, you might want to add to the main urls.py:

if settings.DEBUG:
    # Serving files uploaded by a user during development
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Credits

Tools used in rendering this package:

History

0.2.0 (2016-11-17)

  • Add django admin interface.

  • Add migrations folder.

  • Add log policies for submitting tasks.

  • Add return value for modified files.

0.1.1 (2016-11-15)

  • Add port number to Server model.

  • Add task to submit jobs.

  • Add status updates to task.

  • Fix unicode error when submitting jobs.

  • Fix verbose/related names for models.

0.1.0 (2016-11-08)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django-remote-submission-0.11.0.tar.gz (21.5 kB view hashes)

Uploaded Source

Built Distribution

django_remote_submission-0.11.0-py2.py3-none-any.whl (28.4 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page