A Django application to manage long running job submission, including starting the job, saving logs, and storing results.
Project description
Django Remote Submission
A Django application to manage long running job submission, including starting the job, saving logs, and storing results.
Documentation
The full documentation is at https://django-remote-submission.readthedocs.org.
Quickstart
Install Django Remote Submission:
pip install django-remote-submission
Then use it in a project:
from django_remote_submission.models import Server, Job from django_remote_submission.tasks import submit_job_to_server server = Server.objects.get_or_create( title='My Server Title', hostname='example.com', port=22, )[0] python2_interpreter = Interpreter.objects.get_or_create( name = 'python2', path = '/usr/bin/python2.7 -u', )[0] python3_interpreter = Interpreter.objects.get_or_create( name = 'python3', path = '/usr/bin/python3.5 -u', )[0] server.interpreters.set([python2_interpreter, python3_interpreter]) job = Job.objects.get_or_create( title='My Job Title', program='print("hello world")', remote_directory='/tmp/', remote_filename='test.py', owner=request.user, server=server, interpreter=python2_interpreter, )[0] modified_files = submit_job_to_server( job_pk=job.pk, password=request.POST.get('password'), )
Features
Able to connect to any server via password-authenticated SSH.
Able to receive logs and write them to a database in realtime.
Able to return any modified files from the remote server.
Running Tests
Does the code actually work?
source <YOURVIRTUALENV>/bin/activate (myenv) $ pip install -r requirements_test.txt (myenv) $ make test
Some of the tests use a test server to check the functional aspects of the library. Specifically, it will try to connect to the server multiple times, run some programs, and check that their output is correct.
To run those tests as well, copy the .env.base file to .env and modify the variables as needed. If this file has not been set up, then those tests will be skipped, but it won’t affect the success or failure of the tests.
Running tests independtely, e.g.:
python runtests.py tests.test_tasks.SubmitJobTaskTest.test_normal_usage python runtests.py tests.test_tasks.SubmitJobTaskTest python runtests.py tests.test_tasks python runtests.py tests.test_models
Running the Example
Launch Redis:
redis-server
Launch Celery:
cd example celery -A server.celery worker --loglevel=info
Launch Django:
cd example PYTHONPATH=../ ./manage.py makemigrations PYTHONPATH=../ ./manage.py migrate PYTHONPATH=../ ./manage.py loaddata fixtures/initial_data.json PYTHONPATH=../ ./manage.py runserver
Open in the browser:
http://localhost:8000/admin/ http://localhost:8000/
Useful notes
When integrating this in django it might be usefull:
Add to settings:
CELERY_ACCEPT_CONTENT = ['pickle'] CELERY_EVENT_SERIALIZER = 'pickle' CELERY_TASK_SERIALIZER = 'pickle' CELERY_RESULT_SERIALIZER = 'pickle'
if using the app 'django_celery_results. Otherwise Result is not serialized.
The Results files are stored in MEDIA. So add to your setings something similar to:
MEDIA_URL = '/media/' MEDIA_ROOT = '../dist/media'
To make media available in DEBUG mode, you might want to add to the main urls.py:
if settings.DEBUG: # Serving files uploaded by a user during development urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
Credits
Tools used in rendering this package:
History
0.2.0 (2016-11-17)
Add django admin interface.
Add migrations folder.
Add log policies for submitting tasks.
Add return value for modified files.
0.1.1 (2016-11-15)
Add port number to Server model.
Add task to submit jobs.
Add status updates to task.
Fix unicode error when submitting jobs.
Fix verbose/related names for models.
0.1.0 (2016-11-08)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for django-remote-submission-0.8.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8caf2b2f82bb1b7b8adc52bad3200a3c31f352deba58f5361f6c03147d46da21 |
|
MD5 | 96e63182398d72221a440e360c15583d |
|
BLAKE2b-256 | eaddba727ce391b70fc4c92776e27e15040b05925f6ad9bb42bc2aa153ce53bb |
Hashes for django_remote_submission-0.8.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c59f2edc08e42136d7640214450a3a525225722db3adb063657c3c0484a01eb8 |
|
MD5 | a7ff3fc83eeb371ed54de51d3ac60126 |
|
BLAKE2b-256 | fab6ccc113ec1c7d4589f09c5e3c533df37ca145edd33fcaa70d8fc00debc80d |