Distributed job execution application built for Galaxy (http://galaxyproject.org/).
This project is a Python server application that allows a Galaxy server to run jobs on remote systems (including Windows) without requiring a shared mounted file systems. Unlike traditional Galaxy job runners - input files, scripts, and config files may be transferred to the remote system, the job is executed, and the results are transferred back to the Galaxy server - eliminating the need for a shared file system.
Full documentation for the project can be found on Read The Docs.
Galaxy job runners are configured in Galaxy’s job_conf.xml file. Some small examples of how to configure this can be found here, but be sure to check out job_conf.xml.sample_advanced in your Galaxy code base or on Github for complete information.
Full details on different ways to install Pulsar can be found in the install section of the documentaiton, but if your machine has the proper Python dependencies available it can be quickly download and a test job run with:
$ mkdir pulsar $ cd pulsar $ python3 -m venv venv $ . venv/bin/activate # venv\Scripts\activate.bat on Windows $ pip install 'pulsar-app[web]' $ pulsar-config $ pulsar --daemon # just `pulsar` on Windows $ pulsar-check # runs a test job
Please note that as of the 0.14.0 release, Pulsar no longer supports any version of Python 2. The minimum supported Python version is 3.5.
The configuration documentation has many details on securing your Pulsar server and enabling advanced features such as cluster integration and message queue communication.
Development and Testing
The recommended approach to setting up a development environment for Pulsar on Linux or macOS is rougly as follows:
$ git clone https://github.com/galaxyproject/pulsar $ cd pulsar $ python3 -m venv .venv $ . .venv/bin/activate # .venv\Scripts\activate on Windows $ pip install -e '.[web]' $ pip install -r dev-requirements.txt
The -e flag to pip installs Pulsar in “editable” mode, meaning that changes you make to the source code will be reflected when running the pulsar commands installed in the virtual environment.
This project is distributed with unit and integration tests (many of which will not run under Windows), the following command will install the needed python components to run these tests. The following command will then run these tests:
$ make tests
The following command will then produce a coverage report corresponding to this test and place it in the coverage_html_report subdirectory of this project.:
$ coverage html
Check out the Contributing documentation for many more details on developing and contributing to Pulsar.
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
This documentation is an incomplete work in progress. There are more ways to configure and use Pulsar than are documented, and a growing number of Pulsar experts who would be more than happy to answer your questions and help with any problems you may run in to while setting up a Pulsar deployment. Please do not hesitate to reach out on the Galaxy Admins Gitter Channel
No changes, working around pypi isssue.
Updated Galaxy+Pulsar container. Pull Request 306
Rework container execution - generalize Kubernetes execution to allow it to work without a message queue and to allow TES execution based on pydantic-tes (https://github.com/jmchilton/pydantic-tes). Pull Request 302
Add documentation and diagrams for container execution scenarios. Pull Request 302
Rework integration tests to use pytest more aggressively.
Fixes to CI to run more tests that weren’t being executed because Tox was not sending environment variables through to pytest.
Add option amqp_key_prefix to direct task queue naming while retaining simple default manager names and such in container scheduling deployments. Pull Request 315
Fixes for extra_file handling. Pull Request 318
Separate tool_stdio and job_stdio handling. Pull Request 318
Re-import MEMORY_STATEMENT.sh from Galaxy. Pull Request 297
Add support for logging to sentry. Pull Request 322
Fix small regression related to building URLs for client action mapping.
Fix small regressions bugs in 0.14.14 - updating runner util code was bigger swap over than it seemed.
Bring in updated Galaxy runner util code. Pull Request 303
Fixes to bring HOME and temp directory handling closer to Galaxy native runners.
Enable globbed from_work_dir outputs for remote metadata.
Don’t error out if annotated galaxy.json is absent. Pull Request 270
Fix for newer Galaxy tool profiles having isolated home directories.
Potential fix for setting file actions via job destination parameters.
Re-attempt release process - published wrong branch with 0.14.3.
Allow transferring fewer files from Pulsar when using extended metadata with Galaxy.
Fix the use of requests, limits, and walltime with coexecution pods. Pull Request 246
Fix the use of named managers with coexecution pods. Pull Request 242
Set the DRMAA workingDirectory to the job’s working directory Pull Request 230
Respond to MQ messages requesting status updates. Pull Request 228
Drop Python 2.7 support in standard transport, drop Python 2.7 tests and fix Python 3.7 wheel install test, general test debugging enhancements. Pull Request 231
Request and register ports for Galaxy ITs when using Kubernetes. Pull Request 223
Implement killing k8s jobs. Pull Request 221
Respond to MQ messages requesting status updates. Pull Request 228
Debug connection string for AMQP. Pull Request 217
Improvements and fixes to container handling. Pull Request 202
Fix a typo in exception logging thanks to @erasche. Pull Request 203
Cleanup config file handling a bit by removing branch for very old Pulsar servers likely no longer supported. Pull Request 201
Pinned all listed requirements. This is the final version of Pulsar to support Python 2.
Various improvements and simplifications to Kubernetes job execution.
Retry botched release that didn’t include all relevant commits.
Revise Python Galaxy dependencies to use newer style Galaxy decomposition. galaxy-lib can no longer be installed in Pulsar’s environment, so you will likely need to rebuild your Pulsar virtualenv for this release. Pull Request 187
Various small improvements to Kubernetes pod execution environment. Pull Request 190
Improve readme linting. Pull Request 186
Implement staging Galaxy metadata input files in the client. 39de377
Refactor toward more structured inputs. f477bc4
Refactor toward passing objectstore identifying information around. Pull Request 180
Rework imports for new Galaxy library structure. da086c9
Revert empty input testing, it really probably should cause a failure to transfer a non-existent file. 8bd5511
Better client mapper documentation. b6278b4
Add configuration parameter to limit stream size read from disk. Pull Request 157
Pass full job status for failed and lost jobs. Pull Request 159
Improve message handling if problems occur during job setup/staging. Pull Request 160
Rework preprocessing job state to improve restartability and reduce job loss. This change should be applied while no jobs are running. Pull Request 164
Add support for overriding config through environment variables (thanks to @nuwang). Pull Request 165
Minor docs updates (thanks to @afgane). Pull Request 170
Create universal wheels to enable Python 3 support when installing from PyPI (thanks to @nsoranzo). Pull Request 156
Support new features in Galaxy job running/scripting so that Pulsar respects $GALAXY_VIRTUAL_ENV and $PRESERVE_GALAXY_ENVIRONMENT. Fix remote metadata in cases where the tool environment changes the python on $PATH. Pull Request 137
Precreate Galaxy tool outputs on the remote before executing (fixes a bug related to missing output files on stage out). Pull Request 141
Support the remote_transfer file action without setting the jobs_directory destination param Pull Request 136
Fix invalid character in job managers documentation (thanks to @mapa17). Pull Request 130
Fix conda_auto_* option resolution and include a sample dependency_resolvers_conf.xml (thanks to @mapa17). Pull Request 132
Fix a bug with AMQP acknowledgement. Pull Request 143
Fix bug causing loops on in response to preprocessing error conditions.
Do a release to circumvent a tool version logic error in Galaxy ( released Galaxy versions think 0.7.0 < 0.7.0.dev3).
Update Makefile to allow release pulsar as an application and a library for Galaxy at the same time.
Small update to test scripts for TravisCI changes.
Improvements for embedded Galaxy runner. (TODO: fill this out)
Remove support for Python 2.6. 60bf962
Updated cluster slots detection for SLURM from Galaxy. cadfc5a
Various changes to allow usage within Galaxy as a library. ce9d4f9
Introduce a separate working and metadata directory as required for Galaxy 16.04 that requires this separation. 6f4328e
Add Tox target for Python 2.7 unit testing. d7c524e
Add Makefile command for setup.py develop. fd82d00
Tweak release process that left 0.6.0 with an incorrect PyPI description page.
Pulsar now depends on the new galaxy-lib Python package instead of manually synchronizing Python files across Pulsar and Galaxy.
Numerous build and testing improvements.
Fixed a documentation bug in the code (thanks to @erasche). e8814ae
Remove galaxy.eggs stuff from Pulsar client (thanks to @natefoo). 00197f2
Add new logo to README (thanks to @martenson). abbba40
Fix project script bug introduced this cycle (thanks to @nsoranzo). 140a069
Fix config.py on Windows (thanks to @ssorgatem). Pull Request 84
Add a job manager for XSEDE jobs (thanks to @natefoo). 1017bc5
Fix pip dependency installation (thanks to @afgane) Pull Request 73
Allow cURL downloader to resume transfers during staging in (thanks to @natefoo). 0c61bd9
Fix to cURL downloaders status code handling (thanks to @natefoo). 86f95ce
Fix non-wheel installs from PyPI. Issue 72
Fix mesos imports for newer versions of mesos (thanks to @kellrott). fe3e919
Changed the name of project to Pulsar, moved to Github.
New RESTful web services interface.
SCP and Rsync file staging options added by E. Rasche. Pull Request
Allow YAML based configuration.
Support for more traditional pip/setup.py-style installs.
Dozens of smaller bugfixes and documentation updates.
Last version named the LWR - found on BitBucket.
Still supported in Galaxy as of 15.03 the release.
Introduced support for submitting to various queueing systems, operation as a Mesos framework, Docker support, and various other advanced deployment options.
Message queue support.
Framework for configurable file actions introduced.
Simple support for running jobs managed by the Python LWR web process.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for pulsar_app-0.15.5-py2.py3-none-any.whl