planemo 0.20.0

Command-line utilities to assist in building tools for the Galaxy project (http://galaxyproject.org/).

Command-line utilities to assist in building and publishing Galaxy tools.

• Free software: Academic Free License version 3.0

• Documentation: https://planemo.readthedocs.org.

• Code: https://github.com/galaxyproject/planemo

Quick Start

This quick start demonstrates using planemo commands to help develop Galaxy tools. Planemo can quickly be installed via Homebrew or as a more traditional Python project.

To install using Homebrew or linuxbrew:

brew tap galaxyproject/tap
brew install planemo

For a more traditional Python installation simply setup a virtualenv for planemo (this example creates a new one in .venv) and then install with pip.

% virtualenv .venv; . .venv/bin/activate
% pip install planemo

Planemo is also available a virtual appliance for Docker or Vagrant (bundled with a preconfigured Galaxy server optimized for tool development).

This quick start will assume you will have a directory with one or more tool XML files. If none is available, one can be quickly create for demonstrating planemo as follows mkdir mytools; cd mytools; planemo project_init --template=demo.

Planemo can check tools containing XML for common problems and best practices using the lint command (also aliased as l).

% planemo lint
...

Like many planemo commands - by default this will search the current directory and use all tool files it finds. It can be explicitly passed other tool files or a directory of tool files.

% planemo l randomlines.xml

The lint command takes in a additional options related to reporting levels, exit code, etc…. These options are descirbed here or (like all available commands) be accessed by passing it --help.:

% planemo l --help
Usage: planemo lint [OPTIONS] TOOL_PATH
...

Once tools are syntically correct - it is time to test. The test command can be used to test a tool or directory of tools.:

% planemo test --galaxy_root=../galaxy-central randomlines.xml

If no --galaxy_root is defined, planemo will check for a default in ~/.planemo.yml) and finally search the tool’s parent directories for a Galaxy root directory (developing tools under Galaxy tools directory is a common development workflow). Planemo can also download and configure a disposable Galaxy instance just for testing by passing it -install_galaxy instead of a Galaxy root.:

% planemo t --install_galaxy

Warning: The features of Planemo that require a --galaxy_root will only work with Galaxy releases from 2015.

Planemo will create a HTML an output report in the current directory named tool_test_output.html (override with --test_output). Here is an example of such a report for Tophat.

Once tools have been linted and tested - the tools can be viewed in a Galaxy interface using the serve (s) command.:

% planemo serve

Like test, serve requires a Galaxy root and one can be explicitly specified with --galaxy_root or installed dynamically with --install_galaxy.

Experimental Features

Planemo can also be used to explore some more experimental features related to Galaxy tooling - including support for Travis CI, Docker, Homebrew.

Tool Shed

For information on using Planemo to publish artifacts to the Galaxy Tool Shed, check out the Publishing to the Tool Shed documentation on reathedocs.org.

TravisCI

When tools are ready to be published to GitHub, it may be valuable to setup contineous integration to test changes committed and new pull requests. Travis CI is a service providing free testing and deep integration with GitHub.

The travis_init command will bootstrap a project with files to ease contineous integration testing of tools using a Planemo driven approach inspired by this great blog post by Peter Cock.

% planemo travis_init .
% # setup Ubuntu 12.04 w/tool dependencies
% vim .travis/setup_custom_dependencies.bash
% git add .travis.yml .travis
% git commit -m "Add Travis CI testing infrastructure for tools."
% git push # and register repository @ http://travis-ci.org/

In this example the file .travis/setup_custom_dependencies.bash should install the dependencies required to test your files on to the Travis user’s PATH.

This testing approach may only make sense for smaller repositories with a handful of tools. For larger repositories, such as tools-devteam or tools-iuc simply linting tool and tool shed artifacts may be more feasible. Check out the .travis.yml file used by the IUC as example.

Docker

Galaxy has experimental support for running jobs in Docker containers. Planemo contains tools to assist in development of Docker images for Galaxy tools.

A shell can be launched to explore the Docker enviornment referenced by tools that are annotated with publically registered Docker images.:

% $(planemo docker_shell bowtie2.xml)

For Docker containers still in development - a Dockerfile can be associated with a tool by sticking it in the tool’s directory. Planemo can then build and tag a Docker image for this tool and launch a shell into it using the following commands.:

% planemo docker_build bowtie2.xml # asssumes Dockerfile in same dir
% $(planemo docker_shell --from_tag bowtie2.xml)

For more details see the documentation for the docker_build and docker_shell commands.

Brew

The Galaxy development team is exploring different options for integrating Homebrew and linuxbrew with Galaxy. One angle is resolving the tool requirements using brew. An experimental approach for versioning of brew recipes will be used. See full discussion on the homebrew-science issues page here - https://github.com/Homebrew/homebrew-science/issues/1191. Information on the implementation can be found https://github.com/jmchilton/platform-brew until a more permanent project home is setup.

% planemo brew_init # install linuxbrew (only need if not already installed)
% planemo brew # install dependencies for all tools in directory.
% planemo brew bowtie2.xml # install dependencies for one tool
% which bowtie2
bowtie2 not found
% . <(planemo brew_env --shell bowtie2.xml) # shell w/brew deps resolved
(bowtie2) % which bowtie2
/home/john/.linuxbrew/Cellar/bowtie2/2.1.0/bin/bowtie2
(bowtie2) % exit
% . <(planemo brew_env bowtie2.xml) # or just source deps in cur env
% which bowtie2
/home/john/.linuxbrew/Cellar/bowtie2/2.1.0/bin/bowtie2

For more information see the documentation for the brew and brew_env commands.

Common Workflow Language

Planemo includes highly experimental support for running a subset of valid Common Workflow Language (CWL) tools using a fork of Galaxy enhanced to run CWL tools.

% planemo project_init --template cwl_draft2_spec
% planemo serve --cwl cwl_draft2_spec/cat1-tool.cwl

History

0.20.0 (2015-11-11)

• More complete I/O capturing for XUnit. 6409449

• Check for select parameter without options when linting tools. Issue 373

• Add --cwl_engine argument to cwl_run command. dd94ddc

• Fixes for select parameter linting. 8b31850

• Fix to demultiplexing repositories after tool uploads. Issue 361

• Fix to update planemo for Galaxy wheels. 25ef0d5

• Various fixes for Python 2.6 and Python 3. c1713d2, 916f610, c444855

0.19.0 (2015-11-03)

• Initial implementation of cwl_run command that runs a CWL tool and job file through Galaxy. 49c5c1e

• Add --cwl flag to serve to experimentally serve CWL tools in Galaxy. Pull Request 339

• Implement highly experimental cwl_script command to convert a CWL job to a bash script. 508dce7

• Add name to all XUnit reports (thanks to @erasche). Pull Request 343

• Capture stdout and stderr for shed_diff and shed_update XUnit reports. Pull Request 344

• More tool linting (conditionals) thanks to @erasche. Pull Request 350

• UTF-8 fixes when handling XUnit reports. Pull Request 345

• Add Epigenetics as Tool Shed category. Pull Request 351

• Merge changes to common modules shared between Galaxy, Planemo, and Pulsar (thanks to @natefoo). Pull Request 356

• Add --cite_url to tool_init. fdb1b51

• tool_init bug fix. f854138

• Fix setup.py for cwltool and bioblend changes. 1a157d4

• Add option to specify template sqlite database locally. c23569f

• Add example IPython notebooks to docs. c8640b6

0.18.1 (2015-10-22)

• Fix issue with test reporting not being populated. 19900a6

0.18.0 (2015-10-20)

• Improvements to docker_shell usability (thanks to @kellrott). Pull Request 334

• Add docker pull attempt when missing Dockerfile (thanks to @kellrott). Pull Request 333

• Fix bug inferring which files are tool files (thanks to @erasche). Pull Request 335, Issue 313

• Initial work toward automating brew recipe update. 4d6f7d9, Issue 329

0.17.0 (2015-10-19)

• Implement basic XUnit report option for shed_update (thanks to @martenson). Pull Request 322

• Fix issues with producing test outputs. 572e754

• Xunit reporting improvements - refactoring, times, diff output (thanks to @erasche). Pull Request 330

• Implement project governance policy and update developer code of conduct to match that of the Galaxy project. Pull Request 316

• Update filters for account for new .txt and .md test outputs (thanks to @erasche). Pull Request 327

• Add verbose logging to galaxy test output handling problems. 5d7db92

• Flake8 fixes (thanks to @martenson). 949a36d

• Remove uses of deprecated mktemp Python standard library function (thanks to @erasche). Pull Request 330

0.16.0 (2015-10-07)

• Adding new command dependency_script to convert Tool Shed dependencies into shell scripts - thanks to @peterjc. Pull Request 310, f798c7e, Issue 303

• Implement profiles in sheds section of the ~/.planemo.yml. Pull Request 314

0.15.0 (2015-10-01)

• Template framework for reporting including new markdown and plain text reporting options for testing - thanks to @erasche. Pull Request 304

• XUnit style reporting for shed_diff command - thanks to @erasche. Pull Request 305

• Add new shed_build command for building repository tarballs - thanks to @kellrott. Pull Request 297

• Fix exit code handling for lint commands - thanks to @mvdbeek. Pull Request 292

• Improved documentation for serve command - thanks to @lparsons. Pull Request 312

• Tiny backward compatible Python 3 tweaks for tool factory - thanks to @peterjc. dad2d9d

• Fixed detection of virtual environment in Makefile - thanks to @lparsons. Pull Request 311

• Updates to Galaxy XSD - thanks to @mr-c. Pull Request 309

• Allow reading shed key option from an environment variable. Pull Request 307

• Allow specifying host to serve Galaxy using -host - thanks in part to @chambm. Pull Request 301

• Allow specifying defaults for -host and --port in ~/.planemo.yml. Pull Request 301

• Improve ~/.planemo.yml sample comments - thanks to @martenson. Pull Request 287

• Update tool shed categories - thanks to @bgruening. Pull Request 285

• Improved output readibility for diff command - thanks to @martenson. Pull Request 284

0.14.0 (2015-08-06)

• Allow -t as shorthand for --shed_target (thanks to Peter Cock). Pull Request 278

• Fix tool_init command to use from_work_dir only if file in command (thanks to bug report and initial fix outline by Gildas Le Corguillé). Pull Request 277

• Various documentation fixes (thanks in part to Peter Cock and Daniel Blankenberg). Pull Request 256, Pull Request 253, Pull Request 254, Pull Request 255, Pull Request 251, Issue 272

0.13.2 (2015-07-06)

• Fix project_init for missing files. cb5b906

• Various documentation improvements.

0.13.1 (2015-07-01)

• Fix for shed_init producing non-standard type hints. Issue 243, f0610d7

• Fix tool linting for parameters that define an argument but not a name. Issue 245, aad1eed

• Many doc updates including a tutorial for developing tools in a test-driven fashion and instructions for using the planemo appliance through Kitematic (with Kitematic screenshots from Eric Rasche).

0.13.0 (2015-06-28)

• If planemo cannot find a Galaxy root, it will now automatically fetch one (specifing --galaxy_install will still force a fetch). Pull Request 235

• Docuementation has been updated to reflect new and vastly improved Docker and Vagrant virtual appliances are now available, as well as a new VirtualBox OVA variant.

• Update linting for new tool XML features (including detect_errors and output collections). Issue 233, 334f2d4

• Fix shed_test help text. Issue 223

• Fix code typo (thanks to Nicola Soranzo). Pull Request 230

• Improvements to algorithm used to guess if an XML file is a tool XML file. Issue 231

• Fix configuration file handling bug. Issue 240

0.12.2 (2015-05-23)

• Fix shed_test and shed_serve for test and local tool sheds. f3cafaa

0.12.1 (2015-05-21)

• Fix to ensure the tab completion script is in the Python source tarball (required for setting up tab-completion for Homebrew). 6b4e7a6

0.12.0 (2015-05-21)

• Implement a --failed flag for the test command to rerun previously faied tests. Pull Request 210

• Implement shed_update to upload contents and update repository metadata. Pull Request 216

• Implement shed_test and shed_serve commands to test and view published artifacts in the Tool Shed. Pull Request 213, Issue 176

• Add shell tab-completion script. 37dcc07

• Many more commands allow specifing multiple tool and/or repository targets. Issue 150

• Add -m as alias for –message in planemo shed_upload (thanks to Peter Cock). Pull Request 200

• Add --ensure_metadata option to shed_lint to ensure .shed.yml files contain many repository. Pull Request 215

• More developer documentation, additional make targets including ones for setting up git pre-commit hooks. cc8abb6, Issue 209

• Small README improvement (thanks to Martin Čech) b53006d

• Fixes for shed operation error handling (thanks to Martin Čech). Pull Request 203, Pull Request 206

• Fix for “smart” shed_diff not in the repository root directory (thanks to Peter Cock). Pull Request 207, Issue 205

• Recursive shed_diff with directories not yet in Tool Shed. Pull Request 208

• Improve error handling and reporting for problematic --shed_target values. Issue 217

• Fix typos in lint messages. Issue 211

0.11.1 (2015-05-12)

• Fix default behavior for planemo lint to use current directory if explicit paths are not supplied. 1e3668a

0.11.0 (2015-05-12)

• More compact syntax for defining multiple custom inclusions in .shed.yml files - thanks to Peter Cock. Issue 180, Pull Request 185, Pull Request 196

• Prevent ambigous destinations when defining custom inclusions in .shed.yml- thanks to Peter Cock. Pull Request 186

• lint now warns if tool ids contain whitespace. Pull Request 190

• Handle empty tar-balls gracefully on older Python versions - thanks to Peter Cock. Pull Request 187

• Tweak quoting in cp command - thanks to Peter Cock. 6bcf699

• Fix regression causing testing to no longer produce “pretty” test results under certain circumstances. Issue 188

• Fix for recursive shed_diff folder naming. Issue 192

• Fix output definitions to tool_init command. Issue 189

0.10.0 (2015-05-06)

• Extend shed_lint to check for valid actions in tool_dependencies.xml files. 8117e03

• Extend shed_lint to check for required files based on repository type. Issue 156

• Ignore common editor backup files during shed_upload. Issue 179

• Fix missing file when installing from source via PyPI. Issue 181

• Fix lint to verify data inputs specify a format attribute. 8117e03

• Docstring fix thanks to @peterjc. fe7ad46

0.9.0 (2015-05-03)

• Add new logo to the README thanks to @petrkadlec from puradesign.cz and @carlfeberhard from the Galaxy Project. Issue 108

• Implement smarter shed_diff command - it now produces a meaningful exit codes and doesn’t report differences if these correspond to attributes that will be automatically populated by the Tool Shed. Issue 167

• Use new smarter shed_diff code to implement a new --check_diff option for shed_upload - to check for meaningful differences before updating repositories. Issue 168

• Record git commit hash during shed_upload if the .shed.yml is located in a git repository. Issue 170

• Allow shed_ operations to operate on git URLs directly. Issue 169

• Fail if missing file inclusion statements encountered during .shed.yml repository resolution - bug reported by @peterjc. Issue 158

• Improved exception handling for tool shed operations including new --fail_fast command-line option. * Issue 114, Pull Request 173

• Implement more validation when using the shed_init command. 1cd0e2d

• Add -r/--recursive option to shed_download and shed_diff commands and allow these commands to work with .shed.yml files defining multipe repositories. 40a1f57

• Add --port option to the serve and tool_factory commands. 15804be

• Fix problem introduced with setup.py during the 0.9.0 development cycle - thanks to @peterjc. Pull Request 171

• Fix clone bug introduced during 0.9.0 development cycle - thanks to @bgruening. Pull Request 175

0.8.4 (2015-04-30)

• Fix for Travis CI testing picking up invalid tests (reported by @takadonet). Issue 161

• Fix tar ordering for consistency (always sort by name) - thanks to @peterjc. Pull Request 164, Issue 159

• Fix exception handling related to tool shed operations - thanks to @peterjc. Pull Request 155, b86fe1f

0.8.3 (2015-04-29)

• Fix bug where shed_lint was not respecting the -r/--recursive flag. 9ff0d2d

• Fix bug where planemo was producing tar files incompatible with the Tool Shed for package and suite repositories. a2ee135

0.8.2 (2015-04-29)

• Fix bug with config_init command thanks to @bgruening. Pull Request 151

• Fix unnessecary lint warning about parallelism tag reported by @peterjc. 9bf1eab

0.8.1 (2015-04-28)

• Fixes for the source distribution to allow installation of 0.8.0 via Homebrew.

0.8.0 (2015-04-27)

• Implement the new shed_lint command that verifies various aspects of tool shed repositories - including XSD validation of repository_dependencies.xml and tool_dependencies.xml files, best practices for README files, and the contents of .shed.yml files. This requires the lxml library to be available to Planemo or the application xmllint to be on its PATH. Pull Request 130 Issue 89 Issue 91 912df02 d26929e 36ac6d8

• Option to enable experimental XSD based validation of tools when lint is executed with the new --xsd flag. This validation occurs against the unofficial Galaxy Tool XSD project maintained by @JeanFred. This requires the lxml library to be available to Planemo or the application xmllint to be on its PATH. Pull Request 130 912df02

• Allow skipping specific linters when using the lint command using the new --skip option. 26e3cdb

• Implement sophisticated options in .shed.yml to map a directory to many, custom Tool Shed repositories during shed operaitons such shed_upload including automatically mapping tools to their own directories and automatically building suites repositories. Pull Request 143

• Make shed_upload more intelligent when building tar files so that package and suite repositories may have README files in source control and they will just be filtered out during upload. 53edd99

• Implement a new shed_init command that will help bootstrap .shed.yml files in the specified directory. cc1a447

• Extend shed_init to automatically build a repository_rependencies.xml file corresponding to a Galaxy workflow (.ga file). Issue 118 988de1d

• In addition to a single file or directory, allow lint to be passed multiple files. 343902d Issue 139

• Add -r/--recursive option to shed_create and lint commands. 63cd431 01f2af9

• Improved output formatting and option to write diffs to a file for the shed_diff command. 965511d

• Fix lint problem when using new Galaxy testing features such as expecting job failures and verifing job output. Issue 138

• Fix typo in test help thanks to first time contributor @pvanheus. Pull Request 129 1982076

• Fix NPE on empty help element when linting tools. Issue 124

• Fix lint warnings when configfiles are defined in a tool. 1a85493

• Fix for empty .shed.yml files. b7d9e96

• Fix the test command for newer versions of nose. 33294d2

• Update help content and documentation to be clear normalize should not be used to update the contents of tool files at this time. 08de8de

• Warn on unknown command attributes when linting tools (anything but interpreter). 4f61025

• Various design, documentation (including new documentation on Tool Shed publishing), and testing related improvements (test coverage has risen from 65% to over 80% during this release cycle).

0.7.0 (2015-04-13)

• Implement shed_create command to create Tool Shed repositories from .shed.yml files (thanks to Eric Rasche). Pull Request 101

• Allow automatic creation of missing repositories during shed_upload with the new --force_repository_creation flag (thanks to Eric Rasche). Pull Request 102

• Allow specifying files to exclude in .shed.yml when creating tar files for shed_upload (thanks to Björn Grüning). Pull Request 99

• Resolve symbolic links when building Tool Shed tar files with shed_upload (thanks to Dave Bouvier). Pull Request 104

• Add a Contributor Code of Conduct. Pull Request 113

• Omit tool_test_output.json from Tool Shed tar file created with shed_upload (thanks to Dave Bouvier). Pull Request 111

• Update required version of bioblend to 0.5.3. Fixed Issue 88.

• Initial work on implementing tests cases for Tool Shed functionality. 182fe57

• Fix incorrect link in HTML test report (thanks to Martin Čech). 4c71299

• Download Galaxy from the new, official Github repository. 7c69bf6

• Update travis_test to install stable planemo from PyPI. 39fedd2

• Enable caching on --install_galaxy by default (disable with --no_cache_galaxy). d755fe7

0.6.0 (2015-03-16)

• Many enhancements to the tool building documentation - descriptions of macros, collections, simple and conditional parameters, etc…

• Fix tool_init to quote file names (thanks to Peter Cock). Pull Request 98.

• Allow ignoring file patterns in .shed.yml (thanks to Björn Grüning). Pull Request 99

• Add --macros flag to tool_init command to generate a macro file as part of tool generation. ec6e30f

• Add linting of tag order for tool XML files. 4823c5e

• Add linting of stdio tags in tool XML files. 8207026

• More tests, much higher test coverage. 0bd4ff0

0.5.0 (2015-02-22)

• Implement --version option. Issue 78

• Implement --no_cleanup option for test and serve commands to persist temp files. 2e41e0a

• Fix bug that left temp files undeleted. Issue 80

• More improvements to release process. fba3874

0.4.2 (2015-02-21)

• Fix setup.py for installing non-Python data from PyPI (required newer for tool_factory command and reStructuredText linting). Thanks to Damion Dooley for the bug report. Issue 83

0.4.1 (2015-02-16)

• Fix README.rst so it renders properly on PyPI.

0.4.0 (2015-02-16)

• Implement tool_init command for bootstrapping creation of new tools (with tutorial.) 78f8274

• Implement normalize command for reorganizing tool XML and macro debugging. e8c1d45

• Implement tool_factory command to spin up Galaxy pre-configured the Tool Factory. 9e746b4

• Added basic linting of command blocks. b8d90ab

• Improved linting of help blocks, including verifying valid reStructuredText. 411a8da

• Fix bug related to serve command not killing Galaxy properly when complete. 53a6766

• Have serve command display tools at the top level instead of in shallow sections. badc25f

• Add additional dependencies to setup.py more functionality works out of the box. 85b9614

• Fix terrible error message related to bioblend being unavailable. Issue 70

• Various smaller documentation and project structure improvements.

0.3.1 (2015-02-15)

• Fixes to get PyPI workflow working properly.

0.3.0 (2015-02-13)

• Add option (-r) to the shed_upload command to recursively upload subdirectories (thanks to Eric Rasche). Pull Request 68

• Fix diff formatting in test reports (thanks to Eric Rasche). Pull Request 63

• Grab updated test database to speed up testing (thanks to approach from Eric Rasche and Dannon Baker). Issue 61, dff4f33

• Fix test data command-line argument name (was test-data now it is test_data). 834bfb2

• Use tool_data_table_conf.xml.sample file if tool_data_table_conf.xml.test is unavailable. Should allow some new tools to be tested without modifying Galaxy’s global tool_data_table_conf.xml file. ac4f828

0.2.0 (2015-01-13)

• Improvements to way Planemo loads its own copy of Galaxy modules to prevent various conflicts when launching Galaxy from Planemo. Pull Request 56

• Allow setting various test output options in ~/.planemo.yml and disabling JSON output. 21bb463

• More experimental Brew and Tool Shed options that should not be considered part of Planemo’s stable API. See bit.ly/gxbrew1 for more details.

• Fix project_init for BSD tar (thanks to Nitesh Turaga for the bug report.) a4110a8

• Documentation fixes for tool linting command (thanks to Nicola Soranzo). Pull Request 51

0.1.0 (2014-12-16)

• Moved repository URL to https://github.com/galaxyproject/planemo.

• Support for publishing to the Tool Shed. Pull Request 6

• Support for producing diffs (shed_diff) between local repositories and the Tool Shed (based on scripts by Peter Cock). Pull Request 33

• Use tool’s local test data when available - add option for configuring test-data target. Pull Request 1

• Support for testing tool features dependent on cached data. 44de95c

• Support for generating XUnit tool test reports. 82e8b1f

• Prettier HTML reports for tool tests. 05cc9f4

• Implement share_test command for embedding test result links in pull requests. Pull Request 40

• Fix for properly resolving links during Tool Shed publishing (thanks to Dave Bouvier). Pull Request 29

• Fix for citation linter (thanks to Michael Crusoe for the bug report). af39061

• Fix tool scanning for tool files with fewer than 10 lines (thanks to Dan Blankenberg). a2c13e4

• Automate more of Travis CI testing so the scripts added to tool repository can be smaller. 20a8680

• Documentation fixes for Travis CI (thanks to Peter Cock). Pull Request 22, Pull Request 23

• Various documentation fixes (thanks to Martin Čech). 36f7cb11, b9232e55

• Various smaller fixes for Docker support, tool linting, and documentation.

0.0.1 (2014-10-04)

• Initial work on the project - commands for testing, linting, serving Galaxy tools - and more experimental features involving Docker and Homebrew. 7d07782