This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Common workflow language reference implementation

Project Description
==================================================================
Common workflow language tool description reference implementation
==================================================================

CWL Conformance test: |Build Status|

Travis: |Unix Build Status|

.. |Unix Build Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=unix%20build
:target: https://travis-ci.org/common-workflow-language/cwltool

This is the reference implementation of the Common Workflow Language. It is
intended to be feature complete and provide comprehensive validation of CWL
files as well as provide other tools related to working with CWL.

This is written and tested for Python 2.7.

The reference implementation consists of two packages. The "cwltool" package
is the primary Python module containing the reference implementation in the
"cwltool" module and console executable by the same name.

The "cwlref-runner" package is optional and provides an additional entry point
under the alias "cwl-runner", which is the implementation-agnostic name for the
default CWL interpreter installed on a host.

Install
-------

Installing the official package from PyPi (will install "cwltool" package as
well)::

pip install cwlref-runner

If installling alongside another CWL implementation then::

pip install cwltool

To install from source::

git clone https://github.com/common-workflow-language/cwltool.git
cd cwltool && python setup.py install
cd cwlref-runner && python setup.py install # co-installing? skip this

Remember, if co-installing multiple CWL implementations then you need to
maintain which implementation ``cwl-runner`` points to via a symbolic file
system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_.

Running tests locally
---------------------

- Running basic tests ``(/tests)``:

.. code:: bash

python setup.py test

- Running the entire suite of CWL conformance tests:

The GitHub repository for the CWL specifications contains a script that tests a CWL
implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_
program

Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/master/CONFORMANCE_TESTS.md

Run on the command line
-----------------------

Simple command::

cwl-runner [tool-or-workflow-description] [input-job-settings]

Or if you have multiple CWL implementations installed and you want to override
the default cwl-runner use::

cwltool [tool-or-workflow-description] [input-job-settings]

Use with boot2docker
--------------------
boot2docker is running docker inside a virtual machine and it only mounts ``Users``
on it. The default behavoir of CWL is to create temporary directories under e.g.
``/Var`` which is not accessible to Docker containers.

To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix``
and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::

$ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json

.. |Build Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance
:target: https://ci.commonwl.org/job/cwltool-conformance/

Tool or workflow loading from remote or local locations
-------------------------------------------------------

``cwltool`` can run tool and workflow descriptions on both local and remote
systems via its support for HTTP[S] URLs.

Input job files and Workflow steps (via the `run` directive) can reference CWL
documents using absolute or relative local filesytem paths. If a relative path
is referenced and that document isn't found in the current directory then the
following locations will be searched:
http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem


Use with GA4GH Tool Registry API
--------------------------------

Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints.

By default, cwltool searches https://dockstore.org/ . Use --add-tool-registry to add other registries to the search path.

For example ::

cwltool --non-strict quay.io/collaboratory/dockstore-tool-bamstats:master test.json

and (defaults to latest when a version is not specified) ::

cwltool --non-strict quay.io/collaboratory/dockstore-tool-bamstats test.json

For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats

.. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas

Import as a module
------------------

Add::

import cwltool

to your script.

The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory::

import cwltool.factory
fac = cwltool.factory.Factory()

echo = f.make("echo.cwl")
result = echo(inp="foo")

# result["out"] == "foo"


Cwltool control flow
--------------------

Technical outline of how cwltool works internally, for maintainers.

#. Use CWL `load_tool()` to load document.

#. Fetches the document from file or URL
#. Applies preprocessing (syntax/identifier expansion and normalization)
#. Validates the document based on cwlVersion
#. If necessary, updates the document to latest spec
#. Constructs a Process object using `make_tool()` callback. This yields a
CommandLineTool, Workflow, or ExpressionTool. For workflows, this
recursively constructs each workflow step.
#. To construct custom types for CommandLineTool, Workflow, or
ExpressionTool, provide a custom `make_tool()`

#. Iterate on the `job()` method of the Process object to get back runnable jobs.

#. `job()` is a generator method (uses the Python iterator protocol)
#. Each time the `job()` method is invoked in an iteration, it returns one
of: a runnable item (an object with a `run()` method), `None` (indicating
there is currently no work ready to run) or end of iteration (indicating
the process is complete.)
#. Invoke the runnable item by calling `run()`. This runs the tool and gets output.
#. Output of a process is reported by an output callback.
#. `job()` may be iterated over multiple times. It will yield all the work
that is currently ready to run and then yield None.

#. "Workflow" objects create a corresponding "WorkflowJob" and "WorkflowJobStep" objects to hold the workflow state for the duration of the job invocation.

#. The WorkflowJob iterates over each WorkflowJobStep and determines if the
inputs the step are ready.
#. When a step is ready, it constructs an input object for that step and
iterates on the `job()` method of the workflow job step.
#. Each runnable item is yielded back up to top level run loop
#. When a step job completes and receives an output callback, the
job outputs are assigned to the output of the workflow step.
#. When all steps are complete, the intermediate files are moved to a final
workflow output, intermediate directories are deleted, and the output
callback for the workflow is called.

#. "CommandLineTool" job() objects yield a single runnable object.

#. The CommandLineTool `job()` method calls `makeJobRunner()` to create a
`CommandLineJob` object
#. The job method configures the CommandLineJob object by setting public
attributes
#. The job method iterates over file and directories inputs to the
CommandLineTool and creates a "path map".
#. Files are mapped from their "resolved" location to a "target" path where
they will appear at tool invocation (for example, a location inside a
Docker container.) The target paths are used on the command line.
#. Files are staged to targets paths using either Docker volume binds (when
using containers) or symlinks (if not). This staging step enables files
to be logically rearranged or renamed independent of their source layout.
#. The run() method of CommandLineJob executes the command line tool or
Docker container, waits for it to complete, collects output, and makes
the output callback.


Extension points
----------------

The following functions can be provided to main(), to load_tool(), or to the
executor to override or augment the listed behaviors.

executor(tool, job_order_object, **kwargs)
(Process, Dict[Text, Any], **Any) -> Tuple[Dict[Text, Any], Text]

A toplevel workflow execution loop, should synchronously execute a process
object and return an output object.

makeTool(toolpath_object, **kwargs)
(Dict[Text, Any], **Any) -> Process

Construct a Process object from a document.

selectResources(request)
(Dict[Text, int]) -> Dict[Text, int]

Take a resource request and turn it into a concrete resource assignment.

versionfunc()
() -> Text

Return version string.

make_fs_access(basedir)
(Text) -> StdFsAccess

Return a file system access object.

fetcher_constructor(cache, session)
(Dict[unicode, unicode], requests.sessions.Session) -> Fetcher

Construct a Fetcher object with the supplied cache and HTTP session.

resolver(document_loader, document)
(Loader, Union[Text, dict[Text, Any]]) -> Text

Resolve a relative document identifier to an absolute one which can be fetched.

logger_handler
logging.Handler

Handler object for logging.
Release History

Release History

This version
History Node

1.0.20170622090721

History Node

1.0.20170525215327

History Node

1.0.20170516234254

History Node

1.0.20170510165748

History Node

1.0.20170510151339

History Node

1.0.20170413194156

History Node

1.0.20170413151007

History Node

1.0.20170329142446

History Node

1.0.20170327143622

History Node

1.0.20170309164828

History Node

1.0.20170308174714

History Node

1.0.20170224141733

History Node

1.0.20170217172322

History Node

1.0.20170213175853

History Node

1.0.20170119234115

History Node

1.0.20170119182607

History Node

1.0.20170118141124

History Node

1.0.20170114120503

History Node

1.0.20170112185927

History Node

1.0.20170112154257

History Node

1.0.20170111193653

History Node

1.0.20170105144051

History Node

1.0.20161227200419

History Node

1.0.20161223144155

History Node

1.0.20161221171240

History Node

1.0.20161216212910

History Node

1.0.20161207161158

History Node

1.0.20161206204434

History Node

1.0.20161206195941

History Node

1.0.20161202203310

History Node

1.0.20161128202906

History Node

1.0.20161123190203

History Node

1.0.20161122201220

History Node

1.0.20161115095848

History Node

1.0.20161114152756

History Node

1.0.20161107145355

History Node

1.0.20161007181528

History Node

1.0.20161005195021

History Node

1.0.20160930152149

History Node

1.0.20160923180109

History Node

1.0.20160922135240

History Node

1.0.20160919152321

History Node

1.0.20160918012352

History Node

1.0.20160915202519

History Node

1.0.20160913171024

History Node

1.0.20160912182208

History Node

1.0.20160907141844

History Node

1.0.20160901133827

History Node

1.0.20160829211335

History Node

1.0.20160829192223

History Node

1.0.20160811184335

History Node

1.0.20160810200423

History Node

1.0.20160810161358

History Node

1.0.20160805221855

History Node

1.0.20160726135535

History Node

1.0.20160714182449

History Node

1.0.20160712154127

History Node

1.0.20160708190014

History Node

1.0.20160707195347

History Node

1.0.20160706132520

History Node

1.0.20160630171631

History Node

1.0.20160629140624

History Node

1.0.20160628194545

History Node

1.0.20160627152300

History Node

1.0.20160626203316

History Node

1.0.20160623183600

History Node

1.0.20160616182520

History Node

1.0.20160614212644

History Node

1.0.20160610111115

History Node

1.0.20160609160402

History Node

1.0.20160608124624

History Node

1.0.20160531173804

History Node

1.0.20160523144113

History Node

1.0.20160519182434

History Node

1.0.20160518201549

History Node

1.0.20160518200809

History Node

1.0.20160515155531

History Node

1.0.20160511213450

History Node

1.0.20160511162129

History Node

1.0.20160511142738

History Node

1.0.20160510161706

History Node

1.0.20160507101510

History Node

1.0.20160505211047

History Node

1.0.20160504183010

History Node

1.0.20160427142240

History Node

1.0.20160426185417

History Node

1.0.20160425140546

History Node

1.0.20160422204730

History Node

1.0.20160422203349

History Node

1.0.20160421171618

History Node

1.0.20160421140153

History Node

1.0.20160415153333

History Node

1.0.20160413143011

History Node

1.0.20160412203002

History Node

1.0.20160411194133

History Node

1.0.20160411021840

History Node

1.0.20160408133555

History Node

1.0.20160331184641

History Node

1.0.20160325210917

History Node

1.0.20160325200114

History Node

1.0.20160323212343

History Node

1.0.20160322201127

History Node

1.0.20160316204054

History Node

1.0.20160316150250

History Node

1.0.20160315233236

History Node

1.0.20160311201238

History Node

1.0.20160311170456

History Node

1.0.20160310215251

History Node

1.0.20160310140736

History Node

1.0.20160309204538

History Node

1.0.20160308152645

History Node

1.0.20160307200623

History Node

1.0.20160304012934

History Node

1.0.20160302134341

History Node

1.0.20160226205952

History Node

1.0.20160225202307

History Node

1.0.20160225040942

History Node

1.0.20160225033850

History Node

1.0.20160222205901

History Node

1.0.20160209222805

History Node

1.0.20160203221531

History Node

1.0.20160203144931

History Node

1.0.20160129183049

History Node

1.0.20160129161544

History Node

1.0.20160129152024

History Node

1.0.20160128142049

History Node

1.0.20160127144612

History Node

1.0.20160126211726

History Node

1.0.20160126152227

History Node

1.0.20160115132942

History Node

1.0.20160108200940

History Node

1.0.20160108161501

History Node

1.0.20151211155311

History Node

1.0.20151211141743

History Node

1.0.20151210154014

History Node

1.0.20151209160516

History Node

1.0.20151130204648

History Node

1.0.20151130190253

History Node

1.0.20151125221324

History Node

1.0.20151125211848

History Node

1.0.20151125172224

History Node

1.0.20151124220039

History Node

1.0.20151124040259

History Node

1.0.20151122025918

History Node

1.0.20151121032923

History Node

1.0.20151121025646

History Node

1.0.20151120220905

History Node

1.0.20151112194920

History Node

1.0.20151110030107

History Node

1.0.20151104062100

History Node

1.0.20151026181844

History Node

1.0.20151022085449

History Node

1.0.20151013173827

History Node

1.0.20151013135545

History Node

1.0.20151013134821

History Node

1.0.20151012173709

History Node

1.0.20151012161407

History Node

1.0.20151009020717

History Node

1.0.20151007133346

History Node

1.0.20150929170517

History Node

1.0.20150925123259

History Node

1.0.20150923183439

History Node

1.0.20150918080732

History Node

1.0.20150916124227

History Node

1.0.20150916041152

History Node

1.0.20150915034626

History Node

1.0.20150910021332

History Node

1.0.20150814191928

History Node

1.0.20150814015745

History Node

1.0.20150813132038

History Node

1.0.20150808190318

History Node

1.0.20150728161219

History Node

1.0.20150723141036

History Node

1.0.20150722144138

History Node

1.0.20150720194125

History Node

1.0.20150715173345

History Node

1.0.20150714132635

History Node

1.0.20150713185212

History Node

1.0.20150713152138

History Node

1.0.20150713135958

History Node

1.0.20150711153503

History Node

1.0.20150709135652

History Node

1.0.20150708115452

History Node

1.0.20150707043955

History Node

1.0.20150706174116

History Node

1.0.20150627015636

History Node

1.0.20150626035417

History Node

1.0.20150626035015

History Node

1.0.20150626032845

History Node

1.0.20150624201710

History Node

1.0.20150624181807

History Node

1.0.20150623031442

History Node

1.0.20150623020304

History Node

1.0.20150622183905

History Node

1.0.20150622144406

History Node

1.0.20150622134207

History Node

1.0.20150621230545

History Node

1.0.20150621213723

History Node

1.0.20150610150530

History Node

1.0.20150609205603

History Node

1.0.20150609024148

History Node

1.0.20150606210100

History Node

1.0.20150605134212

History Node

1.0.20150602212140

History Node

1.0.20150601200103

History Node

1.0.20150601172915

History Node

1.0.20150527172438

History Node

1.0.20150525010411

History Node

1.0.20150522025529

History Node

1.0.20150512164512

History Node

1.0.20150512154247

History Node

1.0.20150508023316

History Node

1.0.20150429022405

History Node

1.0.20150415152714

History Node

1.0.20150409193818

History Node

1.0.20150407211342

History Node

1.0.20150403144650

History Node

1.0.20150401014645

History Node

1.0.20150331014533

History Node

1.0.20150327211418

History Node

1.0.20150326211508

History Node

1.0.20150326005309

History Node

1.0.20150324211351

History Node

1.0.20150324203244

History Node

1.0.20150324200038

History Node

1.0.20150324194338

History Node

1.0.20150324173916

History Node

1.0.20150324172633

History Node

1.0.20150324012706

History Node

1.0.20150324011020

History Node

1.0.20150318015654

History Node

1.0.20150318010828

History Node

1.0.20150317015350

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
cwltool-1.0.20170622090721-py2-none-any.whl (300.3 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Jun 22, 2017
cwltool-1.0.20170622090721.tar.gz (238.0 kB) Copy SHA256 Checksum SHA256 Source Jun 22, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting