ewms-pilot 0.24.10

EWMS's Pilot: A Job Pilot for Processing EWMS Events/Tasks

ewms-pilot

An Event-Task Pilot for EWMS

The EWMS Pilot is a non-user-facing wrapper for task container instances in the Event Workflow Management System (EWMS), running on an HTCondor Execution Point (EP). The pilot:

• Triggers task instances for each inbound event.
• Interfaces with EWMS events as input/output files.
• Isolates task containers from one another.
• Provides fault tolerance for failed tasks, CPUs, etc.

The following outlines what users need to know to operate within EWMS.

Overview

The Pilot is designed to be invisible to users. However, some key details are necessary for running a task container:

Event I/O

An input event is provided to the task container as a file. The pilot injects the file path by replacing the string {{INFILE}} in the image_args of the task directive. Similarly, the task container creates an output event by writing to the file path given by {{OUTFILE}}. The file extensions are defined by environment variables EWMS_PILOT_INFILE_EXT and EWMS_PILOT_OUTFILE_EXT: by default, these are .in and .out, respectively.

No other event or message handling is required by the task container.

File I/O

Task containers can interact with external files in two ways:

Inter-Task Files

To transfer files between task containers, a shared directory is available to all task containers. This is provided by the environment variable EWMS_TASK_DATA_HUB_DIR, referred to as the "data hub".

Note:

• The data hub directory is writable, but there is no protection against race conditions for parallelized tasks.
• This directory is also available to the init container.

External Files

Externally-mounted directories are supported in EWMS. See the WMS documentation for more details.

The Init Container

An init container is an optional, user-supplied image used to set up the environment, wait for conditions, or perform other preparatory actions before running task containers. It is configured using the EWMS_PILOT_INIT_IMAGE and EWMS_PILOT_INIT_ARGS environment variables.

EWMS Glossary Applied to the Pilot

Workflow

Does not exist within the Pilot. Compare to WMS.

Message Queue

The message queue is abstracted from the task container and can be ignored. Compare to WMS.

Event

An event is an object transferred via event I/O. Compare to WMS.

Task

In the context of the Pilot, the task is the runtime instance of the task image (a task container) applied to an inbound event, possibly producing outbound events. Compare to WMS.

Task Container

The task container is an instance of a task image and is nearly synonymous with task.

Task Directive

Does not exist within the Pilot. Compare to WMS.

Taskforce

Does not exist within the Pilot. Compare to WMS.