Py4HEAppE (Python for HEAppE Middleware)

Release: 3.1.0

Py4HEAppE provides both a command-line interface and versioned Python wrappers for the HEAppE middleware API.

You can use it in two ways:

• HEAppE CLI commands for interactive and scripting use
• Versioned Python API wrappers for direct integration into your own applications

Supported HEAppE Versions

Requirements

• Python 3.11 or newer
• Access to a deployed HEAppE instance
• HEAppE instance URL
• HEAppE accounting string for the target computational project

Installation

If the py4heappe executable is not on your PATH, use the full path to the installed script or add the script directory to PATH.

On Windows, use py4heappe.exe instead of py4heappe.

HEAppE CLI

The CLI is intended for users who want to work with HEAppE directly from a terminal without writing Python code.

Initial Setup

Before using the CLI for the first time, initialize the HEAppE instance URL and the accounting string:

py4heappe Conf Init

Available Command Groups

py4heappe --help

Current command groups:

• Conf for local CLI configuration
• Auth for authentication
• CmdTemp for command template operations
• Info for version and cluster information
• Job for job lifecycle management
• FileTransfer for job file upload and download operations
• Report for resource usage and reporting

Useful Help Commands

py4heappe Auth --help
py4heappe Info --help
py4heappe Job --help
py4heappe FileTransfer --help
py4heappe Report --help

Basic CLI Examples

Authenticate with username and password:

py4heappe Auth UserPass

Show the HEAppE API version:

py4heappe Info Version

List available clusters:

py4heappe Info ClusterInfo

List your jobs:

py4heappe Job List

Create a Job from a JSON Template

Py4HEAppE supports a JSON-first workflow for job creation.

Generate a minimal job specification template in the current directory:

py4heappe Job InitJobSpecification

Generate the full swagger-shaped job specification template instead:

py4heappe Job InitJobSpecification --full

Or generate it in a specific directory or target file:

py4heappe Job InitJobSpecification --file-destination ./job_specs
py4heappe Job InitJobSpecification --file-destination ./job_specs/demo_job.json

Create a job from the JSON file and override selected values from the command line:

py4heappe Job Create --json-job-spec-file ./job_specification.json \
    --name demo-job \
    --cluster-id 2 \
    --project-id 1 \
    --task-name 0:demo-task \
    --task-max-cores 0:128 \
    --walltime-limit 0:1800 \
    --cluster-node-type-id 0:18 \
    --cmd-template-id 0:3 \
    --cmd-template-parameters inputParam:testValue

CLI overrides always take precedence over values loaded from --json-job-spec-file.

Save the final parsed job specification that will be submitted to HEAppE:

py4heappe Job Create --json-job-spec-file ./job_specification.json \
    --save-result-job-specification

This writes resultJobSpecJson.json into the current working directory.

Override Format for Nested Task Data

Some Job Create options work on indexed task items and nested collections.

Examples:

# Task scalar override
py4heappe Job Create --json-job-spec-file ./job_specification.json --task-name 0:main-task

# Task environment variable override
py4heappe Job Create --json-job-spec-file ./job_specification.json \
  --task-environment-variable 0:0:Name:OMP_NUM_THREADS \
  --task-environment-variable 0:0:Value:8

# Task template parameter override
py4heappe Job Create --json-job-spec-file ./job_specification.json \
  --task-template-parameter-value 0:0:CommandParameterIdentifier:inputParam \
  --task-template-parameter-value 0:0:ParameterValue:testValue

File Transfer Examples

List files changed during job execution:

py4heappe FileTransfer ListChanged --help

Download a single file from a job directory:

py4heappe FileTransfer Download --help

Upload files directly into the job execution directory:

py4heappe FileTransfer Stream --help

Python API Wrapper

Py4HEAppE also ships versioned Python wrappers generated from the HEAppE API.

Use the wrapper that matches your target HEAppE version:

import py4heappe.heappe_v6.core as hp
# or:
# import py4heappe.heappe_v5.core as hp
# import py4heappe.heappe_v4.core as hp

There is no single unversioned py4heappe.core module. Use the versioned wrapper explicitly.

Minimal Python Example

import json

import py4heappe.heappe_v6.core as hp

configuration = hp.Configuration()
configuration.host = "https://heappe.example.org"
api_client = hp.ApiClient(configuration)

try:
    auth_api = hp.UserAndLimitationManagementApi(api_client)
    management_api = hp.ManagementApi(api_client)
    cluster_api = hp.ClusterInformationApi(api_client)

    auth_body = {
        "_preload_content": False,
        "body": {
            "Credentials": {
                "Username": "username",
                "Password": "password",
            }
        },
    }

    auth_response = auth_api.heappe_user_and_limitation_management_authenticate_user_password_post(
        **auth_body
    )
    session_code = json.loads(auth_response.data)

    version_response = management_api.heappe_management_version_information_get(
        _preload_content=False,
        SessionCode=session_code,
    )
    print(json.dumps(json.loads(version_response.data), indent=2))

    cluster_response = cluster_api.heappe_cluster_information_list_available_clusters_get(
        _preload_content=False,
        SessionCode=session_code,
    )
    print(json.dumps(json.loads(cluster_response.data), indent=2))

finally:
    api_client.pool.close()
    api_client.pool.join()

Create a Job Through the Python Wrapper

import json

import py4heappe.heappe_v6.core as hp

configuration = hp.Configuration()
configuration.host = "https://heappe.example.org"
api_client = hp.ApiClient(configuration)

try:
    job_api = hp.JobManagementApi(api_client)

    body = {
        "_preload_content": False,
        "body": {
            "SessionCode": "your-session-code",
            "JobSpecification": {
                "Name": "demo-job",
                "ProjectId": 1,
                "ClusterId": 2,
                "FileTransferMethodId": 1,
                "Tasks": [
                    {
                        "Name": "demo-task",
                        "MinCores": 1,
                        "MaxCores": 16,
                        "WalltimeLimit": 1800,
                        "StandardOutputFile": "stdout",
                        "StandardErrorFile": "stderr",
                        "ProgressFile": "stdprog",
                        "LogFile": "stdlog",
                        "ClusterNodeTypeId": 18,
                        "CommandTemplateId": 3,
                        "TemplateParameterValues": [
                            {
                                "CommandParameterIdentifier": "inputParam",
                                "ParameterValue": "testValue",
                            }
                        ],
                    }
                ],
            },
        },
    }

    response = job_api.heappe_job_management_create_job_post(**body)
    job_id = json.loads(response.data)["Id"]
    print(f"Created job: {job_id}")

finally:
    api_client.pool.close()
    api_client.pool.join()

Running CLI tests

The repository includes live CLI integration tests under tests/ that require a reachable HEAppE instance and several environment variables.

Quick steps:

• Create a tests/.env from the example and edit it:

  • Unix: cp tests/.env.example tests/.env
  • Windows (PowerShell): Copy-Item tests\.env.example tests\.env

• Populate required values in tests/.env (or export the variables). At minimum set PY4HEAPPE_RUN_LIVE_CLI_TESTS=1 and PY4HEAPPE_TEST_URL and PY4HEAPPE_TEST_PROJECT. To allow tests that change remote state also set PY4HEAPPE_TEST_RUN_MUTATING_COMMANDS=1.

• Run the CLI tests (default, using the standard library's unittest):

  • Run all CLI tests: python -m unittest discover -s tests -p "test_cli_*.py" -v
  • Run a single test module: python -m unittest tests.test_cli_auth -v

• Optional: run with pytest (pytest can run unittest-style tests and offers improved output):

  • Install pytest: python -m pip install -U pytest
  • Run with pytest: python -m pytest -q tests/test_cli_*.py

Do NOT commit tests/.env with real credentials or session tokens.

Notes

• CLI configuration is stored locally after py4heappe Conf Init.
• Most CLI commands require a valid session, so authenticate before running job or file-transfer operations.
• For detailed option lists, use the built-in --help on each command group and subcommand.

Acknowledgement

This work was supported by the Ministry of Education, Youth and Sports of the Czech Republic through e-INFRA CZ (ID:90254).