Requirements

Before the installation, the following components are required to be installed on your x86 Node of SX-Aurora TSUBASA.

• Alternative VE Offloading (AVEO)

  • required version: >= 3.6.0

• NEC MPI

  • required NEC MPI version: >= 3.10.0 (for NVIDIA DOCA 3.1.0)

• Python

  • required version: 3.12

• NumPy

  • required version: v1.26.0

• NLC(optional)

  • required version: >= 3.0.0

• NLCPy(optional)

  • required version: >= 3.1.0

Since December 2022, mpi4py-ve has been provided as a software of NEC SDK (NEC Software Development Kit for Vector Engine). If NEC SDK on your machine has been properly installed or updated after that, mpi4py-ve is available by using /usr/bin/python3 command.

Install from wheel

You can install mpi4py-ve by executing either of the following commands.

• Install from PyPI

  $ pip install mpi4py-ve

• Install from your local computer

  1. Download the wheel package from GitHub.

  2. Put the wheel package to your any directory.

  3. Install the local wheel package via pip command.

     $ pip install <path_to_wheel>

The shared objects for Vector Host, which are included in the wheel package, are compiled by gcc 4.8.5 and tested by using following software:

NEC MPI	v3.10.0
NumPy	v1.26.0
NLCPy	v3.1.0

Install from source (with building)

Before building this package, you need to execute the environment setup script necmpivars.sh or necmpivars.csh once advance.

• When using sh or its variant:

  For VE30

  $ source /opt/nec/ve3/mpi/X.X.X/bin/necmpivars.sh

  For VE20, VE10, or VE10E

  $ source /opt/nec/ve/mpi/X.X.X/bin/necmpivars.sh

• When using csh or its variant:

  For VE30

  % source /opt/nec/ve3/mpi/X.X.X/bin/necmpivars.csh

  For VE20, VE10, or VE10E

  % source /opt/nec/ve/mpi/X.X.X/bin/necmpivars.csh

Here, X.X.X denotes the version number of NEC MPI.

After that, execute the following commands:

$ git clone https://github.com/SX-Aurora/mpi4py-ve.git
$ cd mpi4py-ve
$ MPICFG=necmpi python -m pip install .

Example

Transfer Array

Transfers an NLCPy’s ndarray from MPI rank 0 to 1 by using comm.Send() and comm.Recv():

from mpi4pyve import MPI
import nlcpy as vp

comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()

if rank == 0:
    x = vp.array([1,2,3], dtype=int)
    comm.Send(x, dest=1)

elif rank == 1:
    y = vp.empty(3, dtype=int)
    comm.Recv(y, source=0)

Sum of Numbers

Sums the numbers locally, and reduces all the local sums to the root rank (rank=0):

from mpi4pyve import MPI
import nlcpy as vp

comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()

N = 1000000000
begin = N * rank // size
end = N * (rank + 1) // size

sendbuf = vp.arange(begin, end).sum()
recvbuf = comm.reduce(sendbuf, MPI.SUM, root=0)

The following table shows the performance results[msec] on VE Type 20B:

Execution

When executing Python script using mpi4py-ve, use mpirun-py wrapper script that invokes the NEC MPI mpirun command on an x86 server of SX-Aurora TSUBASA. mpirun-py wrapper script is used to set up necessary environment variables required to use NEC MPI from mpi4py-ve. So please do not use mpirun command directly, otherwise your Python script may cause unexpected errors. Before running the Python script, you need to execute the environment the following setup scripts once advance.

• When using sh or its variant:

  For VE30

  $ source /opt/nec/ve3/mpi/X.X.X/bin/necmpivars.sh gnu 4.8.5
  $ source `$(which python) -c "import mpi4pyve;print(mpi4pyve.__path__[0])"`/bin/mpi4pyvevars.sh

  For VE20, VE10, or VE10E

  $ source /opt/nec/ve/mpi/X.X.X/bin/necmpivars.sh gnu 4.8.5
  $ source `$(which python) -c "import mpi4pyve;print(mpi4pyve.__path__[0])"`/bin/mpi4pyvevars.sh

• When using csh or its variant:

  For VE30

  % source /opt/nec/ve3/mpi/X.X.X/bin/necmpivars.csh gnu 4.8.5
  % source `$(which python) -c "import mpi4pyve;print(mpi4pyve.__path__[0])"`/bin/mpi4pyvevars.csh

  For VE20, VE10, or VE10E

  % source /opt/nec/ve/mpi/X.X.X/bin/necmpivars.csh gnu 4.8.5
  % source `$(which python) -c "import mpi4pyve;print(mpi4pyve.__path__[0])"`/bin/mpi4pyvevars.csh

Here, X.X.X denotes the version number of NEC MPI.

When using the mpirun-py command:

$ mpirun-py -veo -np N $(which python) sample.py

$ export NMPI_USE_COMMAND_SEARCH_PATH=ON
$ mpirun-py -veo -np N python sample.py

Execution Examples

The following examples show how to launch MPI programs that use mpi4py-ve and NLCPy on the SX-Aurora TSUBASA.

• Interactive Execution

  • Execution on one VE

    Example of using 4 processes on local VH and 4 VE processes (ncore / 4 OpenMP parallel per process) on VE#0 of local VH

    $ mpirun-py -veo -np 4 python a.py

  • Execution on multiple VEs on a VH

    Example of using 4 processes on local VH and 4 VE processes (1 process per VE, ncore OpenMP parallel per process) on VE#0 to VE#3 of local VH

    $ VE_NLCPY_NODELIST=0,1,2,3 mpirun-py -veo -np 4 python a.py

    Example of using 32 processes on local VH and 32 VE processes (8 processes per VE, ncore / 8 OpenMP parallel per process) on VE#0 to VE# 3 of local VH

    $ VE_NLCPY_NODELIST=0,1,2,3 mpirun-py -veo -np 32 python a.py

  • Execution on multiple VEs on multiple VHs

    Example of using a total of 32 processes on two VHs host1 and host2, and a total of 32 VE processes on VE#0 and VE#1 of each VH (8 processes per VE, ncore / 8 OpenMP parallel per process)

    $ VE_NLCPY_NODELIST=0,1 mpirun-py -hosts host1,host2 -veo -np 32 python a.py

• NQSV Request Execution

  • Execution on a specific VH, on a VE

    Example of using 32 processes on logical VH#0 and 32 VE processes on logical VE#0 to logical VE#3 on logical VH#0 (8 processes per VE, ncore / 8 OpenMP parallel per process)

    #PBS -T necmpi
    #PBS -b 2 # The number of logical hosts
    #PBS --venum-lhost=4 # The number of VEs per logical host
    #PBS --cpunum-lhost=32 # The number of CPUs per logical host
    
    source /opt/nec/ve/mpi/X.X.X/bin/necmpivars.sh
    export NMPI_USE_COMMAND_SEARCH_PATH=ON
    mpirun-py -host 0 -veo -np 32 python a.py

  • Execution on a specific VH, on a specific VE

    Example of using 16 processes on logical VH#0, 16 VE processes in total on logical VE#0 and logical VE#3 on logical VH#0 (8 processes per VE, ncore / 8 OpenMP parallel per process)

    #PBS -T necmpi
    #PBS -b 2 # The number of logical hosts
    #PBS --venum-lhost=4 # The number of VEs per logical host
    #PBS --cpunum-lhost=16 # The number of CPUs per logical host
    
    source /opt/nec/ve/mpi/X.X.X/bin/necmpivars.sh
    export NMPI_USE_COMMAND_SEARCH_PATH=ON
    VE_NLCPY_NODELIST=0,3 mpirun-py -host 0 -veo -np 16 python a.py

  • Execution on all assigned VEs

    Example of using 32 processes in total on 4 VHs and using 32 VE processes in total from logical VE#0 to logical VE#7 on each of VHs (1 process per VE, ncore OpenMP parallel per process).

    #PBS -T necmpi
    #PBS -b 4 # The number of logical hosts
    #PBS --venum-lhost=8 # The number of VEs per logical host
    #PBS --cpunum-lhost=8 # The number of CPUs per logical host
    #PBS --use-hca=2 # The number of HCAs
    
    source /opt/nec/ve/mpi/X.X.X/bin/necmpivars.sh
    export NMPI_USE_COMMAND_SEARCH_PATH=ON
    mpirun-py -veo -np 32 python a.py

Here, X.X.X denotes the version number of NEC MPI.

Profiling

NEC MPI provides the facility of displaying MPI communication information. There are two formats of MPI communication information available as follows:

You can control the display and format of MPI communication information by setting the environment variable NMPI_COMMINF at runtime as shown in the following table.

The Settings of NMPI_COMMINF:

When using the mpirun-py command:

$ export NMPI_COMMINF=ALL
$ mpirun-py -veo -np N python sample.py

Use mpi4py-ve with homebrew classes (without NLCPy)

Below links would be useful to use mpi4py-ve with homebrew classes (without NLCPy):

• use mpi4py-ve with homebrew classes (without NLCPy)

Other Documents

Below links would be useful to understand mpi4py-ve in more detail:

• mpi4py-ve tutorial

Restriction

• The current version of mpi4py-ve does not support some functions that are listed in the section “List of Unsupported Functions” of mpi4py-ve tutorial.

• Communication of type bool between NumPy and NLCPy will fail because of the different number of bytes.

Notices

• If you import NLCPy before calling MPI_Init()/MPI_Init_thread(), a runtime error will be raised.

  Not recommended usage:

  $ mpirun-py -veo -np 1 $(which python) -c "import nlcpy; from mpi4pyve import MPI"
  RuntimeError: NLCPy must be import after MPI initialization

  Recommended usage:

  $ mpirun-py -veo -np 1 $(which python) -c "from mpi4pyve import MPI; import nlcpy"

  MPI_Init() or MPI_Init_thread() is called when you import the MPI module from the mpi4pyve package.

• If you use the Lock/Lock_all function for one-sided communication using NLCPy array data, you need to put in NLCPy synchronization control.

  Synchronization usage:

  import mpi4pyve
  from mpi4pyve import MPI
  import nlcpy as vp
  
  comm = MPI.COMM_WORLD
  size = comm.Get_size()
  rank = comm.Get_rank()
  
  array = vp.array(0, dtype=int)
  
  if rank == 0:
      win_n = MPI.Win.Create(array,  comm=MPI.COMM_WORLD)
  else:
      win_n = MPI.Win.Create(None, comm=MPI.COMM_WORLD)
  if rank == 0:
      array.fill(1)
      array.venode.synchronize()
      comm.Barrier()
  if rank != 0:
     comm.Barrier()
      win_n.Lock(MPI.LOCK_EXCLUSIVE, 0)
      win_n.Get([array, MPI.INT], 0)
      win_n.Unlock(0)
      assert array == 1
  comm.Barrier()
  win_n.Free()

License