MPI on the grid

This will explain how to run MPI jobs on the grid. In this document we will not elaborate on cross site MPI jobs, only jobs within one site (cluster).

MPI code

We use a standard MPI flavor (OpenMPI) and standard C code that uses MPI. This code should run on any MPI-enabled machine (like a cluster).

Code for mpi-test.c:

/*
*Simple "Hello World" program in MPI.
 *
*/

#include "mpi.h"
#include <stdio.h>
int main(int argc, char *argv[]) {

  int numprocs;  /* Number of processors */
  int procnum;   /* Processor number */

  /* Initialize MPI */
  MPI_Init(&argc, &argv);

  /* Find this processor number */
  MPI_Comm_rank(MPI_COMM_WORLD, &procnum);

  /* Find the number of processors */
  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
  printf ("Hello world! from processor %d out of %d\n", procnum, numprocs);

  /* Shut down MPI */
  MPI_Finalize();
  return 0;
}

MPI wrapper and JDL file

In this section we will prepare our MPI job for execution on the grid; we need a wrapper like "mpirun", and of course a JDL file.

The wrapper code is necessary to start the job on multiple processors (or cores). This is mpi-start-wrapper.sh:

#!/bin/bash

# Pull in the arguments.
MY_EXECUTABLE=<tt>pwd</tt>/$1
MPI_FLAVOR=$2

# Convert flavor to lowercase for passing to mpi-start.
MPI_FLAVOR_LOWER=<tt>echo $MPI_FLAVOR | tr '[:upper:]' '[:lower:]'</tt>

# Pull out the correct paths for the requested flavor.
eval MPI_PATH=<tt>printenv MPI_${MPI_FLAVOR}_PATH</tt>

# Ensure the prefix is correctly set.  Don't rely on the defaults.
eval I2G_${MPI_FLAVOR}_PREFIX=$MPI_PATH
export I2G_${MPI_FLAVOR}_PREFIX

# Touch the executable.  It exist must for the shared file system check.
# If it does not, then mpi-start may try to distribute the executable
# when it shouldn't.
touch $MY_EXECUTABLE

# Setup for mpi-start.
export I2G_MPI_APPLICATION=$MY_EXECUTABLE
export I2G_MPI_APPLICATION_ARGS=
export I2G_MPI_TYPE=$MPI_FLAVOR_LOWER
export I2G_MPI_PRE_RUN_HOOK=mpi-hooks.sh
export I2G_MPI_POST_RUN_HOOK=mpi-hooks.sh

# If these are set then you will get more debugging information.
export I2G_MPI_START_VERBOSE=1
#export I2G_MPI_START_DEBUG=1

# Invoke mpi-start.
$I2G_MPI_START

This is mpi-hooks.sh:

#!/bin/sh

#
# This function will be called before the MPI executable is started.
# You can, for example, compile the executable itself.
#
pre_run_hook () {

  # Compile the program.
  echo "Compiling ${I2G_MPI_APPLICATION}"

  # Actually compile the program.
  cmd="mpicc ${MPI_MPICC_OPTS} -o ${I2G_MPI_APPLICATION} ${I2G_MPI_APPLICATION}.c"
  echo $cmd
  $cmd
  if [ ! $? -eq 0 ]; then
    echo "Error compiling program.  Exiting..."
    exit 1
  fi

  # Everything's OK.
  echo "Successfully compiled ${I2G_MPI_APPLICATION}"

  return 0
}

#
# This function will be called before the MPI executable is finished.
# A typical case for this is to upload the results to a storage element.
#
post_run_hook () {

  echo "Executing post hook."
  echo "Finished the post hook."

  return 0
}

And this is the JDL file, mpi-test.jdl:

#
# mpi-test.jdl
#
JobType        = "MPICH";
CPUNumber      = 16;
Executable     = "mpi-start-wrapper.sh";
Arguments      = "mpi-test OPENMPI";
StdOutput      = "mpi-test.out";
StdError       = "mpi-test.err";
InputSandbox   = {"mpi-start-wrapper.sh","mpi-hooks.sh","mpi-test.c"};
OutputSandbox  = {"mpi-test.err","mpi-test.out"};
Requirements =
  Member("MPI-START", other.GlueHostApplicationSoftwareRunTimeEnvironment)
  && Member("OPENMPI", other.GlueHostApplicationSoftwareRunTimeEnvironment);
#
# - the end
#

The requirement-lines make sure the job is only started on a site supporting OpenMPI and MPI-START. The line with CPUNumber configures the job to run on 16 cpu's (or cores); only a site with at least this number of cpu's will be selected.

Execution and result

Submit the job using the standard methods:

$ glite-wms-job-submit -a mpi-test.jdl

Connecting to the service https://wms.begrid.be:7443/glite_wms_wmproxy_server


====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://wms.begrid.be:9000/baN2oRxNEA3vQK6YMLsy8A

==========================================================================

When the job is finished, get the outputs:

$ glite-wms-job-output --noint --dir . https://wms.begrid.be:9000/baN2oRxNEA3vQK6YMLsy8A

Connecting to the service https://wms.begrid.be:7443/glite_wms_wmproxy_server


Warning - Directory already exists:
/data/home/jo/tmp/MPI


Warning - JobPurging not allowed
 (The Operation is not allowed: Unable to complete job purge)


================================================================================

                        JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://wms.begrid.be:9000/baN2oRxNEA3vQK6YMLsy8A
have been successfully retrieved and stored in the directory:
/data/home/jo/tmp/MPI

================================================================================

The output of this job should be something resembling this:

$ cat mpi-test.out
Modified mpirun: Executing command: mpi-start-wrapper.sh mpi-test OPENMPI
************************************************************************
UID     =  beapps028
HOST    =  node12-10a.wn.cc.kuleuven.be
DATE    =  Thu Jun 26 16:55:47 CEST 2008
VERSION =  0.0.52
************************************************************************
mpi-start [INFO   ]: search for scheduler
mpi-start [INFO   ]: activate support for pbs
mpi-start [INFO   ]: activate support for openmpi
mpi-start [INFO   ]: call backend MPI implementation
mpi-start [INFO   ]: start program with mpirun
-<START PRE-RUN HOOK>---------------------------------------------------
Compiling /user/beapp<...>/mpi-test
mpicc -m32 -o /user/beapp<...>/mpi-test /user/beapp<...>/mpi-test.c
Successfully compiled /user/beapp<...>/mpi-test
-<STOP PRE-RUN HOOK>----------------------------------------------------
==[START]=================================================================
Hello world! from processor 0 out of 16
Hello world! from processor 2 out of 16
Hello world! from processor 15 out of 16
Hello world! from processor 6 out of 16
Hello world! from processor 3 out of 16
Hello world! from processor 14 out of 16
Hello world! from processor 9 out of 16
Hello world! from processor 4 out of 16
Hello world! from processor 8 out of 16
Hello world! from processor 5 out of 16
Hello world! from processor 10 out of 16
Hello world! from processor 11 out of 16
Hello world! from processor 13 out of 16
Hello world! from processor 12 out of 16
Hello world! from processor 7 out of 16
Hello world! from processor 1 out of 16
==[FINISHED]==============================================================
-<START POST-RUN HOOK>---------------------------------------------------
Executing post hook.
Finished the post hook.
-<STOP POST-RUN HOOK>----------------------------------------------------