PnMPI Tool Infrastructure

by Martin Schulz, schulzm@llnl.gov, LLNL-CODE-402774

PnMPI is a dynamic MPI tool infrastructure that builds on top of the standardized PMPI interface. It allows the user to

run multiple PMPI tools concurrently
activate PMPI tools without relinking by just changing a configuration file
multiplex toolsets during a single run
write cooperative PMPI tools

The package contains two main components:

The PnMPI core infrastructure
Tool modules that can explicitly exploit PnMPI's capabilities

So far, this software has mainly been tested on Linux clusters with RHEL-based OS distributions as well as IBM's BG/P systems. Continuous integration tests run at Ubuntu 12.04 and 14.04, OSX El Capitan and macOS Sierra. Some preliminary experiments have also included SGI Altix systems. Ports to other platforms should be straightforward, but this is not extensively tested. Please file an issue if you run into problems porting PnMPI or if you successfully deployed PnMPI on a new platform.

Many thanks to our contributors.

A) Building PnMPI

PnMPI uses CMake for its build system.

A1) Dependencies

CMake (at least version 2.8.11.2 is required).
argp is usually included in glibc. However at macOS you have to install it from homebrew as argp-standalone package.
binutils and libiberty for the patcher utility (optional)
Doxygen for generating the docs and man-pages. (optional)
help2man for generating man-pages. (optional)
An MPI implementation with header files. So far, PnMPI has been tested with MPICH, OpenMPI and IntelMPI.

In addition, PnMPI uses git submodules for several CMake modules, wrap and adept-utils. While the deploy source tar-ball includes all required submodules, git users need to checkout them with the following command in the root of the cloned repository:

git submodule update --init --recursive

A2) Configure the project

In the simplest case, you can run this in the top-level directory of the PnMPI tree:

cmake -DCMAKE_INSTALL_PREFIX=/path/to/install/destination
make
make install

This will configure, build, and install PnMPI to the destination specified. PnMPI supports parallel make with the -j parameter. E.g., for using eight build tasks, use:

cmake -DCMAKE_INSTALL_PREFIX=/path/to/install/destination
make -j8
make install

On more complex machines, such as those with filesystems shared among multiple platforms, you will want to separate out your build directories for each platform. CMake makes this easy.

Create a new build directory named according to the platform you are using, cd into it, an run cmake there. For example:

cd <pnmpi>
mkdir x86_64
cd x86_64
cmake -DCMAKE_INSTALL_PREFIX=/path/to/install/destination ..

Here, <pnmpi> is the top-level directory in the PnMPI tree. Note that when you run CMake this way, you need to supply the path to the PnMPI source directory as the last parameter. Here, that's just .. as we are building in a subdirectory of the source directory. Once you run CMake, simply run make and make install as before:

make -j8
make install

The PnMPI build should auto-detect your MPI installation and determine library and header locations. If you want to build with a particular MPI that is NOT the one auto-detected by the build, you can supply your particular MPI compiler as a parameter:

cmake \
  -DCMAKE_INSTALL_PREFIX=/path/to/install/destination \
  -DMPI_C_COMPILER=/path/to/my/mpicc \
  ..

See the documentation in FindMPI.cmake for more details on MPI build configuration options.

If you have problems, you may want to build PnMPI in debug mode. You can do this by supplying an additional parameter to cmake, e.g.:

cmake \
  -DCMAKE_INSTALL_PREFIX=/path/to/install/destination \
  -DCMAKE_BUILD_TYPE=Debug \
  ..

The extra/build directory contains a few sample invocations of CMake that have been successfully used on LLNL systems.

A3) Configuring with/without Fortran

By default PnMPI is configured to work with C/C++ and Fortran codes. However, on systems where Fortran is not available, the system should auto-detect this and not build the Fortran libraries and test cases. It can also be manually turned off by adding

-DENABLE_FORTRAN=OFF

to the cmake configuration command.

The PnMPI distribution contains test cases for C and Fortran that allow you to test the correct linkage.

A3a) Optional configuration options

If you want to change the default build configuration, you can enable / disable features by adding the following flags to the cmake configuration command.

-DBUILD_DOC=ON: Generate the public code documentation and man-pages. PnMPI developers may enable BUILD_DOC_INTERNAL instead for additional PnMPI internal documentation. (requires Doxygen and help2man)
-DENABLE_DEBUG=OFF: Disable PnMPI on-demand debug logging. Disable this option only for performance optimization.
-DENABLE_MODULES=OFF: Don't build the built-in modules.
-DENABLE_TESTING=ON: Build the test cases and enable the test target.
-DENABLE_THREAD_SAFETY=OFF: Build PnMPI without thread safety. PnMPI will limit the MPI threading level if this option is enabled, so it still will be thread safe. Disable this option only for performance optimization.
-DENABLE_PNMPIZE=OFF: Don't build the PnMPI invocation tool and don't run any tests for it. This option may be required when testing PnMPI at a platform that doesn't support the execvp() syscall.
-DENABLE_ADEPT_UTILS=ON: Enable support for adept-utils, to check the module's symbols for their origin. (Not supported with all compilers.)
CMake-codecov provides the additional -DENABLE_COVERAGE=ON option to enable code coverage. You can generate coverage reports with the gcov and lcov targets. Read their README for more information.
cmake-sanitizers provides additional options to enable sanitizers for all PnMPI targets. E.g. for AddressSanitizer add -DSANITIZE_ADDRESS=ON. Read their README for more information.

A4) Configuring for cross-compiled environments

When configuring PnMPI in cross-compiled environments (such as Blue Gene/Q systems), it is necessary to provide a matching tool chain file. Many toolchain files are included in CMake, additional example files that allow the compilation on certain LC machines can be found in cmakemodules/Platform and cmakemodules/Toolchain.

For example, to configure PnMPI for a BG/Q machine using the GNU compiler suite, add the following to the cmake configuration command:

-DCMAKE_TOOLCHAIN_FILE=../cmakemodules/Toolchain/BlueGeneQ-gnu.cmake

You may need to modify the toolchain file for your system.

A5) Installed structure

Once you've installed, all your PnMPI files and executables should be in <CMAKE_INSTALL_PREFIX>, the path specified during configuration. Roughly, the install tree looks like this:

bin/
  pnmpi                PnMPI invocation tool
  pnmpi-patch          Library patching utility
lib/
  libpnmpi[f].[so,a]   PnMPI runtime libraries
  pnmpi-modules/       System-installed tool modules
  cmake/               Build files for external modules
include/
  pnmpi/               PnMPI header directory
    debug_io.h         PnMPI module debug print functions.
    hooks.h            PnMPI module hook definitions.
    service.h          PnMPI module service functions.
    ...
  pnmpi.h              PnMPI main header
  pnmpimod.h           PnMPI module support (legacy)
  pnmpi-config.h       CMake generated configuration file
share/
  cmake/               CMake files to support tool module builds

Test programs are not installed, but in the tests/src folder of the build directory, there should also be test programs built with PnMPI. See below for details on running these to test your PnMPI installation.

A6) Environment setup

You will need to set one environment variable to run PnMPI:

PNMPI_LIB_PATH should be set to the full path to the PnMPI modules directory. If not set, the default path of your installation directory will be used.
PNMPI_CONF should be set to the full to the path of the PnMPI configuration file to be used.
PNMPI_BE_SILENT will silence the PnMPI banner. For benchmark purposes you should also disable ENABLE_DEBUG in your CMake configuration. Note:

PnMPI

Install / Use

README