                      MPE (Multi-Processing Environment)
                      ----------------------------------

                          Version 2.2.0. May, 2005

                    Mathematics and Computer Science Division
                           Argonne National Laboratory

I.  INTRODUCTION
----------------

The Multi-Processing Environment (MPE) attempts to provide programmers with 
a complete suite of performance analysis tools for their MPI programs based
on post processing approach.  These tools include a set of profiling libraries, 
a set of utility programs, and a set of graphical tools.

The first set of tools to be used with user MPI programs is profiling libraries
which provide a collection of routines that create log files.  These log files
can be created manually by inserting MPE calls in the MPI program, or 
automatically by linking with the appropriate MPE libraries, or by combining 
the above two methods.  Currently, the MPE offers the following 4 profiling 
libraries.

   1) Tracing Library - Traces all MPI calls.  Each MPI call is preceded by a 
      line that contains the rank in MPI_COMM_WORLD of the calling process, 
      and followed by another line indicating that the call has completed.
      Most send and receive routines also indicate the values of count, tag,
      and partner (destination for sends, source for receives).  Output is to
      standard output.

   2) Animation Libraries - A simple form of real-time program animation 
      that requires X window routines.  

   3) Logging Libraries - The most useful and widely used profiling libraries 
      in MPE.  These libraries form the basis to generate log files from 
      user MPI programs.  There are several different log file formats
      available in MPE.  The default log file format is CLOG2.  It is a low
      overhead logging format, a simple collection of single timestamp events.
      The old format ALOG, which is not being developed for years, is not
      distributed here.  The powerful visualization format is SLOG-2, stands 
      for Scalable LOGfile format version II which is a total redesign of the
      original SLOG format.  SLOG-2 allows for much improved scalability for 
      visualization purpose.  CLOG2 file can be easily converted to 
      SLOG-2 file through the new SLOG-2 viewer, Jumpshot-4.

   4) Collective and datatype checking library - An argument consistency 
      checking library for MPI collective calls.  It checks for datatype, root,
      and various argument consistency in MPI collective calls.

The set of utility programs in MPE includes log format converter (e.g. 
clogTOslog2), logfile print (e.g. slog2print) and logfile viewer and
convertor (e.g. jumpshot).  These new tools, clog2TOslog2, slog2print and
jumpshot(Jumpshot-4) replace old tools, clog2slog, slog_print and logviewer
(i.e. Jumpshot-2 and Jumpshot-3).  For more information of various
logfile formats and their viewers, see 

http://www.mcs.anl.gov/perfvis



II. CONFIGURATION
-----------------

MPE can be configured and installed as an extension to most MPI standard
-compliant MPI implementations, e.g. MPICH-2, MPICH, LAM/MPI, SGI's MPI,
HP-UX's MPI and IBM's MPI.  It has been integrated seamlessly into 
MPICH and MPICH-2 distributions, so MPE will be installed automatically
during MPICH-X's installation process.  

For details of configuration of MPE, see the INSTALL file.

III. INSTALLATION INSTRUCTIONS
-------------------------------

For details of installation instruction/examples of MPE, see the INSTALL file.



IV.  EXAMPLE PROGRAMS
----------------------

As previously noted, the MPE library is composed of 3 different profiling
libraries.  During configure, the compiler's library linkage flags and 
appropriate libraries are determined.  These variables are first substituted
in the Makefiles in the directories, mpe2/src/wrappers/test,
mpe2/src/graphics/contrib/test and mpe2/src/collchk/test.  The Makefiles for
mpe2/src/wrappers/test and mpe2/src/graphics/contrib/test are then installed
into directory share/ as examples_logging/ and examples_graphics/ during the 
final installation process.  The following are some of the crucial variables:

LOG_LIBS     = library flag that links with the logging libraries
TRACE_LIBS   = library flag that links with the tracing library
ANIM_LIBS    = library flag that links with the animation library
COLLCHK_LIBS = library flag that links with the collective and datatype
               checking library

The variable FLIB_PATH is the compiler's library path needed to link fortran 
MPI programs with the logging library.

During make, small test programs cpi.c, cpilog.c and fpilog.f will be linked
with each of the above libraries.  In the output from Make, a message will be 
printed to indicate the status of each attempted link test.  The success 
of these linkage tests will also be included in the Make output.  If the
linkage tests are successful, then these library linkage flags can be used
for your programs as well.   

The following example programs are also included in the distribution:

    mpe/src/graphics/contrib/mandel is a Mandelbrot program that uses the MPE 
    graphics package.

    mpe/src/graphics/contrib/mastermind is a program for solving the Mastermind
    puzzle in parallel.

These programs should work on all MPI implementations, but have not been
extensively tested.





V.  MPEINSTALL
--------------

A 'mpeinstall' script is created during configuration.  If configuring with
MPICH and MPICH2, then the 'mpiinstall' script will invoke the 'mpeinstall'
script.  However, 'mpeinstall' can also be used by itself.  This is only 
optional and is of use only if you wish to install the MPE library in a 
public place so that others may use it.  Final install directory will 
consist of an include, lib, bin, sbin and share subdirectories.  Examples 
and various logfile viewers will be installed under share.





VI. USAGE
---------

The final install directory contains the following subdirectories.

    include/ contains all the include files that user program needs to read.
    lib/     contains all the libraries that user program needs to link with.
    bin/     contains all the utility programs that user needs to use.
    doc/     contains available MPE documentation, e.g. Jumpshot-4's userguide.
    sbin/    contains the MPE uninstall script to uninstall the installation.
    share/   contains user read-only data.  Besides share/examples_logging/
             and share/examples_graphics/, user usually does NOT need to know
             the details of other subdirectories.

In terms of usage of MPE, user usually only need to know about the files
that have been installed in include/, lib/ and bin/.  



VI. a) CUSTOMIZING LOGFILES
---------------------------

In addition to using the predefined MPE logging libraries to log all MPI 
calls, MPE logging calls can be inserted into user's MPI program to define
and log states.  These states are called User-Defined states.  States may 
be nested, allowing one to define a state describing a user routine that 
contains several MPI calls, and display both the user-defined state and 
the MPI operations contained within it.

The routine MPE_Log_get_event_number() has to be used to get unique
event numbers (this is important if you are writing a library that uses
the MPE logging routines) from the MPE system.  The routines
MPE_Describe_state() and MPE_Log_event() are then used to describe 
user-defined states.

\begin{verbatim}

int eventID_begin, eventID_end;
...
eventID_begin = MPE_Log_get_event_number();
eventID_end   = MPE_Log_get_event_number();
...
MPE_Describe_state( eventID_begin, eventID_end, "Multiplication", "red" );
...
MyAmult( Matrix m, Vector v )
{
    /* Log the start event along with the size of the matrix */
    MPE_Log_event( eventID_begin, 0, NULL );
    ... Amult code, including MPI calls ...
    MPE_Log_event( eventID_end, 0, NULL );
}

\end{verbatim}

The logfile generated by this code will have the MPI routines within the
routine MyAmult() indicated by a containing bluegreen rectangle.

If the MPE logging library, liblmpe.a, are NOT linked with the user program, 
MPE_Init_log() and MPE_Finish_log() need to be used before and after all 
the MPE calls.   Sample programs cpilog.c and fpi.f are available in MPE 
source directory src/logging/test or the installed directory 
share/examples_logging to illustrate the use of these MPE routines.  

For undefined user-defined state, i.e. corresponding MPE_Describe_state() 
has not been issued, new jumpshot (Jumpshot-4) may display the legend name as
"UnknownType-INDEX" where INDEX is the internal MPE category index.



VI. b) EXAMPLE MAKEFILE 
-----------------------

The install directories, share/examples_logging and share/example_graphics,
contain some very useful and simple example programs.  The Makefiles in these
directories illustrate the usage of MPE routines and the linkage of MPE 
libraries to generate logfiles. In most cases, users can simply copy the 
share/examples_logging/Makefile to their home directory, and do a "make" to 
compile the suggested targets.  Users don't need to copy the .c and .f files
when MPE has been compiled with a MAKE that has VPATH support.  The created 
executables can be launched with mpirun from the MPI implementation to 
generate sample logfiles.



VI. c) ENVIRONMENTAL VARIABLES
------------------------------

There are 2 environmental variables, TMPDIR and MPE_LOGFILE_PREFIX, that user 
may need to set before the generation of logfiles :

CLOG_BLOCK_SIZE: The integer value determines the clog2 buffer block size
                 which set the least minimum clog2 file size.  If
                 CLOG_BLOCK_SIZE is not set, 64K per block is assumed.

CLOG_BUFFERED_BLOCKS: The integer value determines the number of blocks
                      witin the CLOG-2.1's internal buffer.  Together with
                      CLOG_BLOCK_SIZE, CLOG_BUFFERED_BLOCKS determines how
                      often the internal buffer is flushed to the disk.  
                      The total buffer size is determined by the product of
                      CLOG_BLOCK_SIZE and CLOG_BUFFERED_BLOCKS.  These 2
                      environmental variables allows user to minimize MPE2
                      logging overhead when large local memory is available.
                      The default value is 128.

MPE_TMPDIR: MPE_TMPDIR takes precedence over TMPDIR.  It specifies a
            directory to be used as temporary storage for each process.
            By default, when MPE_TMPDIR and TMPDIR are NOT set,
            /tmp will be used.  When user needs to generate a very large
            logfile for long-running MPI job, user needs to make sure that
            MPE_TMPDIR(or TMPDIR) is big enough to hold the temporary local
            logfile which will be deleted if the merged logfile can be
            created successfully.  In order to minimize the overhead of the
            logging to the MPI program, it is highly recommended user to
            use a *local* file system for TMPDIR.

             Note : The final merged logfile will be written back to the 
                    file system where process 0 is.

MPE_DELETE_LOCALFILE:  The boolean value determines whether to delete local
                       clog2 file.  This allows user to launch a separate
                       process to merge the local clog2 files in case 
                       MPI_Finalize() fails due to user program overwritten
                       to MPE/MPI internal data structures
                       (No clog_mpimeger yet)

MPE_CLOCKS_SYNC: The boolean value determines the behavior of
                 MPE_Log_sync_clocks()

MPE_LOGFILE_PREFIX: specifies the filename prefix of the output logfile.
                    The file extension will be determined by the output
                    logfile format, i.e. MPE_LOG_FORMAT.

MPE_LOG_FORMAT: determines the format of the logfile generated from the 
                execution of application linked with MPE logging libraries.  
                The allowed value for MPE_LOG_FORMAT is CLOG2 only.
                So there is no need to use this variable at the moment.

MPE_LOG_OVERHEAD: The boolean value determines to log MPE/CLOG2's internal
                  profiling state CLOG_Buffer_write2disk(). The default
                  setting is yes.  CLOG_Buffer_write2disk labels region
                  in each process that MPE/CLOG2 spends on flushing logging
                  data in the memory to the disk.  The frequency and location
                  of CLOG_Buffer_write2disk state can be altered by changing
                  CLOG_BLOCK_SIZE and/or CLOG_BUFFERED_BLOCKS.

Possible boolean values are "true", "false", "yes" and "no" in either
all lower or upper cases.



VI. d) UTILITY PROGRAMS
-----------------------

In bin/, user can find several useful utility programs when manipulating 
logfiles.  These includes log format converters, log format print programs,
and logfile display program,


Log Format Converters
---------------------

clog2TOslog2 : a CLOG2 to SLOG-2 logfile convertor.  For more details,
              do "clog2TOslog2 -h".

rlogTOslog2 : a RLOG to SLOG-2 logfile convertor.  For more details,
              do "rlogTOslog2 -h".  Where RLOG is an internal MPICH2 logging
              format.

logconvertor : a standalone GUI based convertor that invokes clog2TOslog2
               or rlogTOslog2 based on logfile extension.  The GUI also
               shows the progress of the conversion.  The same convertor
               can be invoked from within the logfile viewer, jumpshot.

slog2filter : a SLOG-2 to SLOG-2 logfile convertor.  It allows for removal
              unwanted categories (when used with slog2print -c).  It also
              allows for changing of the SLOG-2 internal structure, e.g.
              modify the duration of preview drawable.  The tool reads
              and writes SLOG-2 file of same version.

slog2updater: a SLOG-2 file format update utility.  It is essentially
              a slog2filter that reads in older SLOG-2 file and writes
              out the latest SLOG-2 file format.


Log Format Print Programs
-------------------------

clog2_print : a stdout print program for CLOG file.  Java version is clogprint.

rlog_print  : a stdout print program for SLOG-2 file.

slog2print  : a stdout print program for SLOG-2 file.



Log File Display Program 
------------------------

jumpshot : the Jumpshot-4 launcher script.  Jumpshot-4 does logfile
           conversion as well as visualization.

To view a logfile, say fpilog.slog2, do

jumpshot fpilog.slog2

The command will select and invoke Jumpshot-4 to display the content
of SLOG-2 file if Jumpshot-4 has been built and installed successfully.

One can also do

jumpshot fpilog.clog2

or

jumpshot barrier.rlog

Both will invoke the logfile convertor first before visualization.


Collective and Datatype Checking
--------------------------------

Linking an MPI application with the collective and datatype checking library
as follows

mpicc -o mpi_pgm *.o -L<mpe2_libdir> -lmpe_collchk.






VII. Using MPE in MPICH
-----------------------

MPE has been seamlessly integrated into MPICH and MPICH2 distributions, 
user may find it easier to use MPE when using it with MPICHx.  Here are 
the differences of using MPE with MPICH and with other MPI implementations.



VII. a) Compilation and Linkage
-------------------------------
MPICH (not MPICH2) provides scripts to help users to compile and link C/C++ 
and F77/F90 programs.  They are mpicc for C programs, mpiCC for C++ programs,
mpif77 for F77 and mpif90 for F90 programs.  In addition, these 4 scripts
allows special options to be used to link with MPE profiling libraries.
These options are :

-mpitrace   - to compile and link with tracing library.
-mpianim    - to compile and link with animation libraries.
-mpilog     - to compile and link with logging libraries.

For instance, the following command creates executable, {\tt fpilog}, which
generates logfile when it is executed.

mpif77 -mpilog -o fpilog fpilog.f 

For other MPI implementations(including MPICH2), user needs to compile and
link their application with MPE profiling libraries explicitly as shown in
the example makefile.



VII. b) Inheritance of Environmental Variables
----------------------------------------------
MPE relies on certain environmental variables (e.g. TMPDIR).  These
variables determine how MPE behaves.  It is important to make sure that
all the MPI processes receive the intended value of environmental variables.
The complication of this issue comes from the fact that different MPI
implementations have different way of passing environmental varaiable.  For
instance, MPICH contains many different devices for different platforms,
some of these devices have their unique way of passing of environmental
variables to other processes.  The often used devices, like ch_p4 and ch_shmem,
do not require special attention to pass the value of the environmental 
variable to spawned processes.  The spawned process inherits the value from
the launching process when the environmental variable in the launching
process has been set.  But this is NOT true for all the devices, for instance,
the ch_p4mpd device requires special option of mpirun to set environmental
variables to all processes.

mpirun -np N fpilog -MPDENV- MPE_LOGFILE_PREFIX=fpilog

In this example, the option -MPDENV- is needed to make sure
that all processes have their environmental variable, MPE_LOGFILE_PREFIX,
set to the desirable output logfile prefix.

In MPICH-2 using MPD as a launcher, passing MPE_LOGFILE_PREFIX and
TMPDIR can be done as follows:

mpiexec -default -env MPE_LOGFILE_PREFIX <output-logname-prefix> \
                 -env TMPDIR <local-tmp-dir> : -n 32 <executable-name>

For other MPI implementations, how environmental variables are passed
remains unchanged.  User needs to get familar with the environment and
set the environmental variables accordingly.



VII. c) Viewing Logfiles
------------------------
MPE's install directory structure is the same as MPICH's and MPICH-2's.
So all MPE's utility programs will be located in the bin/ directory of 
MPICH and MPICH-2.

