

		   README for MPD 1.0.0 - May, 2005


General
-------

MPD is a process management system for starting parallel jobs,
especially MPICH jobs.  Before running a job (with mpiexec), the
mpd daemons must be running on each host and connected into a ring.
This README explains how to do that and also test and manage the daemons 
after they have been started.  New in this version is the parallel
debugger mpigdb.

You need to have Python version 2.2 or later installed to run the mpd.
You can type

    which python

to make sure you have it installed, and 

    python

to find out what your version is.  The current version can be obtained
from  www.python.org.

Type 

    mpdhelp

for a list of mpd-related commands.  Each command can be run with the
--help argument for usage information.


How to use MPD
--------------

(Another version of these startup instructions can be found in the
mpich2/README at the top level of the MPICH2 distribution.)

You can start one mpd on the current host by running

   mpd &

This starts a ring of one mpd.  Other mpd's join the ring by being run
with host and port arguments for the first mpd.  You can automate this
process by using mpdboot.
 
Make a file with machine names in it.  This file may or may not include the
local machine.  It will be handy to use the default, which is ./mpd.hosts .

donner% cat ./mpd.hosts
donner.mcs.anl.gov
foo.mcs.anl.gov
shakey.mcs.anl.gov
terra.mcs.anl.gov
donner% 

After mpich is built, the mpd commands are in mpich2/bin, or the bin
subdirectory of the install directory if you have done an install.
You should put this (bin) directory in your PATH 
in your .cshrc or .bashrc, so that it will be picked up by the mpd's that
are started remotely:

Put in .cshrc:  setenv PATH /home/you/mpich2/bin:$PATH

Put in .bashrc: export PATH=/home/you/mpich2/bin:$PATH

To start some mpds, use mpdboot.  It uses the mpd.hosts file:

donner% mpdboot -n 4 
donner%

This command starts a total of 4 daemons, one on the local machine and the
rest on machines in the mpd.hosts file.  You can specify another file (-f) or
another mpd command (-m).  The mpdboot command uses ssh to start the mpd on
each machine in the mpd.hosts file.

You can use mpdtrace to see where your mpd's are running:

donner% mpdtrace
donner
foo
shakey
donner% 

You can run something with mpdrun

donner% mpdrun -np 2 hostname 
donner.mcs.anl.gov
foo.mcs.anl.gov
donner%

You can run an mpich2 job:

donner% mpiexec -np 10 /home/lusk/hellow
Hello world from process 0 of 10
Hello world from process 1 of 10
Hello world from process 2 of 10
Hello world from process 3 of 10
Hello world from process 4 of 10
Hello world from process 5 of 10
Hello world from process 6 of 10
Hello world from process 7 of 10
Hello world from process 9 of 10
Hello world from process 8 of 10
donner% 

You can take down the daemons:

donner% mpdallexit
donner%

If things go bad and daemons seem to be in a bad state, you can remove the
Unix sockets on all the machines in mpd.hosts by doing a cleanup:

donner% mpdcleanup


Here is the usage information for all of the mpd commands:

************************************************************* mpd
 
usage: mpd [--host=<host> --port=<portnum>] [--noconsole] \ 
           [--trace] [--echo] [--daemon] [--bulletproof] \ 
           [--if <interface/hostname>] [--listenport <listenport>]

Long parameter names may be abbreviated to their first letters by using
  only one hyphen and no equal sign:
     mpd -h donner -p 4268 -n
  is equivalent to
     mpd --host=donner --port=4268 --noconsole

--host and --port must be specified together; they tell the new mpd where
  to enter an existing ring;  if they are omitted, the new mpd forms a
  stand-alone ring that other mpds may enter later
--noconsole is useful for running 2 mpds on the same machine; only one of
  them will accept mpd commands
--trace yields lots of traces thru mpd routines; currently too verbose
  them can have a unix socket which a console program can connect to
--echo causes the mpd echo its listener port by which other mpds may connect
--daemon causes mpd to run backgrounded, with no controlling tty
--bulletproof says to turn bulletproofing on (experimental)
--if specifies an alternate interface/hostname for the host this mpd is running on
--listenport specifies a port for this mpd to listen on; by default it will
  acquire one from the system.

A file named .mpd.conf file must be present in the user's home directory
  with read and write access only for the user, and must contain at least
  a line with secretword=<secretword>

To run mpd as root, install it while root and instead of a .mpd.conf file
use mpd.conf (no initial dot) in the /etc directory.

This version of mpd is (0, 4, 0, 'June, 2004 release')
 
************************************************************* mpdallexit
 
usage: mpdallexit (no args)
causes all mpds in the ring to exit
 
************************************************************* mpdboot
 
usage:  mpdboot --totalnum=<n_to_start> [--file=<hostsfile>]  [--help] \ 
                [--rsh=<rshcmd>] [--user=<user>] [--mpd=<mpdcmd>] \ 
                [--loccons] [--remcons] [--shell] [--verbose] [-1]
 or, in short form, 
        mpdboot -n n_to_start [-f <hostsfile>] [-h] [-r <rshcmd>] [-u <user>] \ 
                [-m <mpdcmd>]  -s -v [-1]

--totalnum specifies the total number of mpds to start; at least
  one mpd will be started locally, and others on the machines specified
  by the file argument; by default, only one mpd per host will be
  started even if the hostname occurs multiple times in the hosts file
-1 means remove the restriction of starting only one mpd per machine; 
  in this case, at most the first mpd on a host will have a console
--file specifies the file of machines to start the rest of the mpds on;
  it defaults to mpd.hosts
--mpd specifies the full path name of mpd on the remote hosts if it is
  not in your path
--rsh specifies the name of the command used to start remote mpds; it
  defaults to ssh; an alternative is rsh
--shell says that the Bourne shell is your default for rsh
--verbose shows the ssh attempts as they occur; it does not provide
  confirmation that the sshs were successful
--loccons says you do not want a console available on local mpd(s)
--remcons says you do not want consoles available on remote mpd(s)
 
************************************************************* mpdcleanup
 
usage: mpdcleanup [-f <hostsfile>] [-r <rshcmd>] [-u <user>] [-c <cleancmd>] or
   or: mpdcleanup [--file=<hostsfile>] [--rsh=<rshcmd>] [-user=<user>] [-clean=<cleancmd>]
Removes the Unix socket on local (the default) and remote machines
This is useful in case the mpd crashed badly and did not remove it, which it normally does
 
************************************************************* mpdexit
 
usage: mpdexit mpdid
    mpdid may be obtained via mpdtrace -l (or may be "localmpd")
 
************************************************************* mpdkilljob
 
usage: mpdkilljob  jobnum  [mpdid]  # as obtained from mpdlistjobs
   or: mpdkilljob  -a jobalias      # as obtained from mpdlistjobs
    mpdid is mpd where process 0 starts
    mpdid of form 1@linux02_32996 (may need \@ in csh)
 
************************************************************* mpdlistjobs
 
usage: mpdlistjobs [-u | --user username] [-a | --alias jobalias]  [-j | --jobid jobid]
  (only use one of jobalias or jobid)
lists jobs being run by an mpd ring, all by default, or filtered
by user, mpd job id, or alias assigned when the job was submitted
 
************************************************************* mpdringtest
 
usage: mpdringtest [number of loops]
Times a single message going around the ring of mpds [num] times (default once)
 
************************************************************* mpdrun
 
mpdrun for mpd version: (0, 4, 0, 'June, 2004 release')
usage: mpdrun [args] pgm_to_execute [pgm_args]
   where args may be: -a alias -np nprocs -hf hostsfile -cpm master_copgm -cpr remote_copgm -l -m -1 -s
       (nprocs must be a positive integer)
       (-hf is a hostsfile containing names of nodes on which to run)
       (-l means attach line labels identifying which client prints each line)
       (-m means merge identical outputs into a single line;
           implies that program produces whole lines;
           implies -l)
       (-1 means do NOT start the first process locally)
       (-a means assign this alias to the job)
       (-s means send stdin to all processes; not just first)
       (-g means assume user will be running gdb and send some initial setup;
           implies -m and -l and initially -s );
or:    mpdrun -f input_xml_filename [-r output_xml_exit_codes_filename]
   where filename contains all the arguments in xml format
 
************************************************************* mpdsigjob
 
usage: mpdsigjob  sigtype  [-j jobid OR -a jobalias] [-s|g]
    sigtype must be the first arg
    jobid can be obtained via mpdlistjobs and is of the form:
        jobnum@mpdid where mpdid is mpd where first process runs, e.g.:
            1@linux02_32996 (may need \@ in some shells)
            1  is sufficient if you are on the machine where the job started
    one of -j or -a must be specified (but only one)
    -s or -g specify whether to signal the single user process or its group (default is g)
Delivers a Unix signal to all the application processes in the job
 
************************************************************* mpdtrace
 
usage: mpdtrace [-l]
Lists the (short) hostname of each of the mpds in the ring
The -l (ell) option shows full hostnames and listening ports
 
************************************************************* mpiexec
 
usage:
mpiexec [ -h   or  -help   or  --help ]
mpiexec -file filename  # where filename contains xml for job description
mpiexec -configfile filename  # where filename contains cmd-line arg-sets
mpiexec [ -default defaultArgs : ] argset : more_arg_sets : ...
    where each argset contains some of:
        -n <n> -host <h> -wdir <w> -path <p> cmd args 
    note: cmd must be specfied for each argset; it can not be a default arg
    other default arguments can be -l (line labels on stdout, stderr) and
    -setenvall (pass entire environment of mpiexec to all processes),
    -env KEY1 VALUE1 -env KEY2 VALUE2 ...
    defaultArgs are passed to all processes unless overridden
sample executions:
    mpiexec -n 1 pwd : -wdir /tmp pwd : printenv
    mpiexec -default -n 2 -wdir /bin -env RMB3=e3 : pwd : printenv

************************************************************* mpigdb

usage:
    mpigdb -a jobid # to attach to a running job (from mpdlistjobs)
    mpigdb -n nprocs pgm [pgmargs]   # to use gdb on a new job


The mpigdb parallel debugger
----------------------------

This release contains an early version of a debuggger for parallel programs
started wiht the mpd process manager.  mpigdb is an interactive version
of mpiexec that starts each application under the control of gdb and
also (initially) broadcasts each gdb command to all processes, merging
identical output from multiple processes into a single, labelled line.
It also addes one extra command, the "z" command, which can be used to
select a process or range of proccesses for input to be directed to.
With no arguments it reverts to broadcasting subsequent commands to all
processes.  This capability can be seen in the following transcript of a
session with mpigdb debugging 10 processes.  After the breakpoint at
line 30 and two single steps, process 0 is selected ("z 0") and stepped
once separately, to synchronize it with the other processes just before
the MPI_Bcast.  Then all processes are selected again ("z") and the next
"n" steps them all through the broadcast together.

Example
-------

donner% mpigdb -n 10 cpi
0-9:  (gdb) l 30
0-9:  25	
0-9:  26	    fprintf(stdout,"Process %d of %d is on %s\n",
0-9:  27		    myid, numprocs, processor_name);
0-9:  28	    fflush(stdout);
0-9:  29	
0-9:  30	    n = 10000;			/* default # of rectangles */
0-9:  31	    if (myid == 0)
0-9:  32		startwtime = MPI_Wtime();
0-9:  33	
0-9:  34	    MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
0-9:  (gdb) b 30
0-9:  Breakpoint 2 at 0x804925c: file /home/lusk/mpich2/examples/cpi.c, line 30.0-9:  (gdb) r
0-9:  Continuing.
2:  Process 2 of 10 is on donner
3:  Process 3 of 10 is on donner
4:  Process 4 of 10 is on donner
5:  Process 5 of 10 is on donner
6:  Process 6 of 10 is on donner
7:  Process 7 of 10 is on donner
8:  Process 8 of 10 is on donner
9:  Process 9 of 10 is on donner
0:  Process 0 of 10 is on donner
1:  Process 1 of 10 is on donner
0-9:  
0-9:  Breakpoint 2, main (argc=1, argv=0xbffff254)
0-9:      at /home/lusk/mpich2/examples/cpi.c:30
0-9:  30	    n = 10000;			/* default # of rectangles */
0-9:  (gdb) n  
0-9:  31	    if (myid == 0)
0-9:  (gdb) n
0:  32		startwtime = MPI_Wtime();
1-9:  34	    MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
0-9:  (gdb) z 0
0:  (gdb) n
0:  34	    MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
0:  (gdb) z
0-9:  (gdb) n
0-9:  36	    h   = 1.0 / (double) n;
0-9:  (gdb) n
0-9:  37	    sum = 0.0;
0-9:  (gdb) n
0-9:  39	    for (i = myid + 1; i <= n; i += numprocs)
0-9:  (gdb) n
0-9:  41		x = h * ((double)i - 0.5);
0-9:  (gdb) n
0-9:  42		sum += f(x);
0-9:  (gdb) n
0-9:  39	    for (i = myid + 1; i <= n; i += numprocs)
0-9:  (gdb) n
0-9:  41		x = h * ((double)i - 0.5);
0-9:  (gdb) n
0-9:  42		sum += f(x);
0-9:  (gdb) n
0-9:  39	    for (i = myid + 1; i <= n; i += numprocs)
0-9:  (gdb) n
0-9:  41		x = h * ((double)i - 0.5);
0-9:  (gdb) n
0-9:  42		sum += f(x);
0-9:  (gdb) p sum
0:  $1 = 7.9999955800048621
1:  $1 = 7.9999946200069978
2:  $1 = 7.999993500009781
3:  $1 = 7.999992220013346
4:  $1 = 7.9999907800178462
5:  $1 = 7.9999891800234542
6:  $1 = 7.9999874200303616
7:  $1 = 7.9999855000387807
8:  $1 = 7.9999834200489417
9:  $1 = 7.999981180061094
0-9:  (gdb) c
0-9:  Continuing.
0:  pi is approximately 3.1415926544231256, Error is 0.0000000008333325
1-9:  
1-9:  Program exited normally.
1-9:  (gdb) 0:  wall clock time = 84.966523
0:  
0:  Program exited normally.
0:  (gdb) q
0-9:  MPIGDB ENDING
donner% 
