PETSC Error on ComputeCanada

ahayden

Hi all,

I have just installed ISSM on the Graham cluster of Compute Canada, no issues with the install. However, when I try to submit a simple example (SquareIceShelf), the run fails (slurm error 59) and the contents of the errlog file are as follows:

[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on exactly when Open MPI kills them.
--------------------------------------------------------------------------
slurmstepd: error: *** STEP 22237762.0 ON gra311 CANCELLED AT 2024-06-17T16:16:46 ***
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
srun: error: gra311: task 0: Exited with exit code 59
srun: Terminating StepId=22237762.0

I tried running the model again, but the same error occurred. I also tried with the Greenland SeaRise example, and the same error comes up. Any ideas on what might be causing this issue?

justinquinn

ahayden Did you use OpenMPI when compiling PETSc? If you configured PETSc with --download-mpich=1, it may be causing a conflict. Also see our development wiki, which implies that PETSc is already available on Compute Canada.

ahayden

justinquinn
Hi Justin,

I didn't download PETSc from external packages since it was already on the cluster.

This is what my configure file looks like:

./configure \
--prefix=$ISSM_DIR \
--with-numthreads=32 \
--with-matlab-dir="$EBROOTMATLAB" \
--with-mkl-libflags="-lmkl_intel_ilp64 -lmkl_sequential -lmkl_core -lmkl_blacs_openmpi_lp64 -lmkl_scalapack_lp64" \
--with-petsc-dir="$EBROOTPETSC" \
--with-mumps-dir="$EBROOTPETSC" \
--with-chaco-dir="$ISSM_DIR/externalpackages/chaco/install" \
--with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \
--with-mpi-include="/some/path/just/to/enable" \
--with-mpi-libflags="-lmpi_cxx -lmpi_mpifh -lmpi" \
--with-fortran-lib="-L$EBROOTGCC/12/ -lgfortran"

Anna

justinquinn

ahayden You might also try reducing the number of threads to something like 4 just as a debugging step.

mathieumorlighem

Hi Anna

segmentation faults are difficult to debug because we don't get a ton of information about what's going on. Do you see anything in the outlog? If you could turn md.verbose to md.verbose=verbose('all'); and tell us everything you see printed to the screen that would be a good start.
Cheers
Mathieu

ahayden

mathieumorlighem

Hi Mathieu,

The outlogs are empty. Perhaps ISSM isn't even launching? Going to try and reduce the number of threads to see what happens. Will report back!

Cheers,

Anna

ahayden

justinquinn

The error file is the same, and the outlog is still empty after reducing the number of threads.

justinquinn

ahayden It actually looks like at least some of the slurm errors are percolating up,

slurmstepd: error: *** STEP 22237762.0 ON gra311 CANCELLED AT 2024-06-17T16:16:46 ***
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
srun: error: gra311: task 0: Exited with exit code 59
srun: Terminating StepId=22237762.0

You may want to contact the system administrators as well to see if they can help with debugging. There may be more to the slurm logs.