ISSM with PETSc and AD

mieke70410

Hi,
is there already a way how to compile ISSM with PETSc and Automatic Differentiation? I would like to solve a system of nonlinear equations iteratively, with the Jacobian provided by AD and the linear system solved iteratively as well.
Or is there another way to use AD with an iterative linear solver than with PETSc in ISSM?

mathieumorlighem

Hello Marieke

that's an excellent question. PETSc does not support AD because we cannot overload IssmDouble to active doubles within PETSc and so we cannot use its solvers or data structures (Mat/Vec) explicitly. But we can use some of the solvers and libraries provided by PETSc, typically MUMPS. For example, here is my configure.sh file for an Ubuntu machine:

./configure \
   --prefix=$ISSM_DIR\
   --without-kriging \
   --without-kml \
   --without-Love \
   --without-Sealevelchange \
   --with-matlab-dir="/usr/local/MATLAB/R2021b" \
   --with-mpi-include=$ISSM_DIR/externalpackages/petsc/install/include  \
   --with-mpi-libflags="-L$ISSM_DIR/externalpackages/petsc/install/lib -lmpi -lmpicxx -lmpifort"\
   --with-metis-dir=$ISSM_DIR/externalpackages/petsc/install \
   --with-mumps-dir=$ISSM_DIR/externalpackages/petsc/install \
   --with-blas-lapack-dir=$ISSM_DIR/externalpackages/petsc/install \
   --with-scalapack-dir=$ISSM_DIR/externalpackages/petsc/install \
   --with-triangle-dir=$ISSM_DIR/externalpackages/triangle/install \
   --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \
   --with-codipack-dir="$ISSM_DIR/externalpackages/codipack/install" \
   --with-medipack-dir="$ISSM_DIR/externalpackages/medipack/install" \
   --with-cxxoptflags="-g -O3 -fPIC -std=c++11 -DCODI_ForcedInlines" \
   --with-numthreads=32 \
   --enable-tape-alloc \
   --enable-development \
   --enable-debugging

You can see that I use lots of libraries compiled by PETSc (MPI, MUMPS, BLAS/LAPACK, etc). Once you want to run a model, you need to specify that you want the "ISSM toolkit" (which means that you do not want to use PETSc's Mat and Vec for vectors and matrices) and you can still request to use MUMPS. For example:

   md.toolkits.DefaultAnalysis=issmmumpssolver();
   md.toolkits.RecoveryAnalysis=issmmumpssolver();

This is a bit technical but hopefully it makes sense. Let me know if you need help!
Mathieu

mieke70410

Thank you very much for your quick reply. I have already compiled and used ISSM with AD and MUMPS, but the problem is that I need an iterative linear solver instead of MUMPS or GSL-LU for my performance tests.

But if I understand you correctly, there is no way to have an iterative solver and AD in ISSM yet? For example GMRES?

mathieumorlighem

You are right... we have not coded interfaces for other solvers yet, primarily because the linear solver is not a bottleneck with AD most of the time. If you look at the "Linear solver elapsed time" you will see that it is generally <5% of the overall compute time. For example, this is for a typical run with 20,000 elements, 5 year transient:

   FemModel initialization elapsed time:   0.360925
   Total Core solution elapsed time:       8483.61
   Linear solver elapsed time:             49.8899 (0.59%)

   Total elapsed time: 2 hrs 21 min 23 sec

The linear solver took ₀.6% of the total elapsed time, so we figured we don't need to add support for other solvers. But let me know if this is not the case for you and I can show you how to do it if you are interested.
All the best
Mathieu

mieke70410

Ok that is good to know.

I just want to test a method that combines AD efficiently with preconditioning to solve systems of nonlinear equations faster and less memory consuming. Therefore I need a good test example and I wanted to see if this is possible with ISSM.

Thank you Mathieu for your great help and fast response.
All the best
Marieke