ISSM & Petsc with AD

ahck

Hi,

I am trying to compile the ISSM AD enabled version using ADOL-C and AMPI. Previously, I got the non-AD version to compile&run successfully on my system.

With AD enabled, I get the following error related to a type mismatch between ISSMDouble and ISSMPDouble for the PetscMat class constructor:

In file included from ./datastructures/../classes/./../toolkits/./objects/toolkitobjects.h, from ./datastructures/../classes/./../toolkits/toolkits.h, from ./datastructures/../classes/./Vertex.h, from ./datastructures/../classes/classes.h, from ./datastructures/DataSet.cpp: ./datastructures/../classes/./../toolkits/./objects/./Matrix.h: In constructor 'Matrix<doubletype>:(IssmPDouble*, int, int, IssmPDouble)': ./datastructures/../classes/./../toolkits/./objects/./Matrix.h: error: no matching function for call to 'PetscMat:(IssmPDouble*&, int&, int&, IssmPDouble&)' this->pmatrix=new PetscMat(serial_mat,M,N,sparsity); ^ ./datastructures/../classes/./../toolkits/./objects/./Matrix.h: note: candidates are: In file included from ./datastructures/../classes/./../toolkits/./petsc/objects/petscobjects.h, from ./datastructures/../classes/./../toolkits/./petsc/petscincludes.h, from ./datastructures/../classes/./../toolkits/toolkits.h, from ./datastructures/../classes/./Vertex.h, from ./datastructures/../classes/classes.h, from ./datastructures/DataSet.cpp: ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: PetscMat:(int, int, int, int) <near match> PetscMat(int M,int N,int connectivity,int numberofdofspernode);

Taking a look at the Matrix class I can see the following constructor related call:


Matrix(IssmPDouble* serial_mat,int M,int N,IssmPDouble sparsity){/*{{{*/
  InitCheckAndSetType();
  if(type==PetscMatType){
  #ifdef _HAVE_PETSC_
    this->pmatrix=new PetscMat(serial_mat,M,N,sparsity); //Line 92
  #endif
     }

As you can see, serial_mat is a passive double array. However, the PetscMat constructor actually expects an active adouble* array (cf. toolkits/petsc/objects):


PetscMat:(IssmDouble* serial_mat,int M,int N,IssmDouble sparsity)

The config I used looks as follows:

./configure \ --prefix=$ISSM_DIR \ --with-matlab-dir="/shared/apps/matlab/R2015b/" \ --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \ --with-mpi-include="$ISSM_DIR/externalpackages/mpich/install/include" \ --with-mpi-libflags="-L$ISSM_DIR/extenalpackages/mpich/install/lib/ -lmpich" \ --with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-metis-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-blas-lapack-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \ --with-numthreads=16 \ --with-fortran-lib="/shared/apps/gcc/4.8.5/lib/../lib64/libgfortran.la" \ --with-adolc-dir="$ISSM_DIR/externalpackages/adolc/install" \ --with-ampi-dir="$ISSM_DIR/externalpackages/adjoinablempi/install" \ --without-kriging \ --without-Gia \ --without-kml \ --with-boost-dir="$ISSM_DIR/externalpackages/boost/install" \

I am not sure, how to proceed here.

EDIT:

For completeness, here is the full output of the compilation:

make V=1 make all-recursive make: Entering directory `/home/a_h_ck/issm_project/issm_p' Making all in src make: Entering directory `/home/a_h_ck/issm_project/issm_p/src' Making all in c make: Entering directory `/home/a_h_ck/issm_project/issm_p/src/c' /bin/sh ../../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../. -I/home/a_h_ck/issm_project/issm_p/externalpackages/petsc/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/adjoinablempi/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/mpich/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/petsc/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/triangle/install/ -I/home/a_h_ck/issm_project/issm_p/externalpackages/adolc/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/boost/install/include -g -O2 -std=c++11 -g -O2 -fPIC -g -O2 -std=c++11 -MT ./datastructures/libISSMCore_la-DataSet.lo -MD -MP -MF ./datastructures/.deps/libISSMCore_la-DataSet.Tpo -c -o ./datastructures/libISSMCore_la-DataSet.lo `test -f './datastructures/DataSet.cpp' || echo './'`./datastructures/DataSet.cpp libtool: compile: g++ -DHAVE_CONFIG_H -I. -I../../. -I/home/a_h_ck/issm_project/issm_p/externalpackages/petsc/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/adjoinablempi/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/mpich/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/petsc/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/triangle/install/ -I/home/a_h_ck/issm_project/issm_p/externalpackages/adolc/install/include -I/home/a_h_ck/issm_project/issm_p/externalpackages/boost/install/include -g -O2 -std=c++11 -g -O2 -fPIC -g -O2 -std=c++11 -MT ./datastructures/libISSMCore_la-DataSet.lo -MD -MP -MF ./datastructures/.deps/libISSMCore_la-DataSet.Tpo -c ./datastructures/DataSet.cpp -fPIC -DPIC -o ./datastructures/.libs/libISSMCore_la-DataSet.o In file included from ./datastructures/../classes/./../toolkits/./objects/toolkitobjects.h, from ./datastructures/../classes/./../toolkits/toolkits.h, from ./datastructures/../classes/./Vertex.h, from ./datastructures/../classes/classes.h, from ./datastructures/DataSet.cpp: ./datastructures/../classes/./../toolkits/./objects/./Matrix.h: In constructor 'Matrix<doubletype>:(IssmPDouble*, int, int, IssmPDouble)': ./datastructures/../classes/./../toolkits/./objects/./Matrix.h: error: no matching function for call to 'PetscMat:(IssmPDouble*&, int&, int&, IssmPDouble&)' this->pmatrix=new PetscMat(serial_mat,M,N,sparsity); ^ ./datastructures/../classes/./../toolkits/./objects/./Matrix.h: note: candidates are: In file included from ./datastructures/../classes/./../toolkits/./petsc/objects/petscobjects.h, from ./datastructures/../classes/./../toolkits/./petsc/petscincludes.h, from ./datastructures/../classes/./../toolkits/toolkits.h, from ./datastructures/../classes/./Vertex.h, from ./datastructures/../classes/classes.h, from ./datastructures/DataSet.cpp: ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: PetscMat:(int, int, int, int) <near match> PetscMat(int M,int N,int connectivity,int numberofdofspernode); ^ ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: no known conversion for argument 1 from 'IssmPDouble* {aka double*}' to 'int' ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: PetscMat:(IssmDouble*, int, int, IssmDouble) PetscMat(IssmDouble* serial_mat,int M,int N,IssmDouble sparsity); ^ ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: no known conversion for argument 1 from 'IssmPDouble* {aka double*}' to 'IssmDouble* {aka adouble*}' ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: PetscMat:(int, int, int, int, int*, int*) PetscMat(int m,int n,int M,int N,int* d_nnz,int* o_nnz); ^ ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: candidate expects 6 arguments, 4 provided ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: PetscMat:(int, int, IssmDouble) PetscMat(int M,int N,IssmDouble sparsity); ^ ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: candidate expects 3 arguments, 4 provided ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: PetscMat:(int, int) PetscMat(int M,int N); ^ ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: candidate expects 2 arguments, 4 provided ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: PetscMat:() PetscMat(); ^ ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: candidate expects 0 arguments, 4 provided ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: constexpr PetscMat:(const PetscMat&) class PetscMat{ ^ ./datastructures/../classes/./../toolkits/./petsc/objects/./PetscMat.h: note: candidate expects 1 argument, 4 provided In file included from ./datastructures/../classes/./../toolkits/./objects/toolkitobjects.h, from ./datastructures/../classes/./../toolkits/toolkits.h, from ./datastructures/../classes/./Vertex.h, from ./datastructures/../classes/classes.h, from ./datastructures/DataSet.cpp: ./datastructures/../classes/./../toolkits/./objects/./Vector.h: In instantiation of 'doubletype* Vector<doubletype>:() ': ./datastructures/../classes/./ExternalResults/GenericExternalResult.h: required from here ./datastructures/../classes/./../toolkits/./objects/./Vector.h: error: cannot convert 'IssmDouble* {aka adouble*}' to 'double*' in assignment vec_serial=this->pvector->ToMPISerial(); ^ make: *** Error 1 make: Leaving directory `/home/a_h_ck/issm_project/issm_p/src/c' make: *** Error 1 make: Leaving directory `/home/a_h_ck/issm_project/issm_p/src' make: *** Error 1 make: Leaving directory `/home/a_h_ck/issm_project/issm_p' make: *** Error 2

mathieumorlighem

Hi,

sorry but ISSM does not compile with ADOL-C and PETSc at the same time. Can you try and delete the --with-petsc-dir line of your configure script? You should keep MUMPS and the other PETSc packages: we want to use PETSc solvers but we cannot use PETSc for vectors and matrices as they do not support adouble.

I will update the website to make that clear,

Let us know if you have any other problem.

Best

Mathieu

ahck

Hi,

thanks for the swift reply. I have done as you advised, but another error appears. Any advice on this one? (Let me know if you need more infos)


./cores/controlad_core.cpp: In function 'FemModel* presimulad(int*, double**, FemModel*)':
./cores/controlad_core.cpp: error: 'class FemModel' has no member named 'comm'
  femmodel=new FemModel(rootpath, inputfilename, outputfilename, toolkitsfilename, lockfilename, restartfilename, femmodel->comm, femmodel->solution_type,NULL);
                                                                                                                            ^
./cores/controlad_core.cpp: In function 'void simulad(long int*, long int*, double*, double*, double*, long int*, float*, void*)':
./cores/controlad_core.cpp: error: 'class FemModel' has no member named 'comm'
  femmodelad=new FemModel(rootpath, inputfilename, outputfilename, toolkitsfilename, lockfilename, restartfilename,femmodel->comm, femmodel->solution_type,X);
make: ***  Error 1
make: *** Waiting for unfinished jobs....
make: Leaving directory `/home/a_h_ck/issm_project/issm_p/src/c'
make: ***  Error 1
make: Leaving directory `/home/a_h_ck/issm_project/issm_p/src'
make: ***  Error 1
make: Leaving directory `/home/a_h_ck/issm_project/issm_p'
make: ***  Error 2

I checked c/classes/FemModel.h, I see no such (conditionally included) member named "comm".

The config:

./configure \ --prefix=$ISSM_DIR \ --with-matlab-dir="/shared/apps/matlab/R2015b/" \ --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \ --with-mpi-include="$ISSM_DIR/externalpackages/mpich/install/include" \ --with-mpi-libflags="-L$ISSM_DIR/extenalpackages/mpich/install/lib/ -lmpich" \ --with-metis-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-blas-lapack-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \ --with-numthreads=16 \ --with-fortran-lib="/shared/apps/gcc/4.8.5/lib/../lib64/libgfortran.la" \ --with-adolc-dir="$ISSM_DIR/externalpackages/adolc/install" \ --with-ampi-dir="$ISSM_DIR/externalpackages/adjoinablempi/install" \ --without-kriging \ --without-Gia \ --without-kml \ --with-boost-dir="$ISSM_DIR/externalpackages/boost/install" \

mathieumorlighem

You are absolutely right, this is weird because it is compiling on our end (and it should not since comm is not a field of FemModel anymore). I will investigate and get back to you. Are you using trunk (as an anon user) or trunk-jpl (development branch) ?

Thanks

Mathieu

ahck

I am using a (git) clone of the trunk (not trunk-jpl). I am reasonable sure that I use the latest version that you guys provide. I will quickly double check and edit my post.

Edit: I have a clone of commit revision 20548 ( tehre was another commit in the meantime but that was only relating to config files w.r.t. Windows)

mathieumorlighem

OK so, the part of the code that breaks is only compiled if you have m1qn3 in your configuration script, and we do not use m1qn3 in our regression tests so it has not been tested in a long time (it has been broken for at least 1 year!). You should take the m1qn3 line out of configure.sh for now, and I will fix the compilation on the development branch ASAP. sorry about that!

Mathieu