Error while loading share libraries: libpetsc

adamjhepburn

Hi,

I have been trying to install ISSM on the UofSheffield Stanage cluster, using intel compilers and the system intel MPI. I installed m1qn3 and PETSc 3.21 (modifying the 3.21-andes install script to do so). Everything seemed to install fine and there is no obvious error messages (some depreceated Intel compiler classic warnings but nothing else). But on running a job, the errlog reads:

/users/gg4ahe/issm/bin/issm.exe: error while loading shared libraries: libpetsc.so.3.21: cannot open shared object file: No such file or directory

libpetsc.so.3.21 is present in:

$ISSM_DIR/externalpackages/petsc/install/lib .

My PETSc install looks like this:

./config/configure.py \ COPTFLAGS="-g -O3" CXXOPTFLAGS="-g -O3" FOPTFLAGS="-g -O3" \ --prefix="${PREFIX}" \ --PETSC_DIR="${PETSC_DIR}" \ --with-make-np=20 \ --with-debugging=0 \ --with-valgrind=0 \ --with-x=0 \ --with-ssl=0 \ --with-pic=1 \ --with-blas-lapack-dir=$MKLROOT \ --download-metis=1 \ --with-mpi-dir=$EBROOTIMPI/mpi/latest/ \ --download-parmetis=1 \ --download-scalapack=1 \ --download-mumps=1 make make install

My configure.sh looks like this:

export CC=mpiicc export CXX=mpiicpc export FC=mpiifort ./configure \ --prefix=$ISSM_DIR \ --with-wrappers=no \ --without-Love \ --without-kml \ --with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \ --with-mpi-include="-I$EBROOTIMPI/mpi/latest/include" \ --with-mpi-libflags="-L$EBROOTIMPI/mpi/latest/lib -lmpicxx -lmpifort -lmpi"\ --with-metis-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-mkl-libflags="-L$MKLROOT/lib/intel64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm" \ --with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-cxxoptflags="-g -O3 -std=c++11 -fp-model=precise -diag-disable=10441" \ --with-fortran-lib="-L$EBROOTIMPI/mpi/latest/lib -lifcore -lifport -lgfortran" \ --enable-development

and my bash_profile includes

module purge module load CMake/3.12.1 Autotools/20220317 intel/2022b export ISSM_DIR=/users/gg4ahe/issm source $ISSM_DIR/etc/environment.sh

Thank you,
Adam

mathieumorlighem

Hi Adam

since you are running ISSM on a cluster, you need to create a new specific cluster class that loads the same modules as the ones you used to install ISSM. Take a look at src/m/classes/clusters/discovery.m for example and let us know if you need help
Cheers
Mathieu

adamjhepburn

Hi Mathieu,

I had created a cluster script based on the computecanada.m function, and it seems to work fine in terms of squeue etc. I have also since tried recompiling with the following added to my configure

`
export FFGLAGS="-Wno-argument-mismatch"

export CXXFLAGS="${CXXFLAGS}-Wno-error=redundant-decls"

export LDFLAGS="-Wl,--copy-dt-needed-entries"
`

But the same error persists. /users/gg4ahe/issm/externalpackages/petsc/install/lib is also on LD_LIBRARY_PATH

mathieumorlighem

if libpetsc.so cannot be found at run time, that's because you need to add:

fprintf(fid,'source $ISSM_DIR/etc/environment.sh\n');

to make sure PETSc is added to your path.

adamjhepburn

mathieumorlighem

After adding that the queue script now sources the environment.sh script but it still happens... I have tried changing them to explicit paths in the queue script too just to check everything was correct there,

thanks,
Adam

mathieumorlighem

you should add

ldd <PATH to ism.exe>/issm.exe

to the queueing script and see what comes out

adamjhepburn

mathieumorlighem here is the output

` linux-vdso.so.1 => (0x00007ffe882f5000)
libISSMCore.so => /users/gg4ahe/issm/lib/libISSMCore.so (0x00007f96ec105000)
libISSMOverload.so => /users/gg4ahe/issm/lib/libISSMOverload.so (0x00007f96ece93000)
libstdc++.so.6 => /opt/apps/testapps/el7/software/staging/GCCcore/12.2.0/lib64/libstdc++.so.6 (0x00007f96ebee2000)
libpetsc.so.3.21 => /users/gg4ahe/issm/externalpackages/petsc/install/lib/libpetsc.so.3.21 (0x00007f96e9878000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f96e9674000)
libparmetis.so => /users/gg4ahe/issm/externalpackages/petsc/install/lib/libparmetis.so (0x00007f96ece20000)
libscalapack.so.2.2 => /users/gg4ahe/issm/externalpackages/petsc/install/lib/libscalapack.so.2.2 (0x00007f96e8f9d000)
libmetis.so => /users/gg4ahe/issm/externalpackages/petsc/install/lib/libmetis.so (0x00007f96ecd85000)
libmkl_intel_lp64.so.2 => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/mkl/2022.2.1/lib/intel64/libmkl_intel_lp64.so.2 (0x00007f96e7d82000)
libmkl_sequential.so.2 => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/mkl/2022.2.1/lib/intel64/libmkl_sequential.so.2 (0x00007f96e635a000)
libmkl_core.so.2 => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/mkl/2022.2.1/lib/intel64/libmkl_core.so.2 (0x00007f96e1f85000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f96e1d69000)
libmpicxx.so.12 => /opt/apps/testapps/el7/software/staging/impi/2021.7.1-intel-compilers-2022.2.1/mpi/2021.7.1/lib/libmpicxx.so.12 (0x00007f96e1b49000)
libmpifort.so.12 => /opt/apps/testapps/el7/software/staging/impi/2021.7.1-intel-compilers-2022.2.1/mpi/2021.7.1/lib/libmpifort.so.12 (0x00007f96e1795000)
libmpi.so.12 => /opt/apps/testapps/el7/software/staging/impi/2021.7.1-intel-compilers-2022.2.1/mpi/2021.7.1/lib/release/libmpi.so.12 (0x00007f96dff60000)
libifcore.so.5 => /opt/apps/testapps/el7/software/staging/intel-compilers/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libifcore.so.5 (0x00007f96dfe28000)
libifport.so.5 => /opt/apps/testapps/el7/software/staging/intel-compilers/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libifport.so.5 (0x00007f96ecd59000)
libgfortran.so.5 => /opt/apps/testapps/el7/software/staging/GCCcore/12.2.0/lib64/libgfortran.so.5 (0x00007f96dfb5c000)
libquadmath.so.0 => /opt/apps/testapps/el7/software/staging/GCCcore/12.2.0/lib64/libquadmath.so.0 (0x00007f96ecd11000)
libm.so.6 => /lib64/libm.so.6 (0x00007f96df85a000)
librt.so.1 => /lib64/librt.so.1 (0x00007f96df652000)
libgcc_s.so.1 => /opt/apps/testapps/el7/software/staging/GCCcore/12.2.0/lib64/libgcc_s.so.1 (0x00007f96eccef000)
libc.so.6 => /lib64/libc.so.6 (0x00007f96df284000)
libimf.so => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libimf.so (0x00007f96dee8c000)
libirng.so => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libirng.so (0x00007f96deb22000)
libcilkrts.so.5 => /opt/apps/testapps/el7/software/staging/intel-compilers/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libcilkrts.so.5 (0x00007f96de8e5000)
libintlc.so.5 => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00007f96de873000)
libsvml.so => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libsvml.so (0x00007f96dd262000)
/lib64/ld-linux-x86-64.so.2 (0x00007f96ecc7a000)
libirc.so => /opt/apps/testapps/el7/software/staging/imkl/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libirc.so (0x00007f96dd1f0000)
libifcoremt.so.5 => /opt/apps/testapps/el7/software/staging/intel-compilers/2022.2.1/compiler/2022.2.1/linux/compiler/lib/intel64_lin/libifcoremt.so.5 (0x00007f96dd081000)

mathieumorlighem

is this from within the job you submitted?

adamjhepburn

mathieumorlighem It is yeah, I added it on the line before the srun command

adamjhepburn

adamjhepburn yeah... I will keep digging tomorrow and let you know if I find anything! Thanks for your help

mathieumorlighem

then libpetsc.so is found....

mathieumorlighem

show us your queue just in case I see something obvious

adamjhepburn

#!/bin/bash
#SBATCH --job-name=STEADY_STATE_GlaDS_NoFluxGate-06-06-2024-16-37
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=5
#SBATCH --mem-per-cpu=1gb
#SBATCH --output=STEADY_STATE_GlaDS_NoFluxGate-06-06-2024-16-37.outlog
#SBATCH --error=STEADY_STATE_GlaDS_NoFluxGate-06-06-2024-16-37.errlog

export ISSM_DIR="/users/gg4ahe/issm/bin/../"
source /users/gg4ahe/issm/etc/environment.sh

cd /mnt/parscratch/users/gg4ahe/SLIDE/STEADY_STATE_GlaDS_NoFluxGate-06-06-2024-16-37

ldd /users/gg4ahe/issm/bin/issm.exe
srun -n 1 /users/gg4ahe/issm/bin/issm.exe TransientSolution /mnt/parscratch/users/gg4ahe/SLIDE/STEADY_STATE_GlaDS_NoFluxGate-06-06-2024-16-37 STEADY_STATE_GlaDS_NoFluxGate-06-06-2024-16-37

mathieumorlighem

that should work 🙁

adamjhepburn

mathieumorlighem hmm okay... I will try a new install tomorrow with a different set of compilers and let you know how I get on!

adamjhepburn

mathieumorlighem

There must have been some sort of incompatibility with open-mpi along the line somewhere. Rerunning everything with intel compilers, letting PETSc install mpich, and forcing m1qn3 to use mpif90 seems to have done the trick. Thanks for your help!

Adam