ISSM-Matlab NetCDF4 reader segmentation fault (with workaround)

rlaw

Hi,

I'm using ISSM development build with Matlab on Linux.

There is an ok workaround for this problem (detailed at the end), so it's not urgent, but I've put it on the forum in case anyone else also experienced this issue, and in case there is a solution that doesn't require too much digging.

I ran into an error when loading NetCDF Version 4 files, which appears to be a result of this version also using the HDF5 data format. NetCDF Version 3 (like that included in the Greenland tutorial) which doesn't use HDF5 works fine, Matlab is also fine loading NetCDF version 4 when I don't preload libhdf5.so.103 ('LD_PRELOAD="$ISSM_DIR/externalpackages/petsc/install/lib/libhdf5.so.103" matlab) however in this case the ISSM linking doesn't work (this was the solution to a previous problem).

The error before Matlab is killed is:

Warning! ***HDF5 library version mismatched error*** The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is linked. Data corruption or segmentation faults may occur if the application continues. This can happen when an application was compiled by one version of HDF5 but linked with a different version of static or shared HDF5 library. You should recompile the application or check your shared library related settings such as 'LD_LIBRARY_PATH'. You can, at your own risk, disable this warning by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 or higher will suppress the warning messages totally. Headers are 1.10.8, library is 1.10.5
So it seems like a mismatch between the installed HDF5 version (1.10.5) and that requested by Matlab (1.10.8). I've copied in the rest of the HDF5 configuration data at the end of this post. There are some people experiencing this issue, but no good solutions seems to exist in online forums (yet).

Things I tried -

Changing the LD_PRELOAD command to LD_PRELOAD="$ISSM_DIR/externalpackages/hdf5/install/lib/libhdf5.so.103. This doesn't disrupt the ISSM-Matlab link, but also doesn't fix the problem

Changing the LD_PRELOAD command to direct it to a precompiled HDF5 binary of the correct version (obtainable here) which causes everything to break.

Workaround -

Just convert the NetCDF to Version 3 using ncks -6 infile.nc outfile.nc from NCO package.

Could be a mountain of a molehill, but perhaps HDF5 functionality will be vital in the future, so putting it up now. Also just curious what the problem is!!

Thanks for the support 🙂

Rob

``SUMMARY OF THE HDF5 CONFIGURATION

General Information:

HDF5 Version: 1.10.5
Configured on: Wed Dec 21 15:15:40 CET 2022
Configured by: rolaw8691@geo0223892
Host system: x86_64-unknown-linux-gnu
Uname information: Linux geo0223892 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Byte sex: little-endian
Installation point: /Home/siv34/rolaw8691/ISSM/ISSM/trunk/externalpackages/petsc/install

Compiling Options:

Build Mode: production
Debugging Symbols: no
Asserts: no
Profiling: no
Optimization Level: high

Linking Options:

Libraries: static, shared
Statically Linked Executables:
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS:
Extra libraries: -lz -ldl -lm -Wl,-rpath,/Home/siv34/rolaw8691/ISSM/ISSM/trunk/externalpackages/petsc/install/lib -L/Home/siv34/rolaw8691/ISSM/ISSM/trunk/externalpackages/petsc/install/lib -lz -lm
Archiver: /usr/bin/ar
AR_FLAGS: cr
Ranlib: ranlib

Languages:

C: yes
C Compiler: /Home/siv34/rolaw8691/ISSM/ISSM/trunk/externalpackages/petsc/install/bin/mpicc ( MPICH version 3.3.1 Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1₂₀.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-Av3uEd/gcc-9-9.4.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu built with gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1₂₀.04.1))
CPPFLAGS: -I/Home/siv34/rolaw8691/ISSM/ISSM/trunk/externalpackages/petsc/install/include
H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API
AM_CPPFLAGS:
C Flags: -fstack-protector -g -O
H5 C Flags: -std=c99 -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O
AM C Flags:
Shared C Library: yes
Static C Library: yes

                    Fortran: yes
           Fortran Compiler: /Home/siv34/rolaw8691/ISSM/ISSM/trunk/externalpackages/petsc/install/bin/mpif90 ( MPICH version 3.3.1 Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-Av3uEd/gcc-9-9.4.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu built with gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1))
              Fortran Flags: -ffree-line-length-0 -g -O
           H5 Fortran Flags:  -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type  -s -O2
           AM Fortran Flags:
     Shared Fortran Library: yes
     Static Fortran Library: yes

                        C++: no

                       Java: no

Features:

Parallel HDF5: yes
Parallel Filtered Dataset Writes: yes
Large Parallel I/O: yes
High-level library: yes
Threadsafety: no
Default API mapping: v18
With deprecated public symbols: yes
I/O filters (external): deflate(zlib)
MPE:
Direct VFD: no
dmalloc: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: no
Memory allocation sanity checks: no
Function stack tracing: no
Strict file format checks: no
Optimization instrumentation: no
Bye...
QXcbGlxIntegration::initialize

Xcb GLX gl-integration successfully initialized

Killed``

mathieumorlighem

Hi Robert

I think the best is that you remove HDF5 from PETSc and from external packages so that it does not conflict with MATLAB. In other words, I would remove $ISSM_DIR/externalpackages/hdf5/install and I would edit PETSc's installation script and remove --download-hdf5=1 , reinstall PETSc and then recompile ISSM.
Mathieu

rlaw

Hi Mathieu,

Ok I'll try this in the next week or so! Does this mean the HDF5 library is only used in conjunction with the Matlab wrapper, and not as a part of the C++ routines?

Best,
Rob

mathieumorlighem

I can't quite remember whether it is for python or for sea level applications but the vast majority of users don't need HDF5 in ISSM.

rlaw

ok!

rlaw

Noting here that this was also a problem for a fresh development installation on my linux laptop. In this case on running issmversion after what seemed like a successful (and mercifully swift) installation I got the error:

Error using issmversion
ISSM not correctly installed. "IssmConfig_matlab" not found

Removing HDF5 from the PETSc install shell script and also from the ISSM configure.sh as outlined above seems to have fixed the problem.

Thanks,
Rob

justinquinn

rlaw
As you have noted, there is a conflict between the copy of libhdf5 that ships with MATLAB and the way that the copy installed by PETSc is configured (namely, that it does not have parallel support). The way that we have gotten around this is to:

remove --download-hdf5 from our PETSc configuration
install a standalone copy of HDF5 with the scripts available at externalpackages/hdf5, setting VER to the same version of libhdf5 that ships with MATLAB
using LD_PRELOAD to make sure our copy of libhdf5 is loaded by MATLAB

For MATLAB R2022b, this version is 1.8.23 and we use the install-1-parallel.sh script to configure and install it. For future versions of MATLAB, let us know if you discover that a newer version of HDF5 is required and we will update our external package repository and install scripts accordingly. Thanks!