error in square shelf symbol not found in flat namespace "_MPI_Allgather")

JohannaBeckmann

justinquinn
that is the error message:
ploading input file and queuing script launching solution sequence on remote cluster dyld[62204]: Library not loaded: @rpath/libgfortran.5.dylib Referenced from: <A6CA13DB-7C2B-3815-9C1B-E25E7C063B95> /Users/jbec0008/trunk/bin/issm.exe Reason: tried: '/Users/jbec0008/trunk/externalpackages/triangle/install/lib/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)
===================================================================================
`= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 62204 RUNNING AT MU00154828X
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES

= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
loading results from cluster`

justinquinn

JohannaBeckmann
Okay, as this is macOS, try adding,

export DYLD_LIBRARY_PATH="/usr/local/gfortran/lib:${DYLD_LIBRARY_PATH}"

to ~/.zshrc, start a new terminal or MATLAB session, then try again.

JohannaBeckmann

justinquinn
Still the same error message although the number change. are we slowly moving forward 🙂?
dyld[7138]: Library not loaded: @rpath/libgfortran.5.dylib Referenced from: <A6CA13DB-7C2B-3815-9C1B-E25E7C063B95> /Users/jbec0008/trunk/bin/issm.exe Reason: tried: '/Users/jbec0008/trunk/externalpackages/triangle/install/lib/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache) )

Why does Issa not look into /usr/local/gfortran/lib
I tried with LIBRARY_PATH too. no change.

justinquinn

JohannaBeckmann
Try recompiling ISSM now, though I'm not entirely sure why the above is not already working. You can do so with cd $ISSM_DIR; make; make install.

JohannaBeckmann

Hi,
I did that no luck. then I recompiled everything also petsc. because in your compile script for petsc is som things I did not understand so well with fPIC, which links the library as well as I understand. could this be a point to change?
However no luck either after recompiling everything.
my error message now is:
dyld[2046]: Library not loaded: @rpath/libgfortran.5.dylib Referenced from: <A6CA13DB-7C2B-3815-9C1B-E25E7C063B95> /Users/jbec0008/trunk/bin/issm.exe Reason: tried: '/Users/jbec0008/trunk/externalpackages/triangle/install/lib/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache) dyld[2047]: Library not loaded: @rpath/libgfortran.5.dylib Referenced from: <A6CA13DB-7C2B-3815-9C1B-E25E7C063B95> /Users/jbec0008/trunk/bin/issm.exe Reason: tried: '/Users/jbec0008/trunk/externalpackages/triangle/install/lib/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)

Issm still does not look into the right path. is there something wrong with how I start issm? I read only to include the dev path is that right? I also tried with including `$ISSM_DIR/bin' but also the same kind of error.

here is my issm alias and command of my .zshrc:

export ISSM_DIR=/Users/jbec0008/trunk alias libenv="source $ISSM_DIR/etc/environment.sh" source $ISSM_DIR/etc/environment.sh alias issm='/Applications/MATLAB_R2022b.app/bin/matlab -nosplash -r "addpath $ISSM_DIR/src/m/dev; devpath; addpath $ISSM_DIR/lib2 $ISSM_DIR/lib"' export LD_LIBRARY_PATH="/usr/local/gfortran/lib:${LD_LIBRARY_PATH}" export DYLD_LIBRARY_PATH="/usr/local/gfortran/lib:${DYLD_LIBRARY_PATH}" export LIBRARY_PATH="/usr/local/gfortran/lib:${LIBRARY_PATH}"

while in lib2 are the binaries for intel I downloaded.

This is way above my head but I never really deinstalled rosetta2. could that be a problem. When I type in Matlab

! file "$(which arch)"
is get
/usr/bin/arch: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e] /usr/bin/arch (for architecture x86_64): Mach-O 64-bit executable x86_64 /usr/bin/arch (for architecture arm64e): Mach-O 64-bit executable arm64e

or is that because of the binaries in lib2?

justinquinn

JohannaBeckmann

There is nothing wrong with your ISSM installation: this is an issue with the linker not looking in the correct locations for libgfortran. What's odd is that it does look in /Users/jbec0008/trunk/externalpackages/triangle/install/lib, so we are baking custom rpath search locations into our binaries correctly in some situations. I have to look further into this.

Back to your case, I'm not quite sure why defining LIBRARY_PATH and DYLD_LIBRARY_PATH is not working. I think what is happening is that you are using the system or Xcode copy of gcc and it is using a different value for @rpath than what might be used by older system or third-party (e.g. Homebrew) distributions of gcc. Try adding export LD_RUNPATH_SEARCH_PATH="/usr/local/gfortran/lib:$LD_RUNPATH_SEARCH_PATH" to ~/.zshrc and try running MATLAB again.

I read only to include the dev path is that right? I also tried with including `$ISSM_DIR/bin' but also the same kind of error.

Only if you used --enable-development in your configuration. Otherwise, it should be "addpath $ISSM_DIR/bin $ISSM_DIR/lib $ISSM_DIR/lib2".

JohannaBeckmann

justinquinn
ok nothing:
launching solution sequence on remote cluster dyld[5047]: Library not loaded: @rpath/libgfortran.5.dylib Referenced from: <A6CA13DB-7C2B-3815-9C1B-E25E7C063B95> /Users/jbec0008/trunk/bin/issm.exe Reason: tried: '/Users/jbec0008/trunk/externalpackages/triangle/install/lib/libgfortran.5.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/libgfortran.5.dylib' (no such file), '/usr/local/lib/libgfortran.5.dylib' (no such file), '/usr/lib/libgfortran.5.dylib' (no such file, not in dyld cache)

I could still try to reinstall everything with brew gcc, in do the symbolic links again. because which gcc would always give me the Xcode one. But I would wait until I have your go on that and what has to be linked / used by the installation. e.g. c++...

now I know where to look for the library path correctly

justinquinn

JohannaBeckmann
If you want to go the Homebrew route, try installing with brew install gcc and run which gcc again. You still get the Xcode copy?

JohannaBeckmann

yep !
/usr/bin/gcc.
with gcc --version
gcc --version Apple clang version 14.0.0 (clang-1400.0.29.202) Target: arm64-apple-darwin22.3.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
I think because brew installed everything under gcc12.

which gcc-12 /opt/homebrew/bin/gcc-12

so last time I went into the brew folder in linked ln -s gcc12 gcc. ans so on. what worked.
did that for c++ as well.
then which gcc would give me the correct version. However, I am afraid of forgetting something and then have a mix of Xcode and brew versions. but if you can tell what to be aware of it could work?

justinquinn

JohannaBeckmann
Right, I forgot that Homebrew doesn't manage links for gcc the same way that it used to (to avoid the exact situation you are point to). On my machine, I do get,

$ which gcc
/usr/bin/gcc

$ gcc --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: x86_64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

and even though I have installed gfortran via Homebrew, I have had no issues compiling and running ISSM (save for some of the path variable modifications we have tried. Let's do this instead: we know where the linker is looking for libgfortran when issm.exe is run because the error message tells us. So let's just symlink from one of those locations to libgfortran with, for example,

sudo ln -s /usr/local/gfortran/lib/libgfortran.5.dylib /usr/local/lib/libgfortran.5.dylib

this should get things working, and then you can diagnose the underlying issue later if you want, and I will address the @rpath issues in our build system when I can.

JohannaBeckmann

justinquinn
Ok so the same error came up with libquadmath so also made a symbolic link to usr/local/lib

now I get a really interesting error 🙂:

Fatal error in internal_Init: Other MPI error, error stack: internal_Init(59)..................: MPI_Init(argc=0x16bc5e89c, argv=0x16bc5e890) failed MPII_Init_thread(209)..............: MPID_Init(77)......................: init_world(192)....................: channel initialization failed MPIDI_CH3_Init(84).................: MPID_nem_init(313).................: MPID_nem_tcp_init(175).............: MPID_nem_tcp_get_business_card(397): GetSockInterfaceAddr(370)..........: gethostbyname failed, MU00154828X (errno 0) Fatal error in internal_Init: Other MPI error, error stack: internal_Init(59)..................: MPI_Init(argc=0x16f41289c, argv=0x16f412890) failed MPII_Init_thread(209)..............: MPID_Init(77)......................: init_world(192)....................: channel initialization failed MPIDI_CH3_Init(84).................: MPID_nem_init(313).................: MPID_nem_tcp_init(175).............: MPID_nem_tcp_get_business_card(397): GetSockInterfaceAddr(370)..........: gethostbyname failed, MU00154828X (errno 0) loading results from cluster Warning:

That does not look like progress, right ?🙂

justinquinn

JohannaBeckmann
Progress! You’re nearly there. The answer to that one is at the bottom of https://issm.jpl.nasa.gov/documentation/faq/matlab/

JohannaBeckmann

YAY! It worked!!!! Thanks. Have a nice weekend.

justinquinn

JohannaBeckmann
Awesome! Glad to hear it. Have a nice weekend too, and let me know if you have any more issues.

« Previous Page