IssmConfig_python.so Error

lawrence-bird

Hi folks:

I have installed ISSM with Python wrappers locally on my MacOS. I have re-installed following a MacOS update. I installed with --enable-development on. I am running the devpath.py file like this:

import os
os.getenv('ISSM_DIR')
with open(os.getenv('ISSM_DIR')+'/src/m/dev/devpath.py') as f:
    exec(f.read())

This allows me to successfully load/access some python modules. However, some modules that (as far as I can tell) rely on IssmConfig are throwing errors. For example, when I run from issmversion import issmversion, I get the following error:

Traceback (most recent call last):
  File "/Users/lawrence.bird/Library/Python/3.9/lib/python/site-packages/IPython/core/interactiveshell.py", line 3550, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-7ef62d322e4f>", line 1, in <module>
    from issmversion import issmversion
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/lawrence.bird/ISSM/src/m/dev/issmversion.py", line 1, in <module>
    from IssmConfig import IssmConfig
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/Users/lawrence.bird/ISSM/src/m/modules/IssmConfig.py", line 1, in <module>
    from IssmConfig_python import IssmConfig_python
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: dlopen(/Users/lawrence.bird/ISSM/lib/IssmConfig_python.so, 0x0002): Library not loaded: /Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib
  Referenced from: <893DECCB-49EC-380A-9385-426A669EA67B> /Users/lawrence.bird/ISSM/lib/IssmConfig_python.so
  Reason: tried: '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/System/Volumes/Preboot/Cryptexes/OS/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/System/Volumes/Preboot/Cryptexes/OS/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib')

I can load others, for example from processmesh import processmesh without issue. Perhaps another @rpath issue, but I'm not sure where best to start.

Any tips welcome -- Thanks!

Lawrence

mathieumorlighem

Hi Lawrence
this is weird, it seems to be related to PETSc's Scalapack library. Could you locate it? It should be in $ISSM_DIR/externalpackages/petsc/install/lib/libscalapack.2.2.dylib (/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib I assume?)
Cheers
Mathieu

lawrence-bird

HI @mathieumorlighem:

Yep, it's there (pointing to libscalapack.2.2.1.dylib) :

libscalapack.2.2.dylib -> libscalapack.2.2.1.dylib

I had to add some symlinks to get ISSM to build:

    sudo ln -s /opt/homebrew/Cellar/gcc/14.2.0_1/lib/gcc/14/libgfortran.5.dylib /usr/local/lib/libgfortran.5.dylib
    sudo ln -s /opt/homebrew/Cellar/gcc/14.2.0_1/lib/gcc/14/libquadmath.0.dylib /usr/local/lib/libquadmath.0.dylib
    sudo ln -s /opt/homebrew/Cellar/gcc/14.2.0_1/lib/gcc/14/libgcc_s.1.1.dylib /usr/local/lib/libgcc_s.1.1.dylib

I wonder if I need to do something similar with? For context -- everything was working, and then my MacOS updated and it broke. I tried a completely new install (including external packages) hoping that it would fix itself, but no luck.

Cheers,

Lawrence

mathieumorlighem

can you do

otool -L /Users/lawrence.bird/ISSM/lib/IssmConfig_python.so

I would like to see the path of the libraries IssmConfig is linked against

lawrence-bird

Thanks @mathieumorlighem -- here's the output of that command:

lawrence.bird@nri-088478 lib % otool -L /Users/lawrence.bird/ISSM/lib/IssmConfig_python.so
/Users/lawrence.bird/ISSM/lib/IssmConfig_python.so:
	/Users/lawrence.bird/ISSM/lib/libISSMPython.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/lawrence.bird/ISSM/lib/libISSMModules.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/lawrence.bird/ISSM/lib/libISSMCore.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/lawrence.bird/ISSM/externalpackages/triangle/install/lib/libtriangle.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/lawrence.bird/ISSM/lib/libISSMApi_python.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libpetsc.3.22.dylib (compatibility version 3.22.0, current version 3.22.3)
	/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib (compatibility version 2.2.0, current version 2.2.1)
	/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libmetis.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libmpicxx.12.dylib (compatibility version 18.0.0, current version 18.0.0)
	/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libmpifort.12.dylib (compatibility version 18.0.0, current version 18.0.0)
	/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libmpi.12.dylib (compatibility version 18.0.0, current version 18.0.0)
	/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libpmpi.12.dylib (compatibility version 18.0.0, current version 18.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
	/opt/homebrew/opt/gcc/lib/gcc/current/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0)
	/opt/homebrew/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/Python (compatibility version 3.9.0, current version 3.9.0)
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 3423.0.0)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1900.178.0)

justinquinn

@mathieumorlighem lawrence-bird Hi Lawrence,
First thing's first. It looks like you're using PyCharm. Does ISSM load as expected when you start python directly from the command line in a Terminal instance?

In conjunction with the above, you might try to run devpath instead with,

export PYTHONPATH="${ISSM_DIR}/src/m/dev"
export PYTHONSTARTUP="${PYTHONPATH}/devpath.py"

before launching python (probably won't make a difference, but for the sake of debugging, I haven't run exec on the devpath script before, so I can't vouch for it).

Finally, we get to what is likely the problem: did you also upgrade Xcode or the Command Line Tools when you updated macOS? A lot of these rpath issues creep in during upgrades to these build tools. Do you have both installed? If you do not need Xcode explicitly, I would recommend uninstalling it and instead installing the Command Line Tools. One thing I also had to do recently after an upgrade was to reinstall Command Line Tools entirely,

rm -rf /Library/Developer/CommandLineTools
xcode-select --install

After reinstallation, you might try first to remake ISSM, if that fails reconfigure and remake ISSM, and if that fails reinstall all of the external packages then ISSM.

Let us know how this goes.

mathieumorlighem

I honestly have no idea... @justinquinn, what do you think?

lawrence-bird

justinquinn Thanks @justinquinn

I tried all of these things are have the same issue. It doesn't matter if I run in PyCharm or directly from the terminal. Using the export calls doesn't have an impact.

I removed and re-installed CLT and tried a completely fresh install. No joy.

Cheers,

Lawrence

justinquinn

lawrence-bird Well that's not ideal. Sorry for sending you barking up the wrong tree, but that is frequently at least part of the solution for me after macOS upgrades.

Okay, let's see,
1) ISSM is compiling successfully
2) External package libs (in particular, ScaLAPACK from PETSc) are not findable at runtime, but if you run otool on them, all linked libraries are found

So it has to be an environment thing. Are you running source $ISSM_DIR/etc/environment.sh before starting up Python? You would have to be running it in your example with otool because all the libraries are found. Can you try this: have two fresh terminals open, run Python in one with your usual ISSM startup, and in the second terminal, source the ISSM environment. Then, print environment variables like PATH, LD_LIBRARY_PATH, and DYLD_LIBRARY_PATH and compare the contents in the two situations. I wonder if Python is somehow clobbering env vars. I wouldn't suspect this normally after an upgrade but I've seen weirder things.

justinquinn

@lawrence-bird Also, I forgot to ask: are you on an Intel- or Silicon-based Mac, and what is the OS version? Just in case I have to try to reproduce the issue.

lawrence-bird

Hi @justinquinn

I tried this and it doesn't seem to matter. I have the following export calls in my .zprofile:

## ------------------------------ ISSM ------------------------------
export ISSM_DIR="/Users/lawrence.bird/ISSM"
source $ISSM_DIR/etc/environment.sh
export PYTHONPATH="${ISSM_DIR}/src/m/dev"
export PYTHONSTARTUP="${PYTHONPATH}/devpath.py"

I am running Sequoia 15.5 on an Apple M4 Pro chip. I have version 16.3.0.0.1.1742442376 of CLT installed.

Cheers,

Lawrence

lawrence-bird

Hi @justinquinn:

The plot thickens.

I also wanted to try ISSM with MATLAB. I am running MATLAB 2024b and got some MEX compilation error when I tried building natively. I am not sure if 2024b is fully compatible with Apple Silicon. Nevertheless, I tried the old approach to get up and running on a silicon chip (compile without wrappers and add in binaries).

ISSM built successfully, and the ISSM development path loaded correctly etc. ISSM functionality looks good with MATLAB when running the SquareIceShelf tutorial, until I try to solve(). This is what my output looks like:

>> md=solve(md,'Stressbalance');
      creating thickness
      creating drag
      initial velocity
      creating flow law parameter
      creating boundary conditions
      boundary conditions for stressbalance model: spc set as zero
      no smb.mass_balance specified: values set as zero
      no basalforcings.groundedice_melting_rate specified: values set as zero
      no basalforcings.floatingice_melting_rate specified: values set as zero
      no balancethickness.thickening_rate specified: values set as zero
      no thermal boundary conditions created: no observed temperature found
checking model consistency
marshalling file Square.bin
uploading input file and queuing script
launching solution sequence on remote cluster
dyld[11387]: Library not loaded: /Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib
  Referenced from: <EDC654BF-D709-3908-9C25-3569E6898078> /Users/lawrence.bird/ISSM/bin/issm.exe
  Reason: tried: '/Users/lawrence.bird/ISSM/externalpackages/triangle/install/lib/libscalapack.2.2.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/System/Volumes/Preboot/Cryptexes/OS/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/Users/lawrence.bird/ISSM/externalpackages/triangle/install/lib/libscalapack.2.2.1.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/System/Volumes/Preboot/Cryptexes/OS/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib')
dyld[11386]: Library not loaded: /Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib
  Referenced from: <EDC654BF-D709-3908-9C25-3569E6898078> /Users/lawrence.bird/ISSM/bin/issm.exe
  Reason: tried: '/Users/lawrence.bird/ISSM/externalpackages/triangle/install/lib/libscalapack.2.2.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/System/Volumes/Preboot/Cryptexes/OS/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/Users/lawrence.bird/ISSM/externalpackages/triangle/install/lib/libscalapack.2.2.1.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib'), '/System/Volumes/Preboot/Cryptexes/OS/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (no such file), '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib/libscalapack.2.2.1.dylib' (duplicate LC_RPATH '/Users/lawrence.bird/ISSM/externalpackages/petsc/install/lib')

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 11386 RUNNING AT nri-088478
=   EXIT CODE: 6
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap: 6 (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
loading results from cluster
cp: /Users/lawrence.bird/ISSM//execution/Square-05-23-2025-12-05-24-4061//Square.outbin: No such file or directory
Warning:
============================================================
   Binary file Square.outbin not found!

   This typically happens when the run crashed.
   Please check for error messages above or in the outlog
============================================================
 
> In loadresultsfromdisk (line 16)
In loadresultsfromcluster (line 50)
In solve (line 180)

The error is similar to that I get in python, relating to the libscalapack.2.2.dylib. The ONLY thing that I changed when installing external packages was when installing cmake, I had to add this line to comment out fdopen because it was clashing with something in the updated CLT.

# Fix fdopen compatability with MacOS SDKs
sed -i.bak '/ifndef fdopen/,/# *endif/ s/^/\/\//g' src/Utilities/cmzlib/zutil.h

Other than that, all external packages are installed using the standard install scripts.

lawrence-bird

Call off the troops @justinquinn.

This was totally on me. I was using our ACCESS-ISSM repo. This uses an older version of cmake (which required me to make the update I mentioned above), probably amongst other differences that I haven't come across yet.

I don't really know what was going on, but using the main ISSM repo, I can build with python wrappers without issue. This is a good lesson for us to think carefully about how often we update our repo!

Sorry for sending us on this unnecessary journey! As always, I appreciate your help!

justinquinn

lawrence-bird Hi Lawrence, no problem at all. Glad to hear it’s working now! Do let us know if you have any other issues.