External Package Installation on Apple M1 Chip

lawrence-bird

Hi Justin:

That command returns the following location: /usr/local/gfortran/lib/libgcc_s.1.1.dylib

Thanks.

Lawrence

justinquinn

lawrence-bird Okay, you may have to,

export LD_LIBRARY_PATH=/usr/local/gfortran/lib:${LD_LIBRARY_PATH}
export DYLD_LIBRARY_PATH=/usr/local/gfortran/lib:${DYLD_LIBRARY_PATH}

in your shell profile (the above assumes bash). You may be able to test this ahead of time by exporting (or otherwise setting these environment variables) directly in the terminal instance and recompiling.

lawrence-bird

justinquinn

Hi Justin - I implemented this, but the same error persists, unfortunately. I assume this would not affect the external package installations and I don't need to recompile those?

lawrence-bird

justinquinn

I am not sure if it's relevant where this error is occurring during the make process, but this is the full error:

ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make[3]: *** [libISSMCore.la] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2

This is preceded by a number of warnings similar to this:

*** Warning: Linking the shared library libISSMCore.la against the *** static library /Users/lbir0005/ISSM/trunk/externalpackages/m1qn3/install/libddot.a is not portable! ld: warning: object file (./modules/FourierLoveCorex/.libs/lnb_param.o) was built for newer macOS version (12.1) than being linked (12.0)

justinquinn

lawrence-bird
I wouldn't think that this would affect any of the external packages, no.

justinquinn

lawrence-bird
Could you post the 10 lines or so before ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64?

lawrence-bird

justinquinn

Here's the full output once things start going awry:

*** Warning: Linking the shared library libISSMCore.la against the *** static library /Users/lbir0005/ISSM/trunk/externalpackages/m1qn3/install/libm1qn3.a is not portable! *** Warning: Linking the shared library libISSMCore.la against the *** static library /Users/lbir0005/ISSM/trunk/externalpackages/m1qn3/install/libddot.a is not portable! ld: warning: object file (./modules/FourierLoveCorex/.libs/lnb_param.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: object file (./modules/FourierLoveCorex/.libs/model.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: object file (./modules/FourierLoveCorex/.libs/util.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: could not create compact unwind for ___model_MOD_earth_nlayers: register 28 saved somewhere other than in frame ld: warning: could not create compact unwind for ___model_MOD_earth_nlayers_viscoelas: register 75 saved somewhere other than in frame ld: warning: could not create compact unwind for ___model_MOD_model_init2: register 73 saved somewhere other than in frame ld: warning: object file (./modules/FourierLoveCorex/.libs/lovenb_sub.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: could not create compact unwind for ___lovenb_sub_MOD_solution_nbloutput: registers 78 and 79 not saved contiguously in frame ld: warning: could not create compact unwind for ___util_MOD_ludcmp: register 24 saved somewhere other than in frame ld: warning: ld: warning: could not create compact unwind for ___util_MOD_derivs: registers 78 and 79 not saved contiguously in frameld: warning: could not create compact unwind for ___lovenb_sub_MOD_solution: registers 78 and 79 not saved contiguously in frameld: warning: object file (./modules/GiaDeflectionCorex/.libs/libISSMCore_la-ojrule.o) was built for newer macOS version (12.1) than being linked (12.0)ld: warning: object file (./modules/GiaDeflectionCorex/.libs/libISSMCore_la-freed.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: ld: warning: object file (./modules/GiaDeflectionCorex/.libs/libISSMCore_la-distme.o) was built for newer macOS version (12.1) than being linked (12.0) object file (./modules/FourierLoveCorex/.libs/love_numbers.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: object file (./modules/GiaDeflectionCorex/.libs/libISSMCore_la-pwise.o) was built for newer macOS version (12.1) than being linked (12.0) object file (./modules/GiaDeflectionCorex/.libs/libISSMCore_la-qwise.o) was built for newer macOS version (12.1) than being linked (12.0)ld: warning: ld: warning: object file (./modules/GiaDeflectionCorex/.libs/libISSMCore_la-what0.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: could not create compact unwind for _what0_: registers 78 and 79 not saved contiguously in frame ld: warning: could not create compact unwind for ___lovenb_sub_MOD_boundary_conditions_matrix: register 77 saved somewhere other than in frame ld: warning: object file (./modules/GiaDeflectionCorex/.libs/libISSMCore_la-stot.o) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: could not create compact unwind for _stot_: registers 72 and 73 not saved contiguously in frame could not create compact unwind for _love_numbers_: registers 27 and 28 not saved contiguously in frame ld: warning: could not create compact unwind for _lnb_setup_: registers 27 and 28 not saved contiguously in frame ld: warning: could not create compact unwind for _distme_: register 75 saved somewhere other than in frame ld: warning: dylib (/Users/lbir0005/ISSM/trunk/externalpackages/petsc/install/lib/libmpifort.dylib) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: dylib (/usr/local/gfortran/lib/gcc/aarch64-apple-darwin21/12.0.0/../../..//libgfortran.5.dylib) was built for newer macOS version (12.1) than being linked (12.0) ld: warning: dylib (/usr/local/gfortran/lib/gcc/aarch64-apple-darwin21/12.0.0/../../..//libquadmath.0.dylib) was built for newer macOS version (12.1) than being linked (12.0) ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make[3]: *** [libISSMCore.la] Error 1 make[2]: *** [all-recursive] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2

justinquinn

lawrence-bird
Hi Lawrence,
The warnings may look alarming but do not indicate an actual failure. I think what might be going on is that the binaries of the this copy of gfortran are functional but the /usr/local/gfortran/lib/libgcc_s.1.1.dylib may not be compiled with ARM as the target. To be perfectly honest, I don't know what is trying to linke against libgcc_s. What do you get from ldd /usr/local/bin/gfortran? Also, what does your configuration look like?

lawrence-bird

justinquinn

Hi Justin:

ldd doesn't appear to work (I'm not sure if it's all Mac's or just these new systems), but apparently otool -L is the equivalent. Output from this:

$ otool -L /usr/local/bin/gfortran /usr/local/bin/gfortran: /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

Perhaps I will try rolling back my gfortran version to the one listed by Mathieu and see if that works on Monterey (as well as it's intended MacOS 11).

justinquinn

lawrence-bird
Right, ldd aliases to otool-classic in my case.

Okay, the output looks the same as mine. You might try running down the dependency trail by running otool -L on each of the libraries that are listed to see if they, in turn, load libgcc_s.1.1.dylib. That said, I suspect it is something else that is linking to this library. It may, after all, be one of the external package libraries, so you may have to check those too with otool.

Try rolling back your gfortran version though first and let's see if that resolves it.

lawrence-bird

justinquinn

Hi Justin:

Rolling back the gfortran version didn't resolve the issue. I've also dug around the external packages using otool -L and can't find any of those that link to libgcc_s.1.1.dylib.

The only thing that I notice in the config.log file is a "fatal error" that occurs a few times, although it dosesn't result in the configure process failing:

configure:3883: checking how to run the C preprocessor configure:3914: gcc -E conftest.c configure:3914: $? = 0 configure:3928: gcc -E conftest.c conftest.c:9:10: fatal error: 'ac_nonexistent.h' file not found #include <ac_nonexistent.h> ^~~~~~~~~~~~~~~~~~ 1 error generated. configure:3928: $? = 1 configure: failed program was: | /* confdefs.h */ | #define PACKAGE_NAME "Ice Sheet System Model (ISSM)" | #define PACKAGE_TARNAME "issm" | #define PACKAGE_VERSION "4.19" | #define PACKAGE_STRING "Ice Sheet System Model (ISSM) 4.19" | #define PACKAGE_BUGREPORT "issm@jpl.nasa.gov" | #define PACKAGE_URL "http://issm.jpl.nasa.gov" | /* end confdefs.h. */ | #include <ac_nonexistent.h>

I don't think this is related, but thought I would share it in case it sheds any light on anything.

Thanks.

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
I'm sorry, I'm running out of ideas on what might be the issue here. It's really hard for me to test on my end without access to an M1 machine. Are you getting the same error as before with the rolled back copy of gfortran?

I can confirm that the fatal errors in the config.log are not an issue:

$ grep "fatal error" config.log 
config.log:155:conftest.c:9:10: fatal error: 'ac_nonexistent.h' file not found
config.log:174:conftest.c:9:10: fatal error: 'ac_nonexistent.h' file not found
config.log:256:gfortran: fatal error: no input files
config.log:261:gfortran: fatal error: no input files
config.log:297:gfortran: fatal error: no input files
config.log:302:gfortran: fatal error: no input files
config.log:545:conftest.cpp:23:10: fatal error: 'ac_nonexistent.h' file not found
config.log:578:conftest.cpp:23:10: fatal error: 'ac_nonexistent.h' file not found

That's from a successful config and compilation.

lawrence-bird

justinquinn
Hi Justin:

No worries - this is a strange one.

I m happy to report that I was able to get ISSM to compile! But I've now got a new issue. On the compilation front, this was my process:

Grabbed a clean version from the trunk
Downloaded the gfortran 11-experimental-2 for ARM Big Sur (macOS 11) version Mathieu noted above
Installed all external packages (noting the updated to LIBTOOL_VER=2.46 for autotools
Ran autoreconf -ivf at $ISSM_DIR
Installed numpy / scipy using homebrew
Installed pyenv using homebrew (to allow me to get a dev copy of Python)
Installed python 3.9-dev using pyenv
Specified the --with-fortran-lib in the configure script to point to the location of gfortran that I'd downloaded. This step was important as another gcc version is installed using homebrew as a dependancy for numpy. Without specifying the fortran location, it would try and use the gcc location in homebrew (based, I presume, on it's location in the $PATH), and it would fail. Interestingly, the gcc version downloaded as a dependancy for numpy is that same version I was originally trying (for Monterey).
Specified the python directory (in pyenv) and the python version (3.9) in the configure script.
./configure
make
make install

This appears to result in a successful compilation!

However, when I try and run the model via Python, I am returned a Segmentation fault: 11. After launching Python, I am entering the following (before be returned the seg fault):

import sys sys.path.append('/Users/lbir0005/ISSM/trunk/bin/') sys.path.append('/Users/lbir0005/ISSM/trunk/lib/') from issmversion import issmversion

I was able to successfully build ISSM with Matlab on a different machine that I am waiting to get remote access right for.

At this point, I am just being stubborn about trying to get it working on my laptop with Python.

Cheers,

Lawrence

justinquinn

lawrence-bird
Nice! Glad to hear you got through compilation at least. The only other suggestions I could make is to make sure you incorporate the following options into your configuration (adjusting the paths as needed, of course):

--with-cxxoptflags="-D_DO_NOT_LOAD_GLOBALS_ -g -O2 -fPIC -std=c++11" \
--with-python-version=3.9 \
--with-python-dir=/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9 \
--with-python-numpy-dir=/usr/local/lib/python3.9/site-packages/numpy

I've included the -D_DO_NOT_LOAD_GLOBALS_ flag as a test because you did not receive an error related to it, but we have seen issues with more recent GCC versions that this has been a short term fix for.

Please do let us know if you get everything compiled and running with the Python API as you're in uncharted territory and it will be of use to others. Thanks!

lawrence-bird

justinquinn

Thanks Justin - I'll try adding the -D_DO_NOT_LOAD_GLOBALS_, just in case.

Two (silly) questions around working with Python and ISSM:

When I tried to compile using Python Version 3.9.10 (Installed along side numpy with homebrew), I received the error: Python.h not found error. Reading the thread here, I installed Python 3.9-dev using pyenv and things were resolved. Is this the correct way of dealing with this, or is there potentially something else resulting in this error?
Do I need to work with a Python IDE (e.g., IPython), or should I be able to get things rolling by working directly in the terminal with Python?

Thanks,

Lawrence

justinquinn

lawrence-bird
(1) Yes, as far as I know, newer versions of macOS do not include the Python header file, so you have to install your own copy. I personally use Homebrew, but I expect that other installations work just fine. (If you're interested in how we find Python.h given --with-python-version and --with-python-dir during configuration, check out m4/issm_options.m4).
(2) You do not need an IDE or GUI unless you wish to get into plotting. All of the computational features will be available to you. You might try pinging @bdefleurian for advice on generating plots under Python.

lawrence-bird

justinquinn
Hi Justin:

Good news! After a few days away, I simply made a clean build of ISSM and I'm now up and running with Python3.9 on my M1 machine!

Using PyCharm, I'm able to walk through the Square Ice Shelf model and generate the resultant basic plot. I've also been able to successfully walk through the majority of the mesh example.

One thing I wanted to confirm - I understand that using ISSM within Python is not fully supported. Does this mean that there are core components of the model that simply don't work in Python, or just that there are fewer users/folks maintaining this? This comes up in the mesh tutorial, when calling exptool() - I can't seem to locate this function within the Python build. Just wondering if this is a result of some of these tools not being available in Python, or something more sinister.

Ultimately, I need to make a decision on which route to go with ISSM (Python - via my laptop, or MATLAB - via remote access to a machine with an Intel processor). I would be interested to explore using the Python version, perhaps with some support from R for pre-/post-processing (simply because I am much more familiar with this framework), but don't want to short change myself too much on built-in available features.

Thanks.

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
That is great news! Thanks for updating us and this thread.

There are indeed some missing components of the Python API, and this does have to do with maintainability. Most of our core developers use the MATLAB API, so when new modules are developed, they sometimes do not get translates to Python right away. As well, modules that have plotting capabilities written for MATLAB are not going to have an easy or direct translation to multiple Python plotting libraries, so we have left it up to the user to decided which library works best for them. In the future, we may decide to go with a default plotting library, but there are no plans currently for this.

If you wanted to go about patching missing functionality/modules, I would go through the following steps:

Check if the missing function is a MATLAB builtin by searching the MATLAB documentation. In many cases there is an identical or similar function in NumPy.

Check if the function call is a module, and if so, if the Python version is missing. For example, from $ISSM_DIR, I might run,

$ find src -name exptool.*
src/m/exp/exptool.m

which shows that the exptool module indeed has not been translated yet. If you were to make such translations I think we would be happy to have you on as contributing developer and get you access to the development branch.

I understand not wanting to commit to the "lesser" of the available APIs, but I'm not sure there is really a huge gap in functionality besides the plotting capabilities. Indeed, most of what ISSM does is actually handled by the C++ core while the APIs are mostly for file I/O, model setup, result retrieval, and plotting. I'd be happy to help come up with a rough number of missing Python modules if you think that would help you make your decision.

Best,
Justin

lawrence-bird

justinquinn
Hi Justin:

Thanks for the summary here - this is what I suspected. I guess my biggest concern is running into snags down the road and having a limited community support for the Python API. That being said, I think it would be really helpful to have an idea how many/which modules may be missing from the Python API. If I could identify if these are generally pretty niche and I am unlikely to require them, that would probably sway me.

Any help you can provide on this would be great!

Thanks.

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
The ISSM Doxygen page indicates that there about 1000 MATLAB files and about 300 Python files in the repo. I don't know that this strictly means that about 2/3 of the capabilities are unimplemented under Python, though it might indeed be something like 50% coverage. I would run through a couple more of the tutorials and see if anything is obviously missing. Again, we may have funding for fleshing out the Python API fully in the near future, but there are no hard plans for this.

Best,
Justin

lawrence-bird

justinquinn
Thanks Justin. I persevered with the Python setup for a couple additional tutorials. I'm running into some issues with the Inversion tutorial, but I am not convinced its due to lack of python modules. I think it stems back to the build on my M1 machine.

When running md=solve(md, 'Stressbalance') with md.inversion.iscontrol=1, the process seems to stall without actually returning any solutions. Below is the full output I get after calling this solve() command:

============================================================ Binary file Square.outbin not found This typically happens when the run crashed. Please check for error messages above or in the outlog ============================================================ WARNING: Square.outbin does not exist [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.15.2, Jul 10, 2021 [0]PETSC ERROR: /Users/lbir0005/ISSM/trunk/bin/issm.exe on a named MU00186957X by lbir0005 Fri Feb 11 15:49:29 2022 [0]PETSC ERROR: Configure options --prefix=/Users/lbir0005/ISSM/trunk/externalpackages/petsc/install --PETSC_DIR=/Users/lbir0005/ISSM/trunk/externalpackages/petsc/src --CFLAGS=-Wno-error=implicit-function-declaration --FFLAGS=-fallow-argument-mismatch --LDFLAGS=-Wl,-no_compact_unwind --with-debugging=0 --with-valgrind=0 --with-x=0 --with-ssl=0 --with-pic=1 --download-fblaslapack=1 --download-mpich=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-zlib=1 [0]PETSC ERROR: #1 User provided function() at unknown file:0 [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Traceback (most recent call last): File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 90, in runcode exec(code, self.locals) File "<input>", line 12, in <module> File "/Users/lbir0005/ISSM/trunk/bin/solve.py", line 164, in solve md = loadresultsfromcluster(md) File "/Users/lbir0005/ISSM/trunk/bin/loadresultsfromcluster.py", line 68, in loadresultsfromcluster remove(md.private.runtimename + '.tar.gz') AttributeError: 'NoneType' object has no attribute 'private'

It looks like this has something to do with Petsc / MPI - I've included my ./configure options below in case there is anything here that you suggest changing.

./configure \ --prefix="$ISSM_DIR" \ --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \ --with-cxxoptflags="-D_DO_NOT_LOAD_GLOBALS_ -g -O2 -fPIC -std=c++11" \ --with-python-dir="/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9" \ --with-python-version=3.9 \ --with-python-numpy-dir="/opt/homebrew/Cellar/numpy/1.22.1/lib/python3.9/site-packages/numpy/core/include/numpy" \ --with-fortran-lib="-L/usr/local/gfortran/lib -lgfortran" \ --with-mpi-include="$ISSM_DIR/externalpackages/petsc/install/include" \ --with-mpi-libflags="-L$ISSM_DIR/externalpackages/petsc/install/lib -lmpi -lmpicxx -lmpifort" \ --with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install/" \ --with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install/" \ --with-blas-lapack-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-metis-dir="$ISSM_DIR/externalpackages/petsc/install/" \ --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \ --with-numthreads=2

I suspect it's some Petsc intricacies, but I'm all ears is you have any ideas!

Cheers,

Lawrence

« Previous Page Next Page »