External Package Installation on Apple M1 Chip

justinquinn

lawrence-bird
Nice! Glad to hear you got through compilation at least. The only other suggestions I could make is to make sure you incorporate the following options into your configuration (adjusting the paths as needed, of course):

--with-cxxoptflags="-D_DO_NOT_LOAD_GLOBALS_ -g -O2 -fPIC -std=c++11" \
--with-python-version=3.9 \
--with-python-dir=/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9 \
--with-python-numpy-dir=/usr/local/lib/python3.9/site-packages/numpy

I've included the -D_DO_NOT_LOAD_GLOBALS_ flag as a test because you did not receive an error related to it, but we have seen issues with more recent GCC versions that this has been a short term fix for.

Please do let us know if you get everything compiled and running with the Python API as you're in uncharted territory and it will be of use to others. Thanks!

lawrence-bird

justinquinn

Thanks Justin - I'll try adding the -D_DO_NOT_LOAD_GLOBALS_, just in case.

Two (silly) questions around working with Python and ISSM:

When I tried to compile using Python Version 3.9.10 (Installed along side numpy with homebrew), I received the error: Python.h not found error. Reading the thread here, I installed Python 3.9-dev using pyenv and things were resolved. Is this the correct way of dealing with this, or is there potentially something else resulting in this error?
Do I need to work with a Python IDE (e.g., IPython), or should I be able to get things rolling by working directly in the terminal with Python?

Thanks,

Lawrence

justinquinn

lawrence-bird
(1) Yes, as far as I know, newer versions of macOS do not include the Python header file, so you have to install your own copy. I personally use Homebrew, but I expect that other installations work just fine. (If you're interested in how we find Python.h given --with-python-version and --with-python-dir during configuration, check out m4/issm_options.m4).
(2) You do not need an IDE or GUI unless you wish to get into plotting. All of the computational features will be available to you. You might try pinging @bdefleurian for advice on generating plots under Python.

lawrence-bird

justinquinn
Hi Justin:

Good news! After a few days away, I simply made a clean build of ISSM and I'm now up and running with Python3.9 on my M1 machine!

Using PyCharm, I'm able to walk through the Square Ice Shelf model and generate the resultant basic plot. I've also been able to successfully walk through the majority of the mesh example.

One thing I wanted to confirm - I understand that using ISSM within Python is not fully supported. Does this mean that there are core components of the model that simply don't work in Python, or just that there are fewer users/folks maintaining this? This comes up in the mesh tutorial, when calling exptool() - I can't seem to locate this function within the Python build. Just wondering if this is a result of some of these tools not being available in Python, or something more sinister.

Ultimately, I need to make a decision on which route to go with ISSM (Python - via my laptop, or MATLAB - via remote access to a machine with an Intel processor). I would be interested to explore using the Python version, perhaps with some support from R for pre-/post-processing (simply because I am much more familiar with this framework), but don't want to short change myself too much on built-in available features.

Thanks.

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
That is great news! Thanks for updating us and this thread.

There are indeed some missing components of the Python API, and this does have to do with maintainability. Most of our core developers use the MATLAB API, so when new modules are developed, they sometimes do not get translates to Python right away. As well, modules that have plotting capabilities written for MATLAB are not going to have an easy or direct translation to multiple Python plotting libraries, so we have left it up to the user to decided which library works best for them. In the future, we may decide to go with a default plotting library, but there are no plans currently for this.

If you wanted to go about patching missing functionality/modules, I would go through the following steps:

Check if the missing function is a MATLAB builtin by searching the MATLAB documentation. In many cases there is an identical or similar function in NumPy.

Check if the function call is a module, and if so, if the Python version is missing. For example, from $ISSM_DIR, I might run,

$ find src -name exptool.*
src/m/exp/exptool.m

which shows that the exptool module indeed has not been translated yet. If you were to make such translations I think we would be happy to have you on as contributing developer and get you access to the development branch.

I understand not wanting to commit to the "lesser" of the available APIs, but I'm not sure there is really a huge gap in functionality besides the plotting capabilities. Indeed, most of what ISSM does is actually handled by the C++ core while the APIs are mostly for file I/O, model setup, result retrieval, and plotting. I'd be happy to help come up with a rough number of missing Python modules if you think that would help you make your decision.

Best,
Justin

lawrence-bird

justinquinn
Hi Justin:

Thanks for the summary here - this is what I suspected. I guess my biggest concern is running into snags down the road and having a limited community support for the Python API. That being said, I think it would be really helpful to have an idea how many/which modules may be missing from the Python API. If I could identify if these are generally pretty niche and I am unlikely to require them, that would probably sway me.

Any help you can provide on this would be great!

Thanks.

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
The ISSM Doxygen page indicates that there about 1000 MATLAB files and about 300 Python files in the repo. I don't know that this strictly means that about 2/3 of the capabilities are unimplemented under Python, though it might indeed be something like 50% coverage. I would run through a couple more of the tutorials and see if anything is obviously missing. Again, we may have funding for fleshing out the Python API fully in the near future, but there are no hard plans for this.

Best,
Justin

lawrence-bird

justinquinn
Thanks Justin. I persevered with the Python setup for a couple additional tutorials. I'm running into some issues with the Inversion tutorial, but I am not convinced its due to lack of python modules. I think it stems back to the build on my M1 machine.

When running md=solve(md, 'Stressbalance') with md.inversion.iscontrol=1, the process seems to stall without actually returning any solutions. Below is the full output I get after calling this solve() command:

============================================================ Binary file Square.outbin not found This typically happens when the run crashed. Please check for error messages above or in the outlog ============================================================ WARNING: Square.outbin does not exist [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.15.2, Jul 10, 2021 [0]PETSC ERROR: /Users/lbir0005/ISSM/trunk/bin/issm.exe on a named MU00186957X by lbir0005 Fri Feb 11 15:49:29 2022 [0]PETSC ERROR: Configure options --prefix=/Users/lbir0005/ISSM/trunk/externalpackages/petsc/install --PETSC_DIR=/Users/lbir0005/ISSM/trunk/externalpackages/petsc/src --CFLAGS=-Wno-error=implicit-function-declaration --FFLAGS=-fallow-argument-mismatch --LDFLAGS=-Wl,-no_compact_unwind --with-debugging=0 --with-valgrind=0 --with-x=0 --with-ssl=0 --with-pic=1 --download-fblaslapack=1 --download-mpich=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-zlib=1 [0]PETSC ERROR: #1 User provided function() at unknown file:0 [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash. application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 Traceback (most recent call last): File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 90, in runcode exec(code, self.locals) File "<input>", line 12, in <module> File "/Users/lbir0005/ISSM/trunk/bin/solve.py", line 164, in solve md = loadresultsfromcluster(md) File "/Users/lbir0005/ISSM/trunk/bin/loadresultsfromcluster.py", line 68, in loadresultsfromcluster remove(md.private.runtimename + '.tar.gz') AttributeError: 'NoneType' object has no attribute 'private'

It looks like this has something to do with Petsc / MPI - I've included my ./configure options below in case there is anything here that you suggest changing.

./configure \ --prefix="$ISSM_DIR" \ --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \ --with-cxxoptflags="-D_DO_NOT_LOAD_GLOBALS_ -g -O2 -fPIC -std=c++11" \ --with-python-dir="/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9" \ --with-python-version=3.9 \ --with-python-numpy-dir="/opt/homebrew/Cellar/numpy/1.22.1/lib/python3.9/site-packages/numpy/core/include/numpy" \ --with-fortran-lib="-L/usr/local/gfortran/lib -lgfortran" \ --with-mpi-include="$ISSM_DIR/externalpackages/petsc/install/include" \ --with-mpi-libflags="-L$ISSM_DIR/externalpackages/petsc/install/lib -lmpi -lmpicxx -lmpifort" \ --with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install/" \ --with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install/" \ --with-blas-lapack-dir="$ISSM_DIR/externalpackages/petsc/install" \ --with-metis-dir="$ISSM_DIR/externalpackages/petsc/install/" \ --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \ --with-numthreads=2

I suspect it's some Petsc intricacies, but I'm all ears is you have any ideas!

Cheers,

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
This indicates that some part of the solution step crashed. I do not believe it has anything explicitly to do with your PETSc config. Could you copy and paste the script that you were trying to run so I can see if it crashes on my end?
Best,
Justin

lawrence-bird

justinquinn
Hi Justin:

I was largely walking through the runme.py script from the Inversion tutorial, with a few tweaks. I've posted the script below - The failure happens on the last line (when solving the inversion). Turning the inversion off (md.inversion.iscontrol = 0) removes the error. I've been having an issue installing the NetCDF4 library, so I haven't been using the export_NetCDF command to save the model to file.

import sys import numpy as np import os sys.path.append('/Users/lbir0005/ISSM/trunk/bin/') sys.path.append('/Users/lbir0005/ISSM/trunk/lib/') # from model import * from triangle import * from setmask import setmask from parameterize import parameterize from setflowequation import setflowequation from generic import generic from solve import solve from plotmodel import plotmodel #from export_netCDF import export_netCDF from m1qn3inversion import m1qn3inversion from verbose import verbose #from loadmodel import loadmodel from cuffey import cuffey' # #Set working directory to Inversion to make use of files os.chdir("/Users/lbir0005/ISSM/trunk/examples/Inversion") # steps = [1, 2, 3, 4] Clims = [1.3 * 1e8, 1.9 * 1e8] # if 1 in steps: #Generate observations md = model() md = triangle(md, 'DomainOutline.exp', 100000) md = setmask(md, 'all', '') md = parameterize(md, 'Square.py') md = setflowequation(md, 'SSA', 'all') md.cluster = generic('np', 1) md = solve(md, 'Stressbalance') plotmodel(md, 'data', md.materials.rheology_B) plotmodel(md, 'data', md.results.StressbalanceSolution.Vel) #plotmodel(md, 'axis#all', 'tight', 'data', md.materials.rheology_B, 'caxis', Clims, 'title', '"True" B', 'data', md.results.StressbalanceSolution.Vel, 'title', '"observed velocities"') #export_netCDF(md, 'model1.nc') # if 2 in steps: #Modify rheology, now constant #md = loadmodel('model1.nc') md.materials.rheology_B[:] = 1.8 * 1e8 # #results of previous run are taken as observations md.inversion = m1qn3inversion() md.inversion.vx_obs = md.results.StressbalanceSolution.Vx md.inversion.vy_obs = md.results.StressbalanceSolution.Vy md.inversion.vel_obs = md.results.StressbalanceSolution.Vel # md = solve(md, 'Stressbalance') plotmodel(md, 'data', md.materials.rheology_B) plotmodel(md, 'data', md.results.StressbalanceSolution.Vel) #plotmodel(md, 'axis#all', 'tight', 'data', md.materials.rheology_B, 'caxis', Clims, 'title', 'B first guess', 'data', md.results.StressbalanceSolution.Vel, 'title', 'modeled velocities') # if 3 in steps: #invert for ice rigidity #md = loadmodel('model2.nc') # #Set up inversion parameters maxsteps = 20 md.inversion.iscontrol = 1 md.inversion.control_parameters = ['MaterialsRheologyBbar'] md.inversion.maxsteps = maxsteps md.inversion.cost_functions = [101] md.inversion.cost_functions_coefficients = np.ones((md.mesh.numberofvertices, 1)) md.inversion.min_parameters = cuffey(273) * np.ones((md.mesh.numberofvertices, 1)) md.inversion.max_parameters = cuffey(200) * np.ones((md.mesh.numberofvertices, 1)) # #Go solve! #md.verbose = verbose(0) md.verbose.solution = True md = solve(md, 'Stressbalance')

Thanks,

Lawrence

lawrence-bird

Hi justinquinn

I haven't been able to resolve this issue unfortunately - I wondered if you were able to execute this script on your end?

Thanks.

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
Sorry for the delay. Yes: I had to add import devpath and update the paths to the respective locations on my machine, but was able to run the script successfully otherwise. My guess is that this is a PETSc issue and is related specifically to running on M1. Have you had any luck digging up info pertaining to the M1 chipset on the PETSc mailing list?

lawrence-bird

justinquinn
Hi Justin:

No worries at all - I appreciate your help! Thanks for testing the script - it's helpful to narrow down the problem. I haven't dug around in PETSc mailing lists or elsewhere. But, I notice that the PETSc Homebrew download supposedly provides full support for Apple Silicon on Monterey. Perhaps I'll attempt a fresh install of all packages using Homebrew, rather than the ISSM External Package scripts? If nothing more than to make sure that I'm grabbing the latest and greatest of all packages.

Unless you see any immediate risk to trying this approach, or an obvious reason that this would not be worthwhile?

Thanks.

Lawrence

justinquinn

lawrence-bird

Hi Lawrence,
Because you're on the bleeding edge with M1, I don't see any issue with at least trying the Homebrew copy of PETSc while we're waiting for developers to catch up. That said, if you look at the PETSc install scripts, we use it to install a bunch of external packages that they manage. You would have to, as you mentioned, also install all of these packages, modify etc/environment.sh to properly handle their locations on disk (if needed), as well as modify m4/issm_options.m4 before configuring/compiling ISSM (if needed). In the best case, this is not too much of a headache. In the worst case, you end up doing considerable customization.

lawrence-bird

justinquinn

Thanks Justin - I'll have a play with a few things and see if I can come up with anything.

I came across this discussion (https://issm.ess.uci.edu/forum/d/265-creative-meshing) that also makes note of this PETSc error (although perhaps not on M1). @mathieumorlighem - do you have any recollection of how this was overcome in this instance?

Thanks!

Lawrence

lawrence-bird

justinquinn

Hi Justin:

Just an update on this, for your reference. I've played with installing packages from Homebrew and have been even less successful than my original install - I can't get ISSM to build. Given that many packages depend on gcc, this is installed by Homebrew as a dependancy for many of the packages. Despite the Homebrew documentation suggesting that 11.2 is a stable release for Monterey on Apple Silicon chips, I'm running into the same ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64 error I faced before during the make process.

Even if I specify the other gfortran (11.0) that @mathieumorlighem noted earlier in our discussion in my ISSM configuration, it is unhappy.

I'm out of ideas at this point. Fingers crossed things catch up to the new chips soon!

Cheers,

Lawrence

mathieumorlighem

Hi @lawrence-bird, these are segmentation faults and are generally due to bugs in the code. We have done so much debugging over the years that they are very rare now and generally happen when we do something "not standard".
Cheers
Mathieu

lawrence-bird

mathieumorlighem

Thanks Matthieu. I guess this gets chalked up to installation on an M1 machine being non-standard at the moment! 😆

lawrence-bird

Hi @justinquinn

I keep tinkering with this in the hope that it will be useful to me, or someone else! I think I've narrowed down the issue (at least a little). Apologies for the longwinded post, but I wanted to document this progress.

Using the ISSM external package install scripts, I've been able to successfully build and run ISSM using the gfortran version here (based on initial recommendation from @mathieumorlighem). However, this build is where I ran into issue with the inversion. This issue looks to stem from the PETSc install.

To try and rectify this, I have been attempting a new build with updates to everything. I updated the package versions called in the ISSM external package installation scripts, and was able to return another successful build using this same gfortan version. The inversion issue remained.

Following this, I grabbed the different experimental gfortran version (here) built intentionally for Monterey on ARM machines. Using this gfortran version, all external packages install successfully, but when running make, I get the pesky ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64 error. This is where I get [alot] out of my depth.

I chased down that the libgfortran.dylib is linked to the libgcc_s.1.1.dylib, so I suspect this is where the issue arises. I think it's the use of @rpath here that's the issue. In the original gfortran version (where make is successful), the linked libraries appear to use absolute paths, and not the @rpath call. For example, running otool -L libgfortran.dylib on the original gfortran version returns:

/usr/local/gfortran/lib/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0) /usr/local/gfortran/lib/libgcc_s.2.dylib (compatibility version 2.0.0, current version 2.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)

Running otool -L libgfortran.dylib on the updated gfortran version (where make is unsuccessful) returns:

@rpath/libgfortran.5.dylib (compatibility version 6.0.0, current version 6.0.0) @rpath/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0) @rpath/libgcc_s.1.1.dylib (compatibility version 1.0.0, current version 1.1.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

I tried using the install_name_tool to force the @rpath to be an absolute path. While this worked to return absolute paths when running otool -L libgfortran.dylib, it broke the gfortran build (probably not unsurprisingly!). When trying to re-install PETSc using this brute-forced change, PETSc fails with the following error:

Cannot run executables created with FC. If this machine uses a batch system to submit jobs you will need to configure using ./configure with the additional option --with-batch. Otherwise there is problem with the compilers. Can you compile and run code with your compiler 'gfortran'?

Do you have any knowledge of the use of @rpath and how I might be able to overcome this linking issue? I'm wondering if there is another flag that can be included in the configure/make step (probably needed for the PETSc build itself) that can specify what/where @rpath refers to? My suspicion/fear is that this all needs to happen during the build of gfortran though 🙁.

Apologies if this all appears way out-to-lunch - I'm definitely out of my depth here and have just been sleuthing the internet for potential clues.

Thanks!

Lawrence

justinquinn

lawrence-bird
Hi Lawrence,
@rpath is set during compilation/linking (libgfortran in this case). This is a good thing though: it allows libraries and binaries to be truly relocatable. Here's a good write-up of the use of rpath on macOS. See the bit about install_name_tool -add_rpath to modify the rpath for existing libraries.

Are you compiling gfortran from source? In this case, another way to go about this is to dig up where rpath is being set in the project makefiles/config and modify it there. Have you checked to see if libgcc_s.1.1.dylib exists on your machine? If not, you will need to compile from source I suspect in order to ensure that linking against the correct list of symbols occurs.

justinquinn

lawrence-bird Hi Lawrence, I just submitted a fix for this to our development branch if you have access to it. Otherwise, I can walk you through the two minor changes needed to fix it.

lawrence-bird

justinquinn
Hi Justin:

Both of the gfortran versions work out of the box. I am not compiling from source, but just using the installer files (*.pkg) provided at those links above. The installations work and I'm able to compile a simple fortran test program with both of them. I'm also able to successfully install PETSc using either version.

The issue arises when I try and make ISSM. The process completes successfully using the first gfortran version (with absolute paths), but fails with the other (with the @rpath calls), returning the ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64 error. The libgcc_s.1.1.dylib exists in the same directory as libgfortran.dylib. Interestingly, the gcc package available from homebrew (reported as a stable release for Monterey M1) also uses @rpath and returns the same error (ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64) during the make process of ISSM.

Is it possible that somewhere along the line, the ISSM make file is looking to parse out an absolute path from these linked libraries?

Thanks.
Lawrence

« Previous Page Next Page »