lawrence-bird
Hi Lawrence,
I'm sorry, I'm running out of ideas on what might be the issue here. It's really hard for me to test on my end without access to an M1 machine. Are you getting the same error as before with the rolled back copy of gfortran?

I can confirm that the fatal errors in the config.log are not an issue:

$ grep "fatal error" config.log 
config.log:155:conftest.c:9:10: fatal error: 'ac_nonexistent.h' file not found
config.log:174:conftest.c:9:10: fatal error: 'ac_nonexistent.h' file not found
config.log:256:gfortran: fatal error: no input files
config.log:261:gfortran: fatal error: no input files
config.log:297:gfortran: fatal error: no input files
config.log:302:gfortran: fatal error: no input files
config.log:545:conftest.cpp:23:10: fatal error: 'ac_nonexistent.h' file not found
config.log:578:conftest.cpp:23:10: fatal error: 'ac_nonexistent.h' file not found

That's from a successful config and compilation.

    justinquinn
    Hi Justin:

    No worries - this is a strange one.

    I m happy to report that I was able to get ISSM to compile! But I've now got a new issue. On the compilation front, this was my process:

    • Grabbed a clean version from the trunk
    • Downloaded the gfortran 11-experimental-2 for ARM Big Sur (macOS 11) version Mathieu noted above
    • Installed all external packages (noting the updated to LIBTOOL_VER=2.46 for autotools
    • Ran autoreconf -ivf at $ISSM_DIR
    • Installed numpy / scipy using homebrew
    • Installed pyenv using homebrew (to allow me to get a dev copy of Python)
    • Installed python 3.9-dev using pyenv
    • Specified the --with-fortran-lib in the configure script to point to the location of gfortran that I'd downloaded. This step was important as another gcc version is installed using homebrew as a dependancy for numpy. Without specifying the fortran location, it would try and use the gcc location in homebrew (based, I presume, on it's location in the $PATH), and it would fail. Interestingly, the gcc version downloaded as a dependancy for numpy is that same version I was originally trying (for Monterey).
    • Specified the python directory (in pyenv) and the python version (3.9) in the configure script.
    • ./configure
    • make
    • make install

    This appears to result in a successful compilation!

    However, when I try and run the model via Python, I am returned a Segmentation fault: 11. After launching Python, I am entering the following (before be returned the seg fault):

    import sys
    sys.path.append('/Users/lbir0005/ISSM/trunk/bin/')
    sys.path.append('/Users/lbir0005/ISSM/trunk/lib/')
    from issmversion import issmversion

    I was able to successfully build ISSM with Matlab on a different machine that I am waiting to get remote access right for.

    At this point, I am just being stubborn about trying to get it working on my laptop with Python.

    Cheers,

    Lawrence

      lawrence-bird
      Nice! Glad to hear you got through compilation at least. The only other suggestions I could make is to make sure you incorporate the following options into your configuration (adjusting the paths as needed, of course):

      --with-cxxoptflags="-D_DO_NOT_LOAD_GLOBALS_ -g -O2 -fPIC -std=c++11" \
      --with-python-version=3.9 \
      --with-python-dir=/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9 \
      --with-python-numpy-dir=/usr/local/lib/python3.9/site-packages/numpy

      I've included the -D_DO_NOT_LOAD_GLOBALS_ flag as a test because you did not receive an error related to it, but we have seen issues with more recent GCC versions that this has been a short term fix for.

      Please do let us know if you get everything compiled and running with the Python API as you're in uncharted territory and it will be of use to others. Thanks!

        justinquinn

        Thanks Justin - I'll try adding the -D_DO_NOT_LOAD_GLOBALS_, just in case.

        Two (silly) questions around working with Python and ISSM:

        • When I tried to compile using Python Version 3.9.10 (Installed along side numpy with homebrew), I received the error: Python.h not found error. Reading the thread here, I installed Python 3.9-dev using pyenv and things were resolved. Is this the correct way of dealing with this, or is there potentially something else resulting in this error?
        • Do I need to work with a Python IDE (e.g., IPython), or should I be able to get things rolling by working directly in the terminal with Python?

        Thanks,

        Lawrence

          lawrence-bird
          (1) Yes, as far as I know, newer versions of macOS do not include the Python header file, so you have to install your own copy. I personally use Homebrew, but I expect that other installations work just fine. (If you're interested in how we find Python.h given --with-python-version and --with-python-dir during configuration, check out m4/issm_options.m4).
          (2) You do not need an IDE or GUI unless you wish to get into plotting. All of the computational features will be available to you. You might try pinging @bdefleurian for advice on generating plots under Python.

            4 days later

            justinquinn
            Hi Justin:

            Good news! After a few days away, I simply made a clean build of ISSM and I'm now up and running with Python3.9 on my M1 machine!

            Using PyCharm, I'm able to walk through the Square Ice Shelf model and generate the resultant basic plot. I've also been able to successfully walk through the majority of the mesh example.

            One thing I wanted to confirm - I understand that using ISSM within Python is not fully supported. Does this mean that there are core components of the model that simply don't work in Python, or just that there are fewer users/folks maintaining this? This comes up in the mesh tutorial, when calling exptool() - I can't seem to locate this function within the Python build. Just wondering if this is a result of some of these tools not being available in Python, or something more sinister.

            Ultimately, I need to make a decision on which route to go with ISSM (Python - via my laptop, or MATLAB - via remote access to a machine with an Intel processor). I would be interested to explore using the Python version, perhaps with some support from R for pre-/post-processing (simply because I am much more familiar with this framework), but don't want to short change myself too much on built-in available features.

            Thanks.

            Lawrence

              lawrence-bird
              Hi Lawrence,
              That is great news! Thanks for updating us and this thread.

              There are indeed some missing components of the Python API, and this does have to do with maintainability. Most of our core developers use the MATLAB API, so when new modules are developed, they sometimes do not get translates to Python right away. As well, modules that have plotting capabilities written for MATLAB are not going to have an easy or direct translation to multiple Python plotting libraries, so we have left it up to the user to decided which library works best for them. In the future, we may decide to go with a default plotting library, but there are no plans currently for this.

              If you wanted to go about patching missing functionality/modules, I would go through the following steps:

              • Check if the missing function is a MATLAB builtin by searching the MATLAB documentation. In many cases there is an identical or similar function in NumPy.
              • Check if the function call is a module, and if so, if the Python version is missing. For example, from $ISSM_DIR, I might run,
                $ find src -name exptool.*
                src/m/exp/exptool.m
                which shows that the exptool module indeed has not been translated yet. If you were to make such translations I think we would be happy to have you on as contributing developer and get you access to the development branch.

              I understand not wanting to commit to the "lesser" of the available APIs, but I'm not sure there is really a huge gap in functionality besides the plotting capabilities. Indeed, most of what ISSM does is actually handled by the C++ core while the APIs are mostly for file I/O, model setup, result retrieval, and plotting. I'd be happy to help come up with a rough number of missing Python modules if you think that would help you make your decision.

              Best,
              Justin

                justinquinn
                Hi Justin:

                Thanks for the summary here - this is what I suspected. I guess my biggest concern is running into snags down the road and having a limited community support for the Python API. That being said, I think it would be really helpful to have an idea how many/which modules may be missing from the Python API. If I could identify if these are generally pretty niche and I am unlikely to require them, that would probably sway me.

                Any help you can provide on this would be great!

                Thanks.

                Lawrence

                  lawrence-bird
                  Hi Lawrence,
                  The ISSM Doxygen page indicates that there about 1000 MATLAB files and about 300 Python files in the repo. I don't know that this strictly means that about 2/3 of the capabilities are unimplemented under Python, though it might indeed be something like 50% coverage. I would run through a couple more of the tutorials and see if anything is obviously missing. Again, we may have funding for fleshing out the Python API fully in the near future, but there are no hard plans for this.

                  Best,
                  Justin

                    justinquinn
                    Thanks Justin. I persevered with the Python setup for a couple additional tutorials. I'm running into some issues with the Inversion tutorial, but I am not convinced its due to lack of python modules. I think it stems back to the build on my M1 machine.

                    When running md=solve(md, 'Stressbalance') with md.inversion.iscontrol=1, the process seems to stall without actually returning any solutions. Below is the full output I get after calling this solve() command:

                    ============================================================
                    Binary file Square.outbin not found
                    This typically happens when the run crashed.
                    Please check for error messages above or in the outlog ============================================================
                    WARNING: Square.outbin does not exist
                    [0]PETSC ERROR: ------------------------------------------------------------------------
                    [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
                    [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
                    [0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
                    [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
                    [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
                    [0]PETSC ERROR: to get more information on the crash.
                    [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
                    [0]PETSC ERROR: Signal received
                    [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
                    [0]PETSC ERROR: Petsc Release Version 3.15.2, Jul 10, 2021
                    [0]PETSC ERROR: /Users/lbir0005/ISSM/trunk/bin/issm.exe on a named MU00186957X by lbir0005 Fri Feb 11 15:49:29 2022
                    [0]PETSC ERROR: Configure options --prefix=/Users/lbir0005/ISSM/trunk/externalpackages/petsc/install --PETSC_DIR=/Users/lbir0005/ISSM/trunk/externalpackages/petsc/src --CFLAGS=-Wno-error=implicit-function-declaration --FFLAGS=-fallow-argument-mismatch --LDFLAGS=-Wl,-no_compact_unwind --with-debugging=0 --with-valgrind=0 --with-x=0 --with-ssl=0 --with-pic=1 --download-fblaslapack=1 --download-mpich=1 --download-metis=1 --download-parmetis=1 --download-scalapack=1 --download-mumps=1 --download-zlib=1
                    [0]PETSC ERROR: #1 User provided function() at unknown file:0
                    [0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
                    application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
                    Traceback (most recent call last):
                    File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 90, in runcode
                    exec(code, self.locals)
                    File "<input>", line 12, in <module>
                    File "/Users/lbir0005/ISSM/trunk/bin/solve.py", line 164, in solve
                    md = loadresultsfromcluster(md)
                    File "/Users/lbir0005/ISSM/trunk/bin/loadresultsfromcluster.py", line 68, in loadresultsfromcluster
                    remove(md.private.runtimename + '.tar.gz')
                    AttributeError: 'NoneType' object has no attribute 'private'

                    It looks like this has something to do with Petsc / MPI - I've included my ./configure options below in case there is anything here that you suggest changing.

                    ./configure \
                    --prefix="$ISSM_DIR" \
                    --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \
                    --with-cxxoptflags="-D_DO_NOT_LOAD_GLOBALS_ -g -O2 -fPIC -std=c++11" \
                    --with-python-dir="/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9" \
                    --with-python-version=3.9 \
                    --with-python-numpy-dir="/opt/homebrew/Cellar/numpy/1.22.1/lib/python3.9/site-packages/numpy/core/include/numpy" \
                    --with-fortran-lib="-L/usr/local/gfortran/lib -lgfortran" \
                    --with-mpi-include="$ISSM_DIR/externalpackages/petsc/install/include" \
                    --with-mpi-libflags="-L$ISSM_DIR/externalpackages/petsc/install/lib -lmpi -lmpicxx -lmpifort" \
                    --with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \
                    --with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install/" \
                    --with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install/" \
                    --with-blas-lapack-dir="$ISSM_DIR/externalpackages/petsc/install" \
                    --with-metis-dir="$ISSM_DIR/externalpackages/petsc/install/" \
                    --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \
                    --with-numthreads=2

                    I suspect it's some Petsc intricacies, but I'm all ears is you have any ideas!

                    Cheers,

                    Lawrence

                      lawrence-bird
                      Hi Lawrence,
                      This indicates that some part of the solution step crashed. I do not believe it has anything explicitly to do with your PETSc config. Could you copy and paste the script that you were trying to run so I can see if it crashes on my end?
                      Best,
                      Justin

                        justinquinn
                        Hi Justin:

                        I was largely walking through the runme.py script from the Inversion tutorial, with a few tweaks. I've posted the script below - The failure happens on the last line (when solving the inversion). Turning the inversion off (md.inversion.iscontrol = 0) removes the error. I've been having an issue installing the NetCDF4 library, so I haven't been using the export_NetCDF command to save the model to file.

                        import sys
                        import numpy as np
                        import os
                        sys.path.append('/Users/lbir0005/ISSM/trunk/bin/')
                        sys.path.append('/Users/lbir0005/ISSM/trunk/lib/')
                        #
                        from model import *
                        from triangle import *
                        from setmask import setmask
                        from parameterize import parameterize
                        from setflowequation import setflowequation
                        from generic import generic
                        from solve import solve
                        from plotmodel import plotmodel
                        #from export_netCDF import export_netCDF
                        from m1qn3inversion import m1qn3inversion
                        from verbose import verbose
                        #from loadmodel import loadmodel
                        from cuffey import cuffey'
                        #
                        #Set working directory to Inversion to make use of files
                        os.chdir("/Users/lbir0005/ISSM/trunk/examples/Inversion")
                        #
                        steps = [1, 2, 3, 4]
                        Clims = [1.3 * 1e8, 1.9 * 1e8]
                        #
                        if 1 in steps:
                        #Generate observations
                        md = model()
                        md = triangle(md, 'DomainOutline.exp', 100000)
                        md = setmask(md, 'all', '')
                        md = parameterize(md, 'Square.py')
                        md = setflowequation(md, 'SSA', 'all')
                        md.cluster = generic('np', 1)
                        md = solve(md, 'Stressbalance')
                        plotmodel(md, 'data', md.materials.rheology_B)
                        plotmodel(md, 'data', md.results.StressbalanceSolution.Vel)
                        #plotmodel(md, 'axis#all', 'tight', 'data', md.materials.rheology_B, 'caxis', Clims, 'title', '"True" B', 'data', md.results.StressbalanceSolution.Vel, 'title', '"observed velocities"')
                        #export_netCDF(md, 'model1.nc')
                        #
                        if 2 in steps:
                        #Modify rheology, now constant
                        #md = loadmodel('model1.nc')
                        md.materials.rheology_B[:] = 1.8 * 1e8
                        #
                        #results of previous run are taken as observations
                        md.inversion = m1qn3inversion()
                        md.inversion.vx_obs = md.results.StressbalanceSolution.Vx
                        md.inversion.vy_obs = md.results.StressbalanceSolution.Vy
                        md.inversion.vel_obs = md.results.StressbalanceSolution.Vel
                        #
                        md = solve(md, 'Stressbalance')
                        plotmodel(md, 'data', md.materials.rheology_B)
                        plotmodel(md, 'data', md.results.StressbalanceSolution.Vel)
                        #plotmodel(md, 'axis#all', 'tight', 'data', md.materials.rheology_B, 'caxis', Clims, 'title', 'B first guess', 'data', md.results.StressbalanceSolution.Vel, 'title', 'modeled velocities')
                        #
                        if 3 in steps:
                        #invert for ice rigidity
                        #md = loadmodel('model2.nc')
                        #
                        #Set up inversion parameters
                        maxsteps = 20
                        md.inversion.iscontrol = 1
                        md.inversion.control_parameters = ['MaterialsRheologyBbar']
                        md.inversion.maxsteps = maxsteps
                        md.inversion.cost_functions = [101]
                        md.inversion.cost_functions_coefficients = np.ones((md.mesh.numberofvertices, 1))
                        md.inversion.min_parameters = cuffey(273) * np.ones((md.mesh.numberofvertices, 1))
                        md.inversion.max_parameters = cuffey(200) * np.ones((md.mesh.numberofvertices, 1))
                        #
                        #Go solve!
                        #md.verbose = verbose(0)
                        md.verbose.solution = True
                        md = solve(md, 'Stressbalance')

                        Thanks,

                        Lawrence

                        5 days later

                        Hi justinquinn

                        I haven't been able to resolve this issue unfortunately - I wondered if you were able to execute this script on your end?

                        Thanks.

                        Lawrence

                          lawrence-bird
                          Hi Lawrence,
                          Sorry for the delay. Yes: I had to add import devpath and update the paths to the respective locations on my machine, but was able to run the script successfully otherwise. My guess is that this is a PETSc issue and is related specifically to running on M1. Have you had any luck digging up info pertaining to the M1 chipset on the PETSc mailing list?

                            justinquinn
                            Hi Justin:

                            No worries at all - I appreciate your help! Thanks for testing the script - it's helpful to narrow down the problem. I haven't dug around in PETSc mailing lists or elsewhere. But, I notice that the PETSc Homebrew download supposedly provides full support for Apple Silicon on Monterey. Perhaps I'll attempt a fresh install of all packages using Homebrew, rather than the ISSM External Package scripts? If nothing more than to make sure that I'm grabbing the latest and greatest of all packages.

                            Unless you see any immediate risk to trying this approach, or an obvious reason that this would not be worthwhile?

                            Thanks.

                            Lawrence

                              8 days later

                              lawrence-bird

                              Hi Lawrence,
                              Because you're on the bleeding edge with M1, I don't see any issue with at least trying the Homebrew copy of PETSc while we're waiting for developers to catch up. That said, if you look at the PETSc install scripts, we use it to install a bunch of external packages that they manage. You would have to, as you mentioned, also install all of these packages, modify etc/environment.sh to properly handle their locations on disk (if needed), as well as modify m4/issm_options.m4 before configuring/compiling ISSM (if needed). In the best case, this is not too much of a headache. In the worst case, you end up doing considerable customization.

                                4 days later

                                Hi @lawrence-bird, these are segmentation faults and are generally due to bugs in the code. We have done so much debugging over the years that they are very rare now and generally happen when we do something "not standard".
                                Cheers
                                Mathieu

                                  justinquinn

                                  Hi Justin:

                                  Just an update on this, for your reference. I've played with installing packages from Homebrew and have been even less successful than my original install - I can't get ISSM to build. Given that many packages depend on gcc, this is installed by Homebrew as a dependancy for many of the packages. Despite the Homebrew documentation suggesting that 11.2 is a stable release for Monterey on Apple Silicon chips, I'm running into the same ld: file not found: @rpath/libgcc_s.1.1.dylib for architecture arm64 error I faced before during the make process.

                                  Even if I specify the other gfortran (11.0) that @mathieumorlighem noted earlier in our discussion in my ISSM configuration, it is unhappy.

                                  I'm out of ideas at this point. Fingers crossed things catch up to the new chips soon!

                                  Cheers,

                                  Lawrence