Installation issues related to PETSC on Apple M1 Chip

Aajjenson · Jan 15, 2024

Hi, I am trying to install ISSM on my MacBook Pro (M1 chip) and am having some issues related to PETSC. At least I think this is the problem.

I installed all other external packages without issues. However, I had some issues installing PETSC using the install-3.17-mac-silicon.sh file. It gave me an error saying it could not located a C compiler, which may be an issue with my paths on local machine. It also had issues finding zlib so I downloaded the tar.gz file and directed to it in the installation file which seemed to work. Eventually I thought the install was successful after getting the message below.

Now to install the libraries do:
make PETSC_DIR=/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/src PETSC_ARCH=arch-darwin-c-opt install
=========================================
*** Using PETSC_DIR=/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/src PETSC_ARCH=arch-darwin-c-opt ***
*** Installing PETSc at prefix location: /Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install  ***
====================================
Install complete.
Now to check if the libraries are working do (in current directory):
make PETSC_DIR=/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install PETSC_ARCH="" check
====================================
/usr/bin/make  --no-print-directory -f makefile PETSC_ARCH=arch-darwin-c-opt PETSC_DIR=/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/src petsc4py-install libmesh-install mfem-install slepc-install hpddm-install amrex-install bamg-install
make[2]: Nothing to be done for `petsc4py-install'.
make[2]: Nothing to be done for `libmesh-install'.
make[2]: Nothing to be done for `mfem-install'.
make[2]: Nothing to be done for `slepc-install'.
make[2]: Nothing to be done for `hpddm-install'.
make[2]: Nothing to be done for `amrex-install'.
make[2]: Nothing to be done for `bamg-install'.

Next, I tried to compile ISSM by running the following

$ cd $ISSM_DIR
$ autoreconf -ivf

In configure.sh file, I copied and pasted the following which I found on the forum for M1 macs:

export FFLAGS=" -arch arm64"
export CFLAGS=" -arch arm64"
export LDFLAGS=" -arch arm64"
export CXXFLAGS=" -arch arm64"

./configure \
   --without-Love --without-kml --without-Sealevelchange \
   --prefix=$ISSM_DIR \
   --without-wrappers \
   --enable-debugging \
   --enable-development \
   --with-mpi-include="$ISSM_DIR/externalpackages/petsc/install/" \
   --with-mpi-libflags="-L$ISSM_DIR/externalpackages/petsc/install/ -lmpich" \
   --with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \
   --with-metis-dir="$ISSM_DIR/externalpackages/petsc/install" \
   --with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install/" \
   --with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install/" \
   --with-numthreads=4

$ cd $ISSM_DIR
$ make
$ make install

After running the above lines, I received the following error message:

In file included from ./cores/love_core.cpp:11:
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:39:14: error: use of undeclared identifier 'PETSC_COMM_SELF'
  PetscError(PETSC_COMM_SELF,__LINE__,PETSC_FUNCTION_NAME,__FILE__,PETSC_ERR_SUP,PETSC_ERROR_INITIAL,"%s - Lapack routine is unavailable.",fname);
             ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:39:39: error: use of undeclared identifier 'PETSC_FUNCTION_NAME'
  PetscError(PETSC_COMM_SELF,__LINE__,PETSC_FUNCTION_NAME,__FILE__,PETSC_ERR_SUP,PETSC_ERROR_INITIAL,"%s - Lapack routine is unavailable.",fname);
                                      ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:39:68: error: use of undeclared identifier 'PETSC_ERR_SUP'
  PetscError(PETSC_COMM_SELF,__LINE__,PETSC_FUNCTION_NAME,__FILE__,PETSC_ERR_SUP,PETSC_ERROR_INITIAL,"%s - Lapack routine is unavailable.",fname);
                                                                   ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:39:82: error: use of undeclared identifier 'PETSC_ERROR_INITIAL'
  PetscError(PETSC_COMM_SELF,__LINE__,PETSC_FUNCTION_NAME,__FILE__,PETSC_ERR_SUP,PETSC_ERROR_INITIAL,"%s - Lapack routine is unavailable.",fname);
                                                                                 ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:40:13: error: use of undeclared identifier 'PETSC_COMM_SELF'
  MPI_Abort(PETSC_COMM_SELF,PETSC_ERR_SUP);
            ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:40:29: error: use of undeclared identifier 'PETSC_ERR_SUP'
  MPI_Abort(PETSC_COMM_SELF,PETSC_ERR_SUP);
                            ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:45:31: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKgetrf_(PetscBLASInt*,PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                              ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:45:45: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKgetrf_(PetscBLASInt*,PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                            ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:45:59: error: unknown type name 'PetscScalar'
BLAS_EXTERN void LAPACKgetrf_(PetscBLASInt*,PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                          ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:45:72: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKgetrf_(PetscBLASInt*,PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                                       ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:45:86: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKgetrf_(PetscBLASInt*,PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                                                     ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:45:100: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKgetrf_(PetscBLASInt*,PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                                                                   ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:46:35: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKREALgetrf_(PetscBLASInt*,PetscBLASInt*,PetscReal*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                  ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:46:49: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKREALgetrf_(PetscBLASInt*,PetscBLASInt*,PetscReal*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:46:63: error: unknown type name 'PetscReal'
BLAS_EXTERN void LAPACKREALgetrf_(PetscBLASInt*,PetscBLASInt*,PetscReal*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                              ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:46:74: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKREALgetrf_(PetscBLASInt*,PetscBLASInt*,PetscReal*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                                         ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:46:88: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKREALgetrf_(PetscBLASInt*,PetscBLASInt*,PetscReal*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                                                       ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:46:102: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKREALgetrf_(PetscBLASInt*,PetscBLASInt*,PetscReal*,PetscBLASInt*,PetscBLASInt*,PetscBLASInt*);
                                                                                                     ^
/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/include/petscblaslapack.h:47:31: error: unknown type name 'PetscBLASInt'
BLAS_EXTERN void LAPACKgetri_(PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*,PetscScalar*,PetscBLASInt*,PetscBLASInt*);
                              ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[3]: *** [cores/libISSMCore_la-love_core.lo] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1

Any suggestions as to where the problem lies?

Thanks,
Amy

Hheleneseroussi · Jan 16, 2024

Hi Amy,

It does not look like a problem with petsc installation since that part finished just fined. It looks like there is a problem with the compilation of the love_core.cpp, which is surprising because it should be excluded since your configure.sh has --without-Love .

If you have Matlab version 2023b or later, you will be able to compile including the wrappers and everything will be simpler, so I would suggest updating Matlab if possible. Here is the link to the configure.sh file I use on a similar mac configuration and this version of Matlab: https://drive.google.com/open?id=1aMG-DkVCX4VCWm6jyXCSUIpi7SS1jMqU&usp=drive_fs

If a similar problem continues with a problem in ./cores/love_core.cpp maybe send us you trunk/config.log file so we can try to figure out why your computer tries to compile love_core.cpp while it should not.

Best,
Helene

Aajjenson · Jan 16, 2024

Hi Helene,

Okay, thanks for the help. I updated my MATLAB to 2023b and I ran the configure file you shared. This time I get the error message below after running make.

...
CXX cores/libISSMCore_la-love_core.lo
./cores/love_core.cpp:11:10: fatal error: 'petscblaslapack.h' file not found
#include "petscblaslapack.h"
~~~~~~~~~~~~~_~~
1 error generated.
make[3]: *** [cores/libISSMCore_la-love_core.lo] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

I have attached the config.log file.

config.log

56kB

Hheleneseroussi · Jan 17, 2024

Hi Amy,

you did not call ./configure.sh but ./configure (i.e. without the .sh extension). The script configure.sh calls configure but provides a bunch of options. If you call ./configure directly, you are not passing any options and this is why it is trying to compile some files that you would like to exclude. Try to call

cd $ISSM_DIR
./configure.sh

and let us know if that works for you.

Best
Helene

Aajjenson · Jan 24, 2024

Hi Helen,

Sorry about that (and for the delay). I seem to be having a different issue then. I removed the "--with-blas-lapack-dir" which seemed to resolve an error I was getting, but now I get the following error after running make. I have attached the config.log file as well.

CXXLD libISSMCore.la
ld: warning: dylib (/Users/amyjenson/svn/issm/trunk/externalpackages/petsc/install/lib/libscalapack.dylib) was built for newer macOS version (12.6) than being linked (12.0)
ld: warning: could not create compact unwind for ___mumps_ana_ord_wrappers_MOD_mumps_pordf_wnd_mixedto64: registers 27 and 28 not saved contiguously in frame
ld: warning: could not create compact unwind for ___mumps_ana_ord_wrappers_MOD_mumps_pordf_mixedto64: registers 27 and 28 not saved contiguously in frame
... (many many more warnings like these)...
ld: warning: could not create compact unwind for cmumps_gather_root: registers 27 and 28 not saved contiguously in frame
ld: file not found: @rpath/libquadmath.0.dylib for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libISSMCore.la] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

config.log

59kB

Hheleneseroussi · Jan 29, 2024

Hi Amy,

You cannot remove the --with-blas-lapack-dir unfortunately because we need. Can you please tell me what the other error was when it was included?

Thanks

Jjustinquinn · Jan 29, 2024

ajjenson Regarding ld: file not found: @rpath/libquadmath.0.dylib for architecture arm64, there are two possibilities here:
1) You should try adding export LIBRARY_PATH="{$LIBRARY_PATH}:<PATH_TO_LIBQUADMATH>" to your shell profile (you can test first in your current terminal session). You might also try the same with LD_LIBRARY_PATH. If you don't know where libquadmath is installed, you might search with find /usr -name libquadmath* or find /opt -name libquadmath*. NOTE: <PATH_TO_LIBQUADMATH> should be the parent directory of libquadmath.
2) You may have the wrong version of gfortran installed for your architecture (seeing this more with owners of Silicon-based Macs installing Intel-based version).

Aajjenson · Jan 31, 2024

heleneseroussi
Oh sorry. I saw "NOTE: If you get the following error: ld: library not found for -lflapack, remove the line -with-blas-lapack-dir (generally on macOS)." on the installation page so I thought it wasn't necessary. I added it back in.

justinquinn
Defining the right path worked.

ISSM is now successfully installed my Mac. I also now have ISSM installed on another computer I have remote access to. Thanks to both of you!

Aahayden · May 2, 2024

Hi all,

I'm also experiencing some similar issues. I'm using Sonoma on an Apple M1 Chip and Matlab 2024a. This is a fresh download of the non-developer version. I didn't have any issues or error messages pop up when I downloaded the packages.

The steps I followed were

cd $ISSM_DIR
autoreconf -ivf
./configure.sh
make

These are the errors that pop up after running make
https://issm.ess.uci.edu/forum/assets/files/2024-05-02/1714666574-373477-image.png

And here's my configure file:

#!/bin/bash
./configure \
--without-Love --without-kml --without-Sealevelchange \
--prefix=${ISSM_DIR} \
--without-wrappers \
--with-numthreads=2 \
--with-matlab-dir="/Applications/MATLAB_R2024a.app" \
--with-fortran-lib="-L/Library/Developer/CommandLineTools/usr/bin -lgfortran" \
--with-mpi-include="${ISSM_DIR}/externalpackages/petsc/install/include" \
--with-mpi-libflags="-L${ISSM_DIR}/externalpackages/petsc/install/lib -lmpi -lmpicxx -lmpifort" \
--with-metis-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-parmetis-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-blas-lapack-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-scalapack-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-mumps-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-petsc-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-triangle-dir="${ISSM_DIR}/externalpackages/triangle/install" \
--with-chaco-dir="${ISSM_DIR}/externalpackages/chaco/install" \
--with-m1qn3-dir="${ISSM_DIR}/externalpackages/m1qn3/install"

Please let me know if any other info might be helpful!

Cheers,

Anna

Aahayden · May 2, 2024

ahayden

Hi all,

Turns out I wasn't running ./configure.sh, but just ./configure. I had to change the permissions of configure.sh to get it to run, but then there are still errors for make:

Jjustinquinn · May 2, 2024

Hi Anna, did you run source $ISSM_DIR/etc/environment.sh before configure? Also, you have --without-Love, so this module shouldn't even be getting compiled.

As an aside, is your path to libgfortran good? Is there a libgfortran.dylib in /Library/Developer/CommandLineTools/usr/bin? CLT doesn't come with gfortran and I don't believe that's where it is installed anyhow. You can run find /opt /usr -name libgfortran*.dylib to find the proper path.

Aahayden · May 2, 2024

justinquinn

Yes, I ran source $ISSM_DIR/etc/environment.sh.

I downloaded gfortran from here: https://hpc.sourceforge.net/, then moved the files to the /Library/Developer/CommandLineTools/usr/bin folder. When I checked, libgfortran.dylib is in /usr/local/lib/libgfortran.dylib

Jjustinquinn · May 2, 2024

ahayden Ah, okay, I wasn't aware of the Sourceforge distribution and that that's the install location.

Long shot, but you might try breaking the --without-Love, --without-kml, and --without-Sealevelchange options on to individual lines? It may be that configure is parsing the single line incorrectly. Again, love_core.cpp should not even be getting compiled. If that doesn't work, you might try autoreconf -ivf, then configure again, then make; make install. It may be that you added the --without-Love option since your last configure.

Another debugging step, generally, would be to check that petscblaslapack.h is in $ISSM_DIR/petsc/install/include.

Aahayden · May 2, 2024

justinquinn
Hi Justin,
Thanks for those suggestions!!

I've made a few changes to my configure.sh file (including deleting the chaco line, which was also causing problems - I removed the appropriate install in the corresponding folder in externalpackages/chaco - but I've since added the chaco line back in). As you suggested, I split up the options on three lines.

Now, different errors are coming up when I run ./configure.sh

But, I did install petsc without any issues:

I installed 3.17-mac-silicon.sh. However, it doesn't look like there's an "include" directory"

I've attached the updated configure.sh file

configure.sh

1kB

All the best,

Anna

Jjustinquinn · May 2, 2024

ahayden Hi Anna,
Can you double check with ls $ISSM_DIR/externalpackages/petsc/install? Also, you should probably use the latest version of the PETSc install script. In case you have a copy of trunk/ and not trunk-jpl/ (the development branch), I'll attach it here. Simply download it to $ISSM_DIR/externalpackages/petsc, rename it to /install-3.21-mac.sh, modify permissions with chmod +x $ISSM_DIR/externalpackages/petsc/install-3.21-mac.sh, then run it. You will also need an updated copy of $ISSM_DIR/etc/environment.sh, which I'll also attach. (Note that these changes will be available on the public branch the next time we merge.)

install-321-mac.sh

1kB

environment.sh

20kB

Aahayden · May 3, 2024

justinquinn

Hi Justin,

Ok, prior to updating the install and environment files, I double-checked the install folder, and it was empty. Then, after following your instructions, there is a /include folder. However, it doesn't have the petscblaslapack.h file.

Then, after I ran
$ cd $ISSM_DIR
$ autoreconf -ivf
$ ./configure.sh

I get the error
./configure.sh: line 14: --with-metis-dir=/Users/annahayden/Dropbox/trunk/externalpackages/petsc/install: No such file or directory

But the directory does exist... Maybe something went wrong when I ran ./configure instead of ./configure.sh?

Jjustinquinn · May 3, 2024

ahayden There's an off-chance that the permissions were somehow messed up on this directory. To test, you could run sudo chmod -R 755 /Users/annahayden/Dropbox/trunk/externalpackages/petsc/install, then try the ISSM install again. If that still doesn't work, you might try a reinstall of PETSc.

Aahayden · May 3, 2024

justinquinn
Hi Justin,
Thanks for sticking with me! Just want to clarify something, when you say

try the ISSM install again

Do you mean start from scratch i.e., re-try downloading trunk using svn?

Jjustinquinn · May 3, 2024

ahayden No worries at all. We'll get you sorted.

No, I mean make; make install.

Aahayden · May 3, 2024

justinquinn
We're getting closer! I reinstalled PETSc. then ran make, which didn't result in any errors!

However make install didn't work: