Error compiling issm from source

anamikaene

I'm attempting to install ISSM from source on my Ubuntu 24.04 system. Below are my notes from attempting to build it following the manual. During compilation (see below) I'm seeing an error about an undefined symbol. I'm not sure if I need to configure differently, or if something else is going wrong - the symbol PythonIOSymbol doesn't seem to be defined in ISSM or any of the dependencies so it is not obvious what is missing. Any help appreciated!

Initial setup:

mkdir -p ~/opt
cd ~/opt
git clone https://github.com/ISSMteam/ISSM
export ISSM_DIR=~/opt/ISSM
source $ISSM_DIR/etc/environment.sh

External package installation:

# autotools
cd $ISSM_DIR/externalpackages/autotools
./install-linux.sh

# cmake
cd $ISSM_DIR/externalpackages/cmake
./install.sh

# petsc
cd $ISSM_DIR/externalpackages/petsc
./install-3.22-linux.sh

# triangle
cd $ISSM_DIR/externalpackages/triangle
./install-linux.sh

#miqn3
cd $ISSM_DIR/externalpackages/m1qn3
./install-linux.sh

Source the environment after installing packages (note: docs actually say to do this after installing each package! Keep this in mind in case it is the reason things are broken later)

source $ISSM_DIR/etc/environment.sh

ISSM compilation:

cd $ISSM_DIR
autoreconf -ivf

Create a configuration script called configure.sh in $ISSM_DIR

./configure \
--prefix="${ISSM_DIR}" \
--with-python-version=3.11 \
--with-python-dir="/usr" \
--with-python-numpy-dir="${HOME}/Desktop/code/.venv/lib/python3.11/site-packages/numpy" \
--with-fortran-lib="-L/usr/lib/gcc/x86_64-linux-gnu/13 -lgfortran" \
--with-mpi-include="${ISSM_DIR}/externalpackages/petsc/install/include" \
--with-mpi-libflags="-L${ISSM_DIR}/externalpackages/petsc/install/lib -lmpi -lmpicxx -lmpifort" \
--with-metis-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-parmetis-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-blas-lapack-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-scalapack-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-mumps-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-petsc-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-triangle-dir="${ISSM_DIR}/externalpackages/triangle/install" \
--with-m1qn3-dir="${ISSM_DIR}/externalpackages/m1qn3/install"

Notes:

python version corresponds to python3.11 installed from the deadsnakes ppa
--with-python-numpy-dir is the path to numpy within a python3.11 virtual environment, created with python3.11 -m venv .venv after installing python3.11 from the deadsnakes ppa - and then installing numpy with pip with the virtual environment active.

the path to gfortran was found by running:

  find /usr -name libgfortran.so*

(perhaps gfortran needed to be apt-get installed before this existed)

Before running the configuration script, install python dev package (also from the deadsnakes ppa) if you haven't already:

sudo apt install python3.11-dev

run the configuration script

bash configure.sh

Build and install ISSM:

cd $ISSM_DIR
make
make install

The following error occurs after running make

...
Making all in python
make[4]: Entering directory '/home/ana/opt/ISSM/src/wrappers/python'
  CXX      io/libISSMPython_la-CheckNumPythonArguments.lo
  CXX      io/libISSMPython_la-FetchPythonData.lo
  CXX      io/libISSMPython_la-WritePythonData.lo
  CXXLD    libISSMPython.la
/usr/bin/ld: ./io/.libs/libISSMPython_la-FetchPythonData.o: in function `FetchChacoData(int*, int**, int**, float**, _object*, _object*)':
/home/ana/opt/ISSM/src/wrappers/python/./io/FetchPythonData.cpp:1222:(.text+0x16b): undefined reference to `PythonIOSymbol'
/usr/bin/ld: ./io/.libs/libISSMPython_la-FetchPythonData.o: relocation R_X86_64_PC32 against undefined hidden symbol `PythonIOSymbol' can not be used when making a shared object
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:1183: libISSMPython.la] Error 1
make[4]: Leaving directory '/home/ana/opt/ISSM/src/wrappers/python'
make[3]: *** [Makefile:466: all-recursive] Error 1
make[3]: Leaving directory '/home/ana/opt/ISSM/src/wrappers'
make[2]: *** [Makefile:464: all-recursive] Error 1
make[2]: Leaving directory '/home/ana/opt/ISSM/src'
make[1]: *** [Makefile:519: all-recursive] Error 1
make[1]: Leaving directory '/home/ana/opt/ISSM'
make: *** [Makefile:451: all] Error 2

justinquinn

Hi, It looks like it may be because Chaco is not installed. If so, a quick fix would be to install it as an external package using the install-linux.sh, then recompile. I'll work on a fix for this module so that the function is skipped over if Chaco is not available. Please let us know if the above works.

anamikaene

Hi, thanks for your prompt reply. I've done as you suggested. When installing external packages I also run:

cd $ISSM_DIR/externalpackages/chaco
./install-linux.sh
source $ISSM_DIR/etc/environment.sh

I've also added the relevant line into the configure script, so it now looks like this:

./configure \
--prefix="${ISSM_DIR}" \
--with-python-version=3.11 \
--with-python-dir="/usr" \
--with-python-numpy-dir="${HOME}/Desktop/code/.venv/lib/python3.11/site-packages/numpy" \
--with-fortran-lib="-L/usr/lib/gcc/x86_64-linux-gnu/13 -lgfortran" \
--with-mpi-include="${ISSM_DIR}/externalpackages/petsc/install/include" \
--with-mpi-libflags="-L${ISSM_DIR}/externalpackages/petsc/install/lib -lmpi -lmpicxx -lmpifort" \
--with-metis-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-parmetis-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-blas-lapack-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-scalapack-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-mumps-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-petsc-dir="${ISSM_DIR}/externalpackages/petsc/install" \
--with-chaco-dir="${ISSM_DIR}/externalpackages/chaco/install" \
--with-triangle-dir="${ISSM_DIR}/externalpackages/triangle/install" \
--with-m1qn3-dir="${ISSM_DIR}/externalpackages/m1qn3/install"

This is the latest error:

...
make[4]: Entering directory '/home/ana/opt/ISSM/src/wrappers/python'
  CXX      io/libISSMPython_la-CheckNumPythonArguments.lo
  CXX      io/libISSMPython_la-FetchPythonData.lo
  CXX      io/libISSMPython_la-WritePythonData.lo
  CXX      io/libISSMApi_la-ApiPrintf.lo
  CXX      ../BamgConvertMesh/BamgConvertMesh_python_la-BamgConvertMesh.lo
  CXX      ../BamgMesher/BamgMesher_python_la-BamgMesher.lo
  CXX      ../BamgTriangulate/BamgTriangulate_python_la-BamgTriangulate.lo
  CXX      ../ContourToMesh/ContourToMesh_python_la-ContourToMesh.lo
  CXX      ../ContourToNodes/ContourToNodes_python_la-ContourToNodes.lo
  CXX      ../ElementConnectivity/ElementConnectivity_python_la-ElementConnectivity.lo
  CXX      ../ExpToLevelSet/ExpToLevelSet_python_la-ExpToLevelSet.lo
  CXX      ../InterpFromGridToMesh/InterpFromGridToMesh_python_la-InterpFromGridToMesh.lo
  CXX      ../InterpFromMesh2d/InterpFromMesh2d_python_la-InterpFromMesh2d.lo
  CXX      ../InterpFromMeshToGrid/InterpFromMeshToGrid_python_la-InterpFromMeshToGrid.lo
  CXX      ../InterpFromMeshToMesh2d/InterpFromMeshToMesh2d_python_la-InterpFromMeshToMesh2d.lo
  CXX      ../InterpFromMeshToMesh3d/InterpFromMeshToMesh3d_python_la-InterpFromMeshToMesh3d.lo
  CXX      ../IssmConfig/IssmConfig_python_la-IssmConfig.lo
  CXX      ../MeshPartition/MeshPartition_python_la-MeshPartition.lo
  CXX      ../MeshProfileIntersection/MeshProfileIntersection_python_la-MeshProfileIntersection.lo
  CXX      ../NodeConnectivity/NodeConnectivity_python_la-NodeConnectivity.lo
  CXX      ../Triangle/Triangle_python_la-Triangle.lo
  CXX      ../ProcessRifts/ProcessRifts_python_la-ProcessRifts.lo
  CXX      ../Chaco/Chaco_python_la-Chaco.lo
  CXXLD    libISSMPython.la
  CXXLD    libISSMApi.la
/usr/bin/ld: ./io/.libs/libISSMPython_la-FetchPythonData.o: in function `FetchChacoData(int*, int**, int**, float**, _object*, _object*)':
/home/ana/opt/ISSM/src/wrappers/python/./io/FetchPythonData.cpp:1222:(.text+0x16b): undefined reference to `PythonIOSymbol'
/usr/bin/ld: ./io/.libs/libISSMPython_la-FetchPythonData.o: relocation R_X86_64_PC32 against undefined hidden symbol `PythonIOSymbol' can not be used when making a shared object
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:1183: libISSMPython.la] Error 1
make[4]: *** Waiting for unfinished jobs....
ar: `u' modifier ignored since `D' is the default (see `U')
make[4]: Leaving directory '/home/ana/opt/ISSM/src/wrappers/python'
make[3]: *** [Makefile:466: all-recursive] Error 1
make[3]: Leaving directory '/home/ana/opt/ISSM/src/wrappers'
make[2]: *** [Makefile:464: all-recursive] Error 1
make[2]: Leaving directory '/home/ana/opt/ISSM/src'
make[1]: *** [Makefile:519: all-recursive] Error 1
make[1]: Leaving directory '/home/ana/opt/ISSM'
make: *** [Makefile:451: all] Error 2

Once again any help is appreciated .
A

justinquinn

Hi guys, just getting back to this now.

mathieumorlighem We definitely need the define to properly generate the Python modules. This works as-is on the rest of our builds.

anamikaene Looks like this is actually related to NumPy version and could have been an issue all along, but was only revealed because of the particular compiler + flags you have. Also possible that it's an issue with the deadsnakes implementation. I'm going to spin up an Ubuntu 24.04 VM to try to recreate it. In the meantime, you might try the standard APT distribution of Python and create a new virtual environment. Here's the method I used on our build node and my own machines,

sudo apt-get install python3-dev python3-minimal python3-pip python3-venv
mkdir ${HOME}/.venv
python3 -m venv ${HOME}/.venv/ISSM
source ${HOME}/.venv/ISSM/bin/activate
pip install matplotlib netcdf4 nose numpy pyshp scipy

Another potential short term fix is to comment out line 51 in subl src/wrappers/python/io/pythonio.h and lines 1218:1256 in src/wrappers/python/io/FetchPythonData.cpp then cd $ISSM_DIR; autoreconf -ivf; make; make install.

justinquinn

anamikaene Hi, just an update, I was able to reproduce the error on Ubuntu 24.04 with the default Python from APT and supplemental packages (e.g. NumPy) via pip's default repository. So, it is likely an issue with either (1) some breaking update to NumPy, or, (2) specific to the OS (perhaps a system library or default compiler flags that are tripping things up). I will try to update a Debian node and see if I can reproduce the issue there,

justinquinn

anamikaene Fix for this issue, affecting NumPy 2.1.0 and later, has been committed to the repo.

mathieumorlighem

it's not Chaco, you don't need it in your installation. The problem is

undefined reference to `PythonIOSymbol'

It seems like this variable is defined in the code (e.g., src/wrapper/python/io/FetchPythonData.cpp:11 and many other places)

#define PY_ARRAY_UNIQUE_SYMBOL PythonIOSymbol
#define NO_IMPORT

@justinquinn, do you know why we put these define there? It seems like it has been there for a very long time. Maybe we don't need them anymore?
Mathieu

mathieumorlighem

I know that Numpy recently released a new version that is very different from the previous one... could definitely be a problem. Can you both run:

>>> import numpy
>>> numpy.version.version

justinquinn

mathieumorlighem Interesting. Definitely could be it. But they said they're using the deadsnakes PPA and version 3.11 which is not cutting edge. I wonder if the NumPy version it's pulling is mismatched?

anamikaene

Thanks for the suggestions @mathieumorlighem and @justinquinn. I checked my numpy version as requested:

Python 3.11.10 (main, Sep  7 2024, 18:35:41) [GCC 13.2.0] on linux
>>> import numpy
>>> numpy.version.version
'2.1.2'

Given the potential compatibility issues with NumPy 2.x, I rebuilt using NumPy 1.26.4, which is the latest stable version in the 1.x series:

pip install numpy==1.26.4 scipy

Things appear to work now, thanks for the help!

justinquinn

anamikaene Thanks for getting back to us and for the initial report. I'll get a fix in for this, and glad to hear in the meantime that it's working for you. Let us know if you run into any other issues.

anamikaene

That's great to hear, once again thank you for all the help!