Hi there,

I saw in the changelog that ISSM version 4.16 has added support for Python 3, but am getting an error when compiling from the SVN source (revision 24493) that appears to be Python 2 related. I've got a Github gist to reproduce this.

configure: ============================================================================
configure: =                      Checking ISSM specific options                      =
configure: ============================================================================
checking for date... /bin/date
checking for build date... Fri Mar 20 01:40:46 UTC 2020
checking user name... jovyan
checking host full OS name and version... linux
checking host cpu... x86_64
checking vendor... pc
checking host OS name... linux
checking host OS version... 4.15.0-92-generic
checking host OS architecture... x86_64
checking for debugging support... yes
checking for development support... yes
checking for standalone modules build... no
checking for standalone executables build... no
checking for standalone libraries build... no
checking for wrappers compilation... yes
checking for vendor compilers... done
checking whether matlab is enabled... no
checking Checking if this is a Win build... ... done
checking for javascript... no
checking for triangle... yes
checking for boost... no
checking for dakota... no
checking for python... yes
checking for python version...   File "<string>", line 1
    import sys; print sys.version[:3]
                        ^
SyntaxError: invalid syntax

checking for python header file Python.h... configure: error: Python.h not found, locate this file and contact ISSM developers

The above error happens when I set the configure script's --with-python-dir= setting to a Python 3 path, and it seems to be hardcoded to point to Python 2 header files. If I set the path to point to Python 2 on /usr, the configure script works, but a different error pops up during the make stage. Is it possible to get a fully Python 3 installation working?

    Hei weiji

    This is related to the way in which ISSM deals with its install and looks for the Python.h path. Ifyou want to point to your proper Python3 install you should add the following line to your configure.sh:
    --with-python-version=3.X \
    X being the minor of your Python install. Be sure also tat the Python 3 you installed is a dev version and includes the "Python.h" file

    Excellent! The configure script worked after installing python3-dev (on an Ubuntu build) and using the two lines below:

    --with-python-version=3.6 \
--with-python-dir="/usr" \

    I'm now getting another error in the make stage though (the same one I saw when using Python 2). The error message is as follows:

    Making all in wrappers
make[3]: Entering directory '/home/jovyan/trunk/src/wrappers'
Making all in python
make[4]: Entering directory '/home/jovyan/trunk/src/wrappers/python'
  CXX      io/libISSMPython_la-WritePythonData.lo
  CXX      io/libISSMPython_la-CheckNumPythonArguments.lo
  CXX      io/libISSMPython_la-FetchPythonData.lo
  CXX      io/libISSMApi_la-ApiPrintf.lo
  CXX      ../BamgConvertMesh/BamgConvertMesh_python_la-BamgConvertMesh.lo
  CXX      ../BamgTriangulate/BamgTriangulate_python_la-BamgTriangulate.lo
  CXX      ../BamgMesher/BamgMesher_python_la-BamgMesher.lo
  CXX      ../ContourToMesh/ContourToMesh_python_la-ContourToMesh.lo
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/FetchPythonData.cpp:14:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1398: recipe for target 'io/libISSMPython_la-FetchPythonData.lo' failed
make[4]: *** [io/libISSMPython_la-FetchPythonData.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../BamgTriangulate/./.././python/include/pythonincludes.h:20,
                 from ../BamgTriangulate/./../bindings.h:11,
                 from ../BamgTriangulate/./BamgTriangulate.h:19,
                 from ../BamgTriangulate/BamgTriangulate.cpp:4:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/CheckNumPythonArguments.cpp:8:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../BamgConvertMesh/./.././python/include/pythonincludes.h:20,
                 from ../BamgConvertMesh/./../bindings.h:11,
                 from ../BamgConvertMesh/./BamgConvertMesh.h:19,
                 from ../BamgConvertMesh/BamgConvertMesh.cpp:4:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/ApiPrintf.cpp:11:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1265: recipe for target '../BamgTriangulate/BamgTriangulate_python_la-BamgTriangulate.lo' failed
make[4]: *** [../BamgTriangulate/BamgTriangulate_python_la-BamgTriangulate.lo] Error 1
Makefile:1391: recipe for target 'io/libISSMPython_la-CheckNumPythonArguments.lo' failed
make[4]: *** [io/libISSMPython_la-CheckNumPythonArguments.lo] Error 1
Makefile:1377: recipe for target 'io/libISSMApi_la-ApiPrintf.lo' failed
make[4]: *** [io/libISSMApi_la-ApiPrintf.lo] Error 1
Makefile:1251: recipe for target '../BamgConvertMesh/BamgConvertMesh_python_la-BamgConvertMesh.lo' failed
make[4]: *** [../BamgConvertMesh/BamgConvertMesh_python_la-BamgConvertMesh.lo] Error 1
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../ContourToMesh/./.././python/include/pythonincludes.h:20,
                 from ../ContourToMesh/./../bindings.h:11,
                 from ../ContourToMesh/./ContourToMesh.h:20,
                 from ../ContourToMesh/ContourToMesh.cpp:6:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/WritePythonData.cpp:14:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1279: recipe for target '../ContourToMesh/ContourToMesh_python_la-ContourToMesh.lo' failed
make[4]: *** [../ContourToMesh/ContourToMesh_python_la-ContourToMesh.lo] Error 1
Makefile:1384: recipe for target 'io/libISSMPython_la-WritePythonData.lo' failed
make[4]: *** [io/libISSMPython_la-WritePythonData.lo] Error 1
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../BamgMesher/./.././python/include/pythonincludes.h:20,
                 from ../BamgMesher/./../bindings.h:11,
                 from ../BamgMesher/./BamgMesher.h:22,
                 from ../BamgMesher/BamgMesher.cpp:4:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1258: recipe for target '../BamgMesher/BamgMesher_python_la-BamgMesher.lo' failed
make[4]: *** [../BamgMesher/BamgMesher_python_la-BamgMesher.lo] Error 1
make[4]: Leaving directory '/home/jovyan/trunk/src/wrappers/python'
Makefile:465: recipe for target 'all-recursive' failed
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory '/home/jovyan/trunk/src/wrappers'
Makefile:463: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/home/jovyan/trunk/src'
Makefile:584: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jovyan/trunk'
Makefile:481: recipe for target 'all' failed
make: *** [all] Error 2

    This forum post suggests that installing libpython3-dev will provide the missing pyconfig.h file, but I'm pretty sure that installing python3-dev automatically installs libpython3-dev. Is there an extra setting I need to make to fix this?

    Hei

    I am not sure about this one. Have you checked that pyconfig.h is actually here. It seems that it is the case. Could you post your configure.sh file and let me know what is your installation routine (I guess the one of the website).

    Hi,

    Been trying several combinations today but no luck so far. The pyconfig.h file is definitely there in /usr/include so I'm not sure what's wrong. The compilation command I'm using is a bit different to the one on the website and goes like this:

    cd $ISSM_DIR && \
    ./configure \
    --prefix="$ISSM_DIR" \
    --disable-static \
    --with-numthreads=8 \
    --with-python-version=3.6 \
    --with-python-dir="/usr" \
    --with-python-numpy-dir="$CONDA_DIR/lib/python3.7/site-packages/numpy" \
    --with-mpi-include="$CONDA_DIR/lib/include" \
    --with-mpi-libflags="-L$CONDA_DIR/lib -lmpi -lmpicxx -lmpifort" \
    --with-metis-dir="$CONDA_DIR/lib" \
    --with-scalapack-dir="$CONDA_DIR/lib" \
    --with-mumps-dir="$CONDA_DIR/lib" \
    --with-petsc-dir="$CONDA_DIR" \
    --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \
    --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install"

cd $ISSM_DIR && make --jobs=8

    and I've posted two files up as a Github gist - configure and Makefile, plus a Dockerfile with the above code. Let me know if you need any other log files and I'll upload them for you.

    So looking at ti and comparing with my install I see two possible issues. The first one is the less likely to do anything but I see that your numpy points to a 3.7 Python, that might be an issue.
    For the problem at hand, could you check that you also have a pyconfig.h in
    /usr/include/x86_64-linux-gnu/python3.6m/, it seems that the user/include pyconfig.h tries to grab this one.

    Yes, pyconfig.h is present:

    $ ls -l /usr/include/x86_64-linux-gnu/python3.6m/
total 44
-rw-r--r-- 1 root root 44045 Nov  7 10:44 pyconfig.h

    I'll see if numpy is the issue, give me a few minutes. Edit: Nope, using system apt installed numpy (Python 3.6) still shows up with same error.

    Hei
    Just had someone else running into issues with numpy. And it appears that their numpy directory set in configure was too shallow. The numpy directory should look like that python3.6/site-packages/numpy/core/include/numpy/ could you check if that solves your issue.

    OK. I am running out of idea. The more I look at it the more it seems that it is not a ISSM issue but more of a Python issue. Have you tried to purge and reinstall your python to see if that fixes it?

    It's a clean install (on docker actually). I've even looked at the jenkins build script and as much of the source code as I can dig into, but exhausted most options already 😅

    Can you post your config.log I can look if there is something fishy in there. Just to confirm you are on the trunk version of ISSM not on the development branch?

    It's a bit big so here's a link to the config.log file. And yes, I should be on the trunk version.

    Hi guys, sorry that I am just now getting the opportunity to take a look at this.

    On Debian, using Python 3.7, everything compiles just fine. There is a runtime crash with "ImportError: dynamic module does not define init function", which may be due to the fact that /usr/bin/python -> /usr/bin/python2 on this particular machine and --with-python-version is not actually being honored. That being said, I am currently experimenting to see if which python is the issue here.

    Of note, I am working out of the development branch, so I will start diffing config files to see if there is a disconnect on our end.

    In the meantime, @weiji14, could you try installing 3.7 (and the corresponding numpy package) with apt and see if you can get ISSM to at least compile? This would at least narrow down the issue(s) we are experiencing.

    After linking /usr/bin/python to /usr/bin/python3, everything compiles and runs as expected. So, I think we can assume that, after the next merge from trunk-jpl/ to trunk/, the issue is not internal. That said, I will work on and commit the following fixes:

    1. Compilation should honor Python version supplied to option --with-python-version.
    2. Polling Python.h with python3 should not error out on the syntax error noted above.

    Please let me know if I have missed anything and I will add it to this list. As well, @weiji14 let me know what if any progress you are able to make.

    Okay, I have fixed (1) and committed to trunk-jpl/. I realized after the fact that (2) was a non-issue: what was happening was that I was compiling with flags for Python 3, but my test deck was running python, which was sym linked to python2. In all, I would advise running a copy of the development branch until we can push to the production branch (trunk/), which will happen soon (I am wrapping another, larger fix). We'll post here to the forum when that has been done.

    In the meantime, @weiji14, could you try installing 3.7 (and the corresponding numpy package) with apt and see if you can get ISSM to at least compile? This would at least narrow down the issue(s) we are experiencing.

    Using the apt provided numpy doesn't work on Ubuntu (and also on Debian). I've made sure to compile on a virtual environment using Python 3 (tested versions 3.6, 3.7, 3.8) so that Python 2 is not used but no luck... I think you're right that there's some symlink of python -> python2 that is messing things up.

    I would advise running a copy of the development branch until we can push to the production branch (trunk/)

    Don't think that I can access the dev branch (assuming that it's https://issm.ess.uci.edu/svn/issm/issm/trunk-jpl?). So I'll just wait things to get into the main trunk 😃

      weiji14 You may have already done the following, but just in case, this is the methodology I used to install Python 2 and 3 with all required packages under Debian:

      # Install development versions
sudo apt-get install python-dev python3-dev 

# Install minimal versions
sudo apt-get install python-minimal python3-minimal 

# Install pip
sudo apt-get install python-pip python3-pip

# Install required packages
sudo su
python -m pip install numpy scipy matplotlib nose
python3 -m pip install numpy scipy matplotlib nose

      And, yes: if you compile for Python 3 via --with-python-version, you will have to manually run python3* or create a new sym link if running python executes version 2.*.

        Thanks for listing down all the dependencies @justinquinn. I was only missing the python3-minimal dependency, but installing it still resulted in the same pyconfig.h: No such file or directory error. I've also symlinked python to python3 and compilation still fails. Might be time to try out the development branch, could you provide me with instructions to clone trunk-jpl if possible?

        5 days later

        Ok, I've updated to using revision 24686 (02 Apr 2020) on trunk and think I've resolved the problem after some tweaking. I've set both --with-python-dir and --with-python-numpy-dir to use $CONDA_DIR (instead of /usr) and that resolves the pyconfig.h not found error. The key was probably to be consistent (i.e. don't mix stuff between virtual environments).

        There was then another error about ld not able to find python3.8m which I resolved by patching the configure file to use python3.8 instead using the below command:

        sed -i 's/-lpython${PYTHON_VERSION}m/-lpython${PYTHON_VERSION}/g' $ISSM_DIR/configure

        Specifically, that patches the below code block in configure:

        if test "x${PYTHON_MAJOR}" == "x3"; then
	PYTHONLIB="-L${PYTHON_ROOT}/lib -lpython${PYTHON_VERSION}m"
else
	PYTHONLIB="-L${PYTHON_ROOT}/lib -lpython${PYTHON_VERSION}"

        Just wondering why the 'm' version is used for python 3.X and not python 2.x?