Error compiling with Python 3

weiji14

Hi there,

I saw in the changelog that ISSM version 4.16 has added support for Python 3, but am getting an error when compiling from the SVN source (revision 24493) that appears to be Python 2 related. I've got a Github gist to reproduce this.

configure: ============================================================================
configure: =                      Checking ISSM specific options                      =
configure: ============================================================================
checking for date... /bin/date
checking for build date... Fri Mar 20 01:40:46 UTC 2020
checking user name... jovyan
checking host full OS name and version... linux
checking host cpu... x86_64
checking vendor... pc
checking host OS name... linux
checking host OS version... 4.15.0-92-generic
checking host OS architecture... x86_64
checking for debugging support... yes
checking for development support... yes
checking for standalone modules build... no
checking for standalone executables build... no
checking for standalone libraries build... no
checking for wrappers compilation... yes
checking for vendor compilers... done
checking whether matlab is enabled... no
checking Checking if this is a Win build... ... done
checking for javascript... no
checking for triangle... yes
checking for boost... no
checking for dakota... no
checking for python... yes
checking for python version...   File "<string>", line 1
    import sys; print sys.version[:3]
                        ^
SyntaxError: invalid syntax

checking for python header file Python.h... configure: error: Python.h not found, locate this file and contact ISSM developers

The above error happens when I set the configure script's --with-python-dir= setting to a Python 3 path, and it seems to be hardcoded to point to Python 2 header files. If I set the path to point to Python 2 on /usr, the configure script works, but a different error pops up during the make stage. Is it possible to get a fully Python 3 installation working?

berkninan

weiji14 checking for python header file Python.h... configure: error: Python.h not found, locate this file and contact ISSM developers

Most of the time these are dependency-issues. Python.h is used by GNU Compiler Collection (gcc) to build applications. You need to install a package called python-dev for building Python modules, extending the Python interpreter or embedding Python in applications. You encounter "Python.h: No such file or directory" error while trying to build a shared library using the file extension of another language ( e.g. 'C' ). If you are trying to build a shared library using the file extension of another language, you need to install the correct development version of Python.

Reason for this error:

You haven't properly installed the header files and static libraries for python dev.
Also, sometimes include files might not be default in the include path.

How to solve this error:

Install the missing files and libraries.
Include Path and Library.
Finally, Compile it.

bdefleurian

Hei weiji

This is related to the way in which ISSM deals with its install and looks for the Python.h path. Ifyou want to point to your proper Python3 install you should add the following line to your configure.sh:
--with-python-version=3.X \
X being the minor of your Python install. Be sure also tat the Python 3 you installed is a dev version and includes the "Python.h" file

weiji14

Excellent! The configure script worked after installing python3-dev (on an Ubuntu build) and using the two lines below:

--with-python-version=3.6 \
--with-python-dir="/usr" \

I'm now getting another error in the make stage though (the same one I saw when using Python 2). The error message is as follows:

Making all in wrappers
make[3]: Entering directory '/home/jovyan/trunk/src/wrappers'
Making all in python
make[4]: Entering directory '/home/jovyan/trunk/src/wrappers/python'
  CXX      io/libISSMPython_la-WritePythonData.lo
  CXX      io/libISSMPython_la-CheckNumPythonArguments.lo
  CXX      io/libISSMPython_la-FetchPythonData.lo
  CXX      io/libISSMApi_la-ApiPrintf.lo
  CXX      ../BamgConvertMesh/BamgConvertMesh_python_la-BamgConvertMesh.lo
  CXX      ../BamgTriangulate/BamgTriangulate_python_la-BamgTriangulate.lo
  CXX      ../BamgMesher/BamgMesher_python_la-BamgMesher.lo
  CXX      ../ContourToMesh/ContourToMesh_python_la-ContourToMesh.lo
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/FetchPythonData.cpp:14:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1398: recipe for target 'io/libISSMPython_la-FetchPythonData.lo' failed
make[4]: *** [io/libISSMPython_la-FetchPythonData.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../BamgTriangulate/./.././python/include/pythonincludes.h:20,
                 from ../BamgTriangulate/./../bindings.h:11,
                 from ../BamgTriangulate/./BamgTriangulate.h:19,
                 from ../BamgTriangulate/BamgTriangulate.cpp:4:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/CheckNumPythonArguments.cpp:8:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../BamgConvertMesh/./.././python/include/pythonincludes.h:20,
                 from ../BamgConvertMesh/./../bindings.h:11,
                 from ../BamgConvertMesh/./BamgConvertMesh.h:19,
                 from ../BamgConvertMesh/BamgConvertMesh.cpp:4:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/ApiPrintf.cpp:11:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1265: recipe for target '../BamgTriangulate/BamgTriangulate_python_la-BamgTriangulate.lo' failed
make[4]: *** [../BamgTriangulate/BamgTriangulate_python_la-BamgTriangulate.lo] Error 1
Makefile:1391: recipe for target 'io/libISSMPython_la-CheckNumPythonArguments.lo' failed
make[4]: *** [io/libISSMPython_la-CheckNumPythonArguments.lo] Error 1
Makefile:1377: recipe for target 'io/libISSMApi_la-ApiPrintf.lo' failed
make[4]: *** [io/libISSMApi_la-ApiPrintf.lo] Error 1
Makefile:1251: recipe for target '../BamgConvertMesh/BamgConvertMesh_python_la-BamgConvertMesh.lo' failed
make[4]: *** [../BamgConvertMesh/BamgConvertMesh_python_la-BamgConvertMesh.lo] Error 1
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../ContourToMesh/./.././python/include/pythonincludes.h:20,
                 from ../ContourToMesh/./../bindings.h:11,
                 from ../ContourToMesh/./ContourToMesh.h:20,
                 from ../ContourToMesh/ContourToMesh.cpp:6:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ./io/./../include/pythonincludes.h:20,
                 from ./io/./pythonio.h:14,
                 from ./io/WritePythonData.cpp:14:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1279: recipe for target '../ContourToMesh/ContourToMesh_python_la-ContourToMesh.lo' failed
make[4]: *** [../ContourToMesh/ContourToMesh_python_la-ContourToMesh.lo] Error 1
Makefile:1384: recipe for target 'io/libISSMPython_la-WritePythonData.lo' failed
make[4]: *** [io/libISSMPython_la-WritePythonData.lo] Error 1
In file included from /usr/include/python3.6/Python.h:8:0,
                 from ../BamgMesher/./.././python/include/pythonincludes.h:20,
                 from ../BamgMesher/./../bindings.h:11,
                 from ../BamgMesher/./BamgMesher.h:22,
                 from ../BamgMesher/BamgMesher.cpp:4:
/usr/include/python3.6/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.6m/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.6m/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:1258: recipe for target '../BamgMesher/BamgMesher_python_la-BamgMesher.lo' failed
make[4]: *** [../BamgMesher/BamgMesher_python_la-BamgMesher.lo] Error 1
make[4]: Leaving directory '/home/jovyan/trunk/src/wrappers/python'
Makefile:465: recipe for target 'all-recursive' failed
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory '/home/jovyan/trunk/src/wrappers'
Makefile:463: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/home/jovyan/trunk/src'
Makefile:584: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory '/home/jovyan/trunk'
Makefile:481: recipe for target 'all' failed
make: *** [all] Error 2

This forum post suggests that installing libpython3-dev will provide the missing pyconfig.h file, but I'm pretty sure that installing python3-dev automatically installs libpython3-dev. Is there an extra setting I need to make to fix this?

bdefleurian

Hei

I am not sure about this one. Have you checked that pyconfig.h is actually here. It seems that it is the case. Could you post your configure.sh file and let me know what is your installation routine (I guess the one of the website).

weiji14

Hi,

Been trying several combinations today but no luck so far. The pyconfig.h file is definitely there in /usr/include so I'm not sure what's wrong. The compilation command I'm using is a bit different to the one on the website and goes like this:

cd $ISSM_DIR && \
    ./configure \
    --prefix="$ISSM_DIR" \
    --disable-static \
    --with-numthreads=8 \
    --with-python-version=3.6 \
    --with-python-dir="/usr" \
    --with-python-numpy-dir="$CONDA_DIR/lib/python3.7/site-packages/numpy" \
    --with-mpi-include="$CONDA_DIR/lib/include" \
    --with-mpi-libflags="-L$CONDA_DIR/lib -lmpi -lmpicxx -lmpifort" \
    --with-metis-dir="$CONDA_DIR/lib" \
    --with-scalapack-dir="$CONDA_DIR/lib" \
    --with-mumps-dir="$CONDA_DIR/lib" \
    --with-petsc-dir="$CONDA_DIR" \
    --with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \
    --with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install"

cd $ISSM_DIR && make --jobs=8

and I've posted two files up as a Github gist - configure and Makefile, plus a Dockerfile with the above code. Let me know if you need any other log files and I'll upload them for you.

bdefleurian

So looking at ti and comparing with my install I see two possible issues. The first one is the less likely to do anything but I see that your numpy points to a 3.7 Python, that might be an issue.
For the problem at hand, could you check that you also have a pyconfig.h in
/usr/include/x86_64-linux-gnu/python3.6m/, it seems that the user/include pyconfig.h tries to grab this one.

weiji14

Yes, pyconfig.h is present:

$ ls -l /usr/include/x86_64-linux-gnu/python3.6m/
total 44
-rw-r--r-- 1 root root 44045 Nov  7 10:44 pyconfig.h

~~I'll see if numpy is the issue, give me a few minutes.~~ Edit: Nope, using system apt installed numpy (Python 3.6) still shows up with same error.

bdefleurian

Hei
Just had someone else running into issues with numpy. And it appears that their numpy directory set in configure was too shallow. The numpy directory should look like that python3.6/site-packages/numpy/core/include/numpy/ could you check if that solves your issue.

weiji14

Nope, still the same error :/

bdefleurian

OK. I am running out of idea. The more I look at it the more it seems that it is not a ISSM issue but more of a Python issue. Have you tried to purge and reinstall your python to see if that fixes it?

weiji14

It's a clean install (on docker actually). I've even looked at the jenkins build script and as much of the source code as I can dig into, but exhausted most options already 😅

bdefleurian

Can you post your config.log I can look if there is something fishy in there. Just to confirm you are on the trunk version of ISSM not on the development branch?

weiji14

It's a bit big so here's a link to the config.log file. And yes, I should be on the trunk version.

justinquinn

Hi guys, sorry that I am just now getting the opportunity to take a look at this.

On Debian, using Python 3.7, everything compiles just fine. There is a runtime crash with "ImportError: dynamic module does not define init function", which may be due to the fact that /usr/bin/python -> /usr/bin/python2 on this particular machine and --with-python-version is not actually being honored. That being said, I am currently experimenting to see if which python is the issue here.

Of note, I am working out of the development branch, so I will start diffing config files to see if there is a disconnect on our end.

In the meantime, @weiji14, could you try installing 3.7 (and the corresponding numpy package) with apt and see if you can get ISSM to at least compile? This would at least narrow down the issue(s) we are experiencing.

justinquinn

After linking /usr/bin/python to /usr/bin/python3, everything compiles and runs as expected. So, I think we can assume that, after the next merge from trunk-jpl/ to trunk/, the issue is not internal. That said, I will work on and commit the following fixes:

Compilation should honor Python version supplied to option --with-python-version.
Polling Python.h with python3 should not error out on the syntax error noted above.

Please let me know if I have missed anything and I will add it to this list. As well, @weiji14 let me know what if any progress you are able to make.

justinquinn

Okay, I have fixed (1) and committed to trunk-jpl/. I realized after the fact that (2) was a non-issue: what was happening was that I was compiling with flags for Python 3, but my test deck was running python, which was sym linked to python2. In all, I would advise running a copy of the development branch until we can push to the production branch (trunk/), which will happen soon (I am wrapping another, larger fix). We'll post here to the forum when that has been done.

weiji14

In the meantime, @weiji14, could you try installing 3.7 (and the corresponding numpy package) with apt and see if you can get ISSM to at least compile? This would at least narrow down the issue(s) we are experiencing.

Using the apt provided numpy doesn't work on Ubuntu (and also on Debian). I've made sure to compile on a virtual environment using Python 3 (tested versions 3.6, 3.7, 3.8) so that Python 2 is not used but no luck... I think you're right that there's some symlink of python -> python2 that is messing things up.

I would advise running a copy of the development branch until we can push to the production branch (trunk/)

Don't think that I can access the dev branch (assuming that it's https://issm.ess.uci.edu/svn/issm/issm/trunk-jpl?). So I'll just wait things to get into the main trunk 😃

justinquinn

weiji14 You may have already done the following, but just in case, this is the methodology I used to install Python 2 and 3 with all required packages under Debian:

# Install development versions
sudo apt-get install python-dev python3-dev 

# Install minimal versions
sudo apt-get install python-minimal python3-minimal 

# Install pip
sudo apt-get install python-pip python3-pip

# Install required packages
sudo su
python -m pip install numpy scipy matplotlib nose
python3 -m pip install numpy scipy matplotlib nose

And, yes: if you compile for Python 3 via --with-python-version, you will have to manually run python3* or create a new sym link if running python executes version 2.*.

weiji14

Thanks for listing down all the dependencies @justinquinn. I was only missing the python3-minimal dependency, but installing it still resulted in the same pyconfig.h: No such file or directory error. I've also symlinked python to python3 and compilation still fails. Might be time to try out the development branch, could you provide me with instructions to clone trunk-jpl if possible?

weiji14

Ok, I've updated to using revision 24686 (02 Apr 2020) on trunk and think I've resolved the problem after some tweaking. I've set both --with-python-dir and --with-python-numpy-dir to use $CONDA_DIR (instead of /usr) and that resolves the pyconfig.h not found error. The key was probably to be consistent (i.e. don't mix stuff between virtual environments).

There was then another error about ld not able to find python3.8m which I resolved by patching the configure file to use python3.8 instead using the below command:

sed -i 's/-lpython${PYTHON_VERSION}m/-lpython${PYTHON_VERSION}/g' $ISSM_DIR/configure

Specifically, that patches the below code block in configure:

if test "x${PYTHON_MAJOR}" == "x3"; then
	PYTHONLIB="-L${PYTHON_ROOT}/lib -lpython${PYTHON_VERSION}m"
else
	PYTHONLIB="-L${PYTHON_ROOT}/lib -lpython${PYTHON_VERSION}"

Just wondering why the 'm' version is used for python 3.X and not python 2.x?