Compiling ISSM on Ubuntu 16.04.5

Iiisola97 · Jun 28, 2019

justinquinn
ignacio@mangala:~$ openssl -v Invalid command '-v'; type "help" for a list.

ignacio@mangala:~$ dpkg -l openssl Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============================================================-====================================-====================================-================================================================================================================================== ii openssl 1.0.2g-1ubuntu4.15 amd64 Secure Sockets Layer toolkit - cryptographic utility

ignacio@mangala:~$ openssl version OpenSSL 1.1.1 11 Sep 2018

Jjustinquinn · Jun 28, 2019

iisola97 Oops: yep, openssl version. Okay, well it looks like, despite setting the version, you still have v1.1.1. What does which openssl give? cd to that directory, then ls -la openssl*. What do you get?

Iiisola97 · Jun 28, 2019

justinquinn
ignacio@mangala:~$ which openssl /usr/local/bin/openssl ignacio@mangala:~$ cd /usr/local/bin/ ignacio@mangala:/usr/local/bin$ ls -la openssl* -rwxr-xr-x 1 root root 756792 Jun 26 11:03 openssl

Jjustinquinn · Jun 28, 2019

iisola97 Interesting. So using the =<version> modifier did not work. I think maybe madison was the wrong repo as well. What do you get from apt-cache policy openssl? sudo apt-get remove openssl; sudo apt-get update; sudo apt-get autoremove then sudo apt-get install openssl=<version> with the version closest to 1.0.2 that you get from apt-cache policy openssl. This should work.

Iiisola97 · Jun 28, 2019

justinquinn
ignacio@mangala:~$ apt-cache policy openssl openssl: Installed: 1.0.2g-1ubuntu4.15 Candidate: 1.0.2g-1ubuntu4.15 Version table: *** 1.0.2g-1ubuntu4.15 500 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages 100 /var/lib/dpkg/status 1.0.2g-1ubuntu4 500 500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages ignacio@mangala:~$ sudo apt-get remove openssl; sudo apt-get update; sudo apt-get autoremove [sudo] password for ignacio: Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: ca-certificates openssl 0 upgraded, 0 newly installed, 2 to remove and 96 not upgraded. After this operation, 1,393 kB disk space will be freed. Do you want to continue? [Y/n] Y (Reading database ... 219196 files and directories currently installed.) Removing ca-certificates (20170717~16.04.2) ... Removing dangling symlinks from /etc/ssl/certs... done. Removing openssl (1.0.2g-1ubuntu4.15) ... Processing triggers for man-db (2.7.5-1) ... Hit:1 http://ppa.launchpad.net/maarten-baert/simplescreenrecorder/ubuntu xenial InRelease Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Hit:3 http://archive.canonical.com/ubuntu xenial InRelease Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB] Hit:5 https://repo.skype.com/deb stable InRelease Ign:6 http://dell.archive.canonical.com/updates xenial-dell-matira-5-7 InRelease Get:7 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB] Ign:8 http://dell.archive.canonical.com/updates xenial-dell-service InRelease Ign:9 http://dell.archive.canonical.com/updates xenial-dell InRelease Hit:10 http://dell.archive.canonical.com/updates xenial-dell-matira-5-7 Release Hit:11 http://dell.archive.canonical.com/updates xenial-dell-service Release Get:12 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB] Hit:13 http://dell.archive.canonical.com/updates xenial-dell Release Get:14 http://security.ubuntu.com/ubuntu xenial-security/main amd64 DEP-11 Metadata [73.8 kB] Get:16 http://security.ubuntu.com/ubuntu xenial-security/main DEP-11 64x64 Icons [73.2 kB] Get:17 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 DEP-11 Metadata [121 kB] Get:18 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 DEP-11 Metadata [325 kB] Get:19 http://security.ubuntu.com/ubuntu xenial-security/universe DEP-11 64x64 Icons [179 kB] Get:21 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 DEP-11 Metadata [2,464 B] Get:23 http://archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons [238 kB] Get:24 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 DEP-11 Metadata [271 kB] Get:25 http://archive.ubuntu.com/ubuntu xenial-updates/universe DEP-11 64x64 Icons [398 kB] Get:26 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 DEP-11 Metadata [5,976 B] Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse DEP-11 64x64 Icons [14.3 kB] Get:28 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 DEP-11 Metadata [3,328 B] Get:29 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 DEP-11 Metadata [5,104 B] Fetched 2,036 kB in 1s (1,249 kB/s) Reading package lists... Done Reading package lists... Done Building dependency tree Reading state information... Done 0 upgraded, 0 newly installed, 0 to remove and 96 not upgraded. ignacio@mangala:~$ sudo apt-get install openssl=1.0.2g-1ubuntu4.15 Reading package lists... Done Building dependency tree Reading state information... Done Suggested packages: ca-certificates The following NEW packages will be installed: openssl 0 upgraded, 1 newly installed, 0 to remove and 96 not upgraded. Need to get 0 B/492 kB of archives. After this operation, 956 kB of additional disk space will be used. Selecting previously unselected package openssl. (Reading database ... 218929 files and directories currently installed.) Preparing to unpack .../openssl_1.0.2g-1ubuntu4.15_amd64.deb ... Unpacking openssl (1.0.2g-1ubuntu4.15) ... Processing triggers for man-db (2.7.5-1) ... Setting up openssl (1.0.2g-1ubuntu4.15) ... ignacio@mangala:~$ sudo apt-get install ca-certificates Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: ca-certificates 0 upgraded, 1 newly installed, 0 to remove and 96 not upgraded. Need to get 0 B/167 kB of archives. After this operation, 436 kB of additional disk space will be used. Preconfiguring packages ... Selecting previously unselected package ca-certificates. (Reading database ... 219021 files and directories currently installed.) Preparing to unpack .../ca-certificates_20170717~16.04.2_all.deb ... Unpacking ca-certificates (20170717~16.04.2) ... Processing triggers for man-db (2.7.5-1) ... Setting up ca-certificates (20170717~16.04.2) ... Processing triggers for ca-certificates (20170717~16.04.2) ... Updating certificates in /etc/ssl/certs... 148 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done.
But still:
ignacio@mangala:~$ openssl version OpenSSL 1.1.1 11 Sep 2018 ignacio@mangala:~$ dpkg -l openssl Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-============================-===================-===================-============================================================= ii openssl 1.0.2g-1ubuntu4.15 amd64 Secure Sockets Layer toolkit - cryptographic utility
And cmake installation still gets stuck at same instance.

Jjustinquinn · Jun 28, 2019

iisola97 Okay, let's try a different approach: sudo apt-get remove openssl; sudo apt-get install libssl-dev, then see if cmake will compile. You may need to try reinstalling openssl afterwards if it fails with this configuration.

Iiisola97 · Jun 28, 2019

justinquinn Unfortunately, I didn't have luck: cmake doesn't compile. I've already reinstalled openssl.

Jjustinquinn · Jun 28, 2019

iisola97 Okay, let’s try one more thing: sudo apt-get remove openssl; sudo apt-get purge openssl; sudo apt-get install openssl=1.0.2. Then try compiling CMake from source again.

Jjustinquinn · Jun 28, 2019

iisola97 If that doesn't work, modify $ISSM_DIR/externalpackages/cmake/install.sh, setting VER=3.14.5, save, and try running again.

Iiisola97 · Jun 28, 2019

justinquinn CMake compilation crashed at the same instance as before.

Iiisola97 · Jun 28, 2019

justinquinn and for this one, this happened:
-- Checking _FILE_OFFSET_BITS for large files -- Checking _FILE_OFFSET_BITS for large files - not needed -- Checking support for ARCHIVE_CRYPTO_MD5_LIBC -- Checking support for ARCHIVE_CRYPTO_MD5_LIBC -- not found -- Checking support for ARCHIVE_CRYPTO_RMD160_LIBC -- Checking support for ARCHIVE_CRYPTO_RMD160_LIBC -- not found -- Checking support for ARCHIVE_CRYPTO_SHA1_LIBC -- Checking support for ARCHIVE_CRYPTO_SHA1_LIBC -- not found -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBC -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBC -- not found -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBC -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBC -- not found -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBC -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBC -- not found -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBC2 -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBC2 -- not found -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBC2 -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBC2 -- not found -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBC2 -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBC2 -- not found -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBC3 -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBC3 -- not found -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBC3 -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBC3 -- not found -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBC3 -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBC3 -- not found -- Checking support for ARCHIVE_CRYPTO_MD5_LIBSYSTEM -- Checking support for ARCHIVE_CRYPTO_MD5_LIBSYSTEM -- not found -- Checking support for ARCHIVE_CRYPTO_SHA1_LIBSYSTEM -- Checking support for ARCHIVE_CRYPTO_SHA1_LIBSYSTEM -- not found -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBSYSTEM -- Checking support for ARCHIVE_CRYPTO_SHA256_LIBSYSTEM -- not found -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBSYSTEM -- Checking support for ARCHIVE_CRYPTO_SHA384_LIBSYSTEM -- not found -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBSYSTEM -- Checking support for ARCHIVE_CRYPTO_SHA512_LIBSYSTEM -- not found -- Checking support for ARCHIVE_CRYPTO_MD5_OPENSSL -- Checking support for ARCHIVE_CRYPTO_MD5_OPENSSL -- found -- Checking support for ARCHIVE_CRYPTO_RMD160_OPENSSL -- Checking support for ARCHIVE_CRYPTO_RMD160_OPENSSL -- found -- Checking support for ARCHIVE_CRYPTO_SHA1_OPENSSL -- Checking support for ARCHIVE_CRYPTO_SHA1_OPENSSL -- found -- Checking support for ARCHIVE_CRYPTO_SHA256_OPENSSL -- Checking support for ARCHIVE_CRYPTO_SHA256_OPENSSL -- found -- Checking support for ARCHIVE_CRYPTO_SHA384_OPENSSL -- Checking support for ARCHIVE_CRYPTO_SHA384_OPENSSL -- found -- Checking support for ARCHIVE_CRYPTO_SHA512_OPENSSL -- Checking support for ARCHIVE_CRYPTO_SHA512_OPENSSL -- found -- Checking for curses support -- Checking for curses support - Failed -- Looking for elf.h -- Looking for elf.h - found -- Looking for a Fortran compiler -- Looking for a Fortran compiler - /usr/bin/f95 -- Performing Test QT4_WORKS -- Performing Test QT4_WORKS - Success -- Performing Test run_pic_test -- Performing Test run_pic_test - Success -- Performing Test run_inlines_hidden_test -- Performing Test run_inlines_hidden_test - Success -- Configuring incomplete, errors occurred! See also "/home/ignacio/research/mars/ISSM_source/trunk/externalpackages/cmake/install/CMakeFiles/CMakeOutput.log". See also "/home/ignacio/research/mars/ISSM_source/trunk/externalpackages/cmake/install/CMakeFiles/CMakeError.log".
--------------------------------------------- Error when bootstrapping CMake: Problem while running initial CMake ---------------------------------------------

Jjustinquinn · Jun 28, 2019

iisola97 One thing is you need to install gfortran, but I believe this is a separate issue: sudo apt-get install gfortran.

Can you somehow get me $ISSM_DIR/trunk/externalpackages/cmake/install/CMakeFiles/CMakeError.log? I don't know if you can attach files here. If not, you have my email address from that message with Nicole.

Iiisola97 · Jun 28, 2019

justinquinn I have gfortran installed already.
I can't attach files here. I've just sent you an email.

Jjustinquinn · Jun 28, 2019

iisola97 Okay, I'm going to have to look into this over the weekend. I'm sure you're eager to get ISSM compiled, so why don't we "cheat": sudo apt-get install cmake and remove the install script for CMake from your configuration script.

I can go back at a later time, create a new VM with Ubuntu 16.04, figure out why this is happening, and update our install instructions accordingly.

Iiisola97 · Jun 30, 2019

justinquinn I partitioned the hhd disk of my personal computer and installed Ubuntu 18.04.2 LTS, just to start a new compilation process from the very scratch. Doing everything step-by-step and installing some programs needed for the compilation process (svn, gcc, g++, gfortran and make; openssl 1.1.1 and ca-certificates were already installed), I was able to run autotools, cmake and mpich without any issues. Regarding petsc, FBLASLAPACK and SCALAPACK compiled perfectly, but it got stuck at installing METIS. This is the error I got:
=============================================================================== Trying to download https://bitbucket.org/petsc/pkg-metis/get/v5.1.0-p4.tar.gz for METIS =============================================================================== ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details):
------------------------------------------------------------------------------- Unable to download METIS Failed to download METIS
*******************************************************************************
I think the package that tries to download is inaccessible at the moment. v5.1.0-p4.tar.gz is not the file you end up downloading if you do it manually with this URL: https://bitbucket.org/petsc/pkg-metis/get/v5.1.0-p4.tar.gz. Instead, you end up downloading: petsc-pkg-metis-0adf3ea7785d.tar.gz. Moreover, if you go to https://bitbucket.org/petsc/pkg-metis/, there is no such get subdirectory. I don't know if this actually means this URL is inaccessible.
I've tried downloading METIS from the apt packages and change the --download-metis=1 \ to --with-metis-dir = <dir> but I didn't have luck because I couldn't find the location of METIS on my computer. I removed this METIS version I've manually installed, just in case.

Good news is that I got to this point as I was at like four days ago (being stuck at METIS while compiling petsc, after successfully installing autotools, cmake and mpich). What do you suggest to do? I was thinking about completing the compilation process here on my personal computer and once I get it working and know how everything works, try to get my other computer at the office compiling successfully.

Once again, thanks a lot for your help during the last week. I had very little experience on this but I'm learning on the way. Thanks Justin!

Jjustinquinn · Jul 1, 2019

iisola97
Hi Ignacio,
That's great! Looks like there was some update to Ubuntu 16 that is causing this (but I don't know why we're not seeing the same thing on our nightly run machines).

Regarding PETSc + METIS, I'm showing metis-5.1.0-p1 on the machine I'm on. PETSC_VER is set to "3.6.2" in the install script you're using? Regardless, both packages are available on Bitbucket: https://bitbucket.org/petsc/pkg-metis/downloads/?tab=tags. Now, I'm not sure exactly how it works, but get/ is probably redirected to a request handling script that parses out the URL and responds with the same tarball that is listed under the link I provided.
I would just try rerunning the install script for PETSc: I know this is annoying because it takes so long. You can save a little bit of time by setting --with-debugging=0. But it should work, and I have seen the same thing you're describing, where it seems to not be able to download a given external package, but on rerunning the script, everything works as expected.

I'm happy to help! There's a lot to learn (I'm still learning every day), and there are not a ton of resources out there once you start getting into the specifics of why a certain package wont download or compile.

Iiisola97 · Jul 2, 2019

justinquinn Good news! Finally, I was able to compile ISSM in my personal computer with Ubuntu 18.04.2 LTS. I definitely needed to change the PETSC version from 3.6.7 to 3.6.2; with this one, PETSc installation ran perfectly. Today I tried to compile ISSM in my office computer that has Ubuntu 16.04.5, following all the steps I did on my personal computer but I haven't had luck so far. I compared the versions of all the programs I need to compile ISSM on both computers and, of course, in my personal computer I have newer versions and everything works perfectly. I'm starting to think that the easiest solution is to upgrade my office computer to Ubuntu 18.

Jjustinquinn · Jul 3, 2019

iisola97 Nice! Yes, upgrading to 18.04 sounds like a good plan for the office computer. I will have to look into this as well for our nightly build machines. Glad to hear things are up and running, and don't hesitate to let me know if you have any other questions.

Iiisola97 · Jul 3, 2019

justinquinn I've upgraded the office computer to Ubuntu 18.04 and all the software packages needed to compile ISSM (subversion, gcc, g++, gfortran, make, openssl and ca-certificates) were upgraded too.
However, when I try to start the compilation process with the installation of the external packages, I got this error:
ignacio@mangala:~$ cd $ISSM_DIR/externalpackages/autotools
ignacio@mangala:~/research/mars/ISSM_source/trunk/externalpackages/autotools$ ./install.sh
=== INSTALLING M4 ==
gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now
This issue was previously solved by installing ca-certificates, but now it's already installed.
It's strange because I'm still able to compile everything perfectly in my personal computer with the exact same software packages. I don't know is something previously installed here is messing everything up.
As I think I won't be able to find which thing is conflicting with the compilation process, I think it's better to delete the operating system on the computer and install Ubuntu 18.04 from scratch. I know this worked with my personal computer so it should work again here. What do you think?

Thanks a lot!

Jjustinquinn · Jul 3, 2019

iisola97
Hey Ignacio,
Yep: a clean OS install sounds like it might be quicker (and less painful) than trying to track down conflicts in this case. Ubuntu is a pretty quick install, luckily. Let me know how it goes.