MATLAB error message: undefined symbol: _intel_fast_memmov

ansonsit

While I am testing ISSM on MATLAB, the following error occurs :

>> issmversion

Invalid MEX-file '/users/username/test/issm/install/lib/IssmConfig.mexa64': /users/username/test/issm/src/externalpackages/petsc/install/lib/libmetis.so: undefined symbol:

_intel_fast_memmove

Error in issmversion (line 13)

disp();

larour

This is because the compilation of the matlab modules is not finding a library, usually libirc.a

On what machine are you compiling, with what configuration? Are you on a cluster?

ansonsit

I installed ISSM in a cluster with SUSE Linux v11 using the following configure.sh :

./configure \

--prefix="$ISSM_DIR/../install" \

--with-matlab-dir="$ISSM_DIR/externalpackages/matlab/install" \

--with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \

--with-mpi-include="$ISSM_DIR/externalpackages/mpich/install/include" \

--with-mpi-libflags="-L$ISSM_DIR/externalpackages/mpich/install/lib/ -lmpich" \

--with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \

--with-metis-dir="$ISSM_DIR/externalpackages/petsc/install" \

--with-blas-lapack-dir="$ISSM_DIR/externalpackages/petsc/install" \

--with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install/" \

--with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install/" \

--with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \

--with-numthreads=8 \

--enable-debugging \

CC=icc CXX=icpc F77=ifort LDFLAGS="-L/opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64 -lirc"

larour

Ok, this makes sense. You are using the intel compiler. I see you already included the -lirc

at the end in the LDFLAGS. That's what I was going to suggest.

1) Did you check that the _intel_fast_memmove undefined symbol exists in the irc library?

by doing:

nm libirc.a | grep intel_fast_memmove

if the symbol shows up with an uppercase T (not U), then that's not the problem. If it shows with a U (undefined)

then the libirc does not contain your undefined symbol, which should be somewhere else.

2) how did you compile petsc? did you have it rely on icc also? You need to do this to have a consistent

compilation. Check out the install-3.6-pleiades.sh configuration file in the trunk/externalpackages/petsc. It will give you hints on how to get petsc to use icc.

3) finally, the mpich you are using, did you compile it or are you using the cluster provided one? Usually, the latter is the better option (admins know which mpich works best on their cluster). This also means you have to tailor the issm final configure file and get the mpi-include and mpi-libflags options adjusted to your cluster.

Check out the config-pleiades-petsc3.4.sh in trunk/configs we used on our NASA cluster (very different from the linux config file you used).

Tell me how it goes

Eric L.

ansonsit

I try to follow your suggestion, but the same error is found.

1. The symbol shows up with T with both libirc.a and libirc.so

> nm /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libirc.a | grep intel_fast_memmove

0000000000000030 T _intel_fast_memmove

0000000000000000 T _intel_fast_memmove.A

0000000000000010 T _intel_fast_memmove.M

0000000000000020 T _intel_fast_memmove.P

> nm /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libirc.so| grep intel_fast_memmove

000000000000dce0 T _intel_fast_memmove

000000000000dcb0 T _intel_fast_memmove.A

000000000000dcc0 T _intel_fast_memmove.M

000000000000dcd0 T _intel_fast_memmove.P

2. I compile petsc with the following configuration :

/config/configure.py \

--prefix="$ISSM_DIR/externalpackages/petsc/install" \

--with-mpi-lib="/opt/share/mpich-3.1.1.intel/lib/libmpi.so" \

--with-mpi-include="/opt/share/mpich-3.1.1.intel/include" \

--PETSC_DIR="$ISSM_DIR/externalpackages/petsc/src" \

--with-debugging=0 \

--with-valgrind=0 \

--with-x=0 \

--with-ssl=0 \

--with-shared-libraries=1 \

--download-metis=1 \

--download-parmetis=1 \

--download-mumps=1 \

--download-scalapack=1 \

--download-fblaslapack=1 \

--with-pic=1

3. And the ISSM configuration is :

./configure \

--prefix="$ISSM_DIR/../install" \

--with-matlab-dir="$ISSM_DIR/externalpackages/matlab/install" \

--with-triangle-dir="$ISSM_DIR/externalpackages/triangle/install" \

--with-mpi-include="/opt/share/mpich-3.1.1.intel/include" \

--with-mpi-libflags="-Lopt/share/mpich-3.1.1.intel/lib -lmpi" \

--with-petsc-dir="$ISSM_DIR/externalpackages/petsc/install" \

--with-metis-dir="$ISSM_DIR/externalpackages/petsc/install" \

--with-blas-lapack-dir="$ISSM_DIR/externalpackages/petsc/install" \

--with-scalapack-dir="$ISSM_DIR/externalpackages/petsc/install/" \

--with-mumps-dir="$ISSM_DIR/externalpackages/petsc/install/" \

--with-m1qn3-dir="$ISSM_DIR/externalpackages/m1qn3/install" \

--with-numthreads=8 \

--enable-debugging \

CC=icc CXX=icpc F77=ifort LDFLAGS="-L/opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64 -lirc"

Furthermore, I try to look the library dependency of IssmConfig.mexa64:

> ldd /users/username/test/issm/install/lib/IssmConfig.mexa64

linux-vdso.so.1 => (0x00007fff82cff000)

libISSMMatlab.so.0 => /users/username/test/issm/src/../install/lib/libISSMMatlab.so.0 (0x00007fb0fab03000)

libISSMModules.so.0 => /users/username/test/issm/src/../install/lib/libISSMModules.so.0 (0x00007fb0fa7df000)

libISSMCore.so.0 => /users/username/test/issm/src/../install/lib/libISSMCore.so.0 (0x00007fb0f9cf9000)

libparmetis.so => /users/username/test/issm/src/externalpackages/petsc/install/lib/libparmetis.so (0x00007fb0f9aa4000)

libISSMApi.so.0 => /users/username/test/issm/src/../install/lib/libISSMApi.so.0 (0x00007fb0f98a2000)

libirc.so => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libirc.so (0x00007fb0f964c000)

libmx.so => not found

libmex.so => not found

libmat.so => not found

libmpi.so.12 => /opt/share/mpich-3.1.1.intel/lib/libmpi.so.12 (0x00007fb0f9122000)

librt.so.1 => /lib64/librt.so.1 (0x00007fb0f8f19000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb0f8cfc000)

libpetsc.so.3.5 => /users/username/test/issm/src/externalpackages/petsc/install/lib/libpetsc.so.3.5 (0x00007fb0f7603000)

libmetis.so => /users/username/test/issm/src/externalpackages/petsc/install/lib/libmetis.so (0x00007fb0f736f000)

libimf.so => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libimf.so (0x00007fb0f6eac000)

libirng.so => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libirng.so (0x00007fb0f6ca4000)

libm.so.6 => /lib64/libm.so.6 (0x00007fb0f6a2b000)

libcilkrts.so.5 => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libcilkrts.so.5 (0x00007fb0f67ec000)

libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fb0f64e5000)

libintlc.so.5 => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libintlc.so.5 (0x00007fb0f628f000)

libsvml.so => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libsvml.so (0x00007fb0f5694000)

libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb0f547d000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007fb0f5279000)

libc.so.6 => /lib64/libc.so.6 (0x00007fb0f4efd000)

libmx.so => not found

libmex.so => not found

libmat.so => not found

libmx.so => not found

libmex.so => not found

libmat.so => not found

libifport.so.5 => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libifport.so.5 (0x00007fb0f4ccb000)

libifcore.so.5 => /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libifcore.so.5 (0x00007fb0f498a000)

/lib64/ld-linux-x86-64.so.2 (0x00007fb0fb202000)

libmpifort.so.12 => /opt/share/mpich-3.1.1.intel/lib/libmpifort.so.12 (0x00007fb0f474d000)

libmpicxx.so.12 => /opt/share/mpich-3.1.1.intel/lib/libmpicxx.so.12 (0x00007fb0f452a000)

libirc.so is linked to the desired location. I don't know where the problem lies.

Thank you

glperez

I think the problem you are having is related to a discrepancy between your environment when running ldd on 'IssmConfig.mexa64' versus when actually trying to run ISSM. It is a bit unusual, but it does happen. Basically, there might be another 'libirc.so' that is getting in the way. We run into occasionally with 'libmex.so'. To check if this is the case, you can try a search for 'libirc.so' to see if you find some other instance of it. If you do find a some other 'libirc.so', and it is unrelated to Intel Composer, then this is likely where the problem is coming from.

If you are looking for a quick fix to this issue, then you can replace '-lirc' with 'libirc.a' in your LDFLAGS. This will forcibly link against the static version of this library and thus circumvent this load-time linking issue. The only downside to this workaround is that all your MEX files will become larger. The exact amount will depend on the size of the 'libirc.a', as every MEX file will contain a copy of it.

Let me know what you get!

-Gil

ansonsit

"libirc.so" was really linked to an incorrect library as the library path was overloaded by the matlab library. Before recompilation, I set LD_LIBRARY_PATH variable within matlab. The linking problem was solved, but the error remained there. Therefore, I replaced '-lirc' with 'libirc.a' and recompiled ISSM. The mex file is still linked to "libirc.so" but not "libirc.a". This does not change anything.

glperez

Let me see if I understand this correctly.

When first attempting to build ISSM you ran into an issue with the 'libirc.so' dependency being resolved to the wrong library. Is this correct? If so, where was this other 'libirc.so'?

Also, would you provide us with the output of:

> readelf -d /path/to/matlab/libraries/libmx.so

and

> readelf -d /path/to/matlab/libraries/libmex.so

Also, it would be useful for us to see the contents of the following environment variables: LD_LIBRARY_PATH and LD_RUN_PATH

Finally, are you attempting to run ISSM from the same login session that you built ISSM from, or is this through a job queue?

ansonsit

1. Yes. The LD_LIBRARY_PATH was overloaded by matlab. The other 'libirc.so' library is in the matlab's directory (/opt/share/MATLAB/R2014a/sys/os/glnxa64/libirc.so). I replaced it with ( /opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/libirc.so )

> readelf -d /opt/share/MATLAB/R2014a/bin/glnxa64/libmex.so

Dynamic section at offset 0x23fe8 contains 37 entries:

Tag Type Name/Value

0x0000000000000001 (NEEDED) Shared library:

0x000000000000000f (RPATH) Library rpath:

0x000000000000000c (INIT) 0xf1d8

0x000000000000000d (FINI) 0x1d9f8

0x0000000000000004 (HASH) 0x158

0x0000000000000005 (STRTAB) 0x4770

0x0000000000000006 (SYMTAB) 0x1260

0x000000000000000a (STRSZ) 27146 (bytes)

0x000000000000000b (SYMENT) 24 (bytes)

0x0000000000000003 (PLTGOT) 0x2243f8

0x0000000000000002 (PLTRELSZ) 4968 (bytes)

0x0000000000000014 (PLTREL) RELA

0x0000000000000017 (JMPREL) 0xde70

0x0000000000000007 (RELA) 0xb710

0x0000000000000008 (RELASZ) 10080 (bytes)

0x0000000000000009 (RELAENT) 24 (bytes)

0x000000006ffffffc (VERDEF) 0xb5e8

0x000000006ffffffd (VERDEFNUM) 4

0x000000006ffffffe (VERNEED) 0xb660

0x000000006fffffff (VERNEEDNUM) 4

0x000000006ffffff0 (VERSYM) 0xb17a

0x000000006ffffff9 (RELACOUNT) 55

0x0000000000000000 (NULL) 0x0

> readelf -d /opt/share/MATLAB/R2014a/bin/glnxa64/libmx.so

Dynamic section at offset 0x123d98 contains 46 entries:

Tag Type Name/Value

0x0000000000000001 (NEEDED) Shared library:

0x000000000000000f (RPATH) Library rpath:

0x000000000000000c (INIT) 0x49110

0x000000000000000d (FINI) 0xf6038

0x0000000000000004 (HASH) 0x158

0x0000000000000005 (STRTAB) 0x14008

0x0000000000000006 (SYMTAB) 0x4a68

0x000000000000000a (STRSZ) 147472 (bytes)

0x000000000000000b (SYMENT) 24 (bytes)

0x0000000000000003 (PLTGOT) 0x324738

0x0000000000000002 (PLTRELSZ) 22536 (bytes)

0x0000000000000014 (PLTREL) RELA

0x0000000000000017 (JMPREL) 0x43908

0x0000000000000007 (RELA) 0x395e8

0x0000000000000008 (RELASZ) 41760 (bytes)

0x0000000000000009 (RELAENT) 24 (bytes)

0x000000006ffffffc (VERDEF) 0x39490

0x000000006ffffffd (VERDEFNUM) 4

0x000000006ffffffe (VERNEED) 0x39508

0x000000006fffffff (VERNEEDNUM) 5

0x000000006ffffff0 (VERSYM) 0x38018

0x000000006ffffff9 (RELACOUNT) 469

0x0000000000000000 (NULL) 0x0

> echo $LD_LIBRARY_PATH

/users/username/test/issm/src/externalpackages/petsc/install/lib:/opt/share/Python-3.4.2/lib:/opt/share/perl-5.20.1/lib:/opt/share/mpich-3.1.1.intel/lib:/opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:/opt/share/intel/composer_xe_2013_sp1.3.174/mpirt/lib/intel64:/opt/share/intel/composer_xe_2013_sp1.3.174/ipp/../compiler/lib/intel64:/opt/share/intel/composer_xe_2013_sp1.3.174/ipp/lib/intel64:/opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:/opt/share/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64:/opt/share/intel/composer_xe_2013_sp1.3.174/tbb/lib/intel64/gcc4.1

LD_RUN_PATH has not been set.

glperez

Everything looks fine there. Would you now run:

> readelf -d $ISSM_DIR/externalpackages/petsc/install/lib/libmetis.so

I expect to see METIS have 'libirc.so' as a dependency, but I'm more interested in the RPATH. If the RPATH is set incorrectly, then this might be causing your problem.

Now, if you are more concerned with just having something that works, then I have a potential fix for your problem that I would like you to try out. The idea is going to be to try and link against 'libirc.a' exclusively. According to your original post the library that first needed 'libirc.so' was the METIS library which is installed as part of the PetsC external package. However, other libraries installed by PetsC might require it explicitly as well. The way I see it you have two options:

1) Build METIS yourself using one of the script in '$ISSM_DIR/externalpackages/metis' and subsequently rebuild PetsC such that it links against this newly installed METIS and not a version that it builds.

2) Rebuild PetsC completely statically.

The first option will require a bit more work than the second, but the end result will be cleaner. The second option is definitely easier, but is overkill.

Let me know if you would like to try fix the linking issue as it currently stands or the two rebuilding options.

-Gil

mathieumorlighem

Hi,

I just had a similar error on pleiades. This is not a problem related to ISSM but to MATLAB. Matlab loads its own libirc.so that does not provide the symbol _intel_fast_memmov. The good news is that it is easy to fix. I posted the fix on the website: http://issm.jpl.nasa.gov/documentation/faq/matlab/.

You basically need to do the following:

cp /opt/share/MATLAB/R2014a/bin/.matlab7rc.sh ~/

You then need to open ~/.matlab7rc.sh and go to the case that corresponds to your architecture (most likely glnx86|glnxa64). In there, replace:

# LDPATH_PREFIX='$MATLAB/sys/opengl/lib/$ARCH'

LDPATH_PREFIX='/opt/share/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64/'

You can then restart matlab from anywhere and the problem should disappear.

Let us know if you still encounter this issue

Best

Mathieu