Topic: Open MPI library

Dear,

I tried to run the configure and build the parallel version. But I still can not figure out what open MPI program should I use and how to do it. Please help me on that. Thanks,

Best Regards,

Xuejian

2

Re: Open MPI library

Hi Xuejian,

1) to compile parallel oofem version, you first need to install MPI library. Depending on your platform, it may come with the installation, or you have to install additional MPI package, or you need to compile MPI library by yourself. If MPI is provided for your platform, use it. if not, we recommend to use openMPI (http://www.open-mpi.org/). To compile the library, please follow the installation instructions provided with the sources of MPI library.
2) petsc (http://www-unix.mcs.anl.gov/petsc/petsc-2/) is also highly recommended (unless you are interested only in explicit dynamics). Most likely, you need to install petsc by yourself.
3) then you can configure and compile parallel oofem. The configuration is explained in
oofem wiki (see particularly this http://www.oofem.org/wiki/doku.php?id=i … el_oofem).

Hope that it helps,
Borek

Re: Open MPI library

Thank you for your instruction, Borek.

I found out the problem and installed the mpi successfully. But I have another difficulty on installing petsc. I still can not use the option "--with-PETSCARCH=arch" correctly. After I configured petsc, the output says PETSC_ARCH: linux-gnu-c-debug.  I used "make PETSC_ARCH=linux-gnu-c-debug" to make petsc. Then I went back to oofem directory and still could not run ./configure correctly. Here is the command:

./configure OOFEM_TARGET=poofem-release CXXFLAGS=-02 --enable-poofem --with-MPIDIR=/home/liuxueji/oofem-1.8/mpi/ --enable-petsc --with-PETSCDIR=/home/liuxueji/oofem-1.8/petsc-3.0.0-p3/ --with-PETSCARCH=linux-gnu-c-debug

Could you help me check what's wrong? Thanks,

Xuejian

4

Re: Open MPI library

Hi Xuejian,

o) I am compiling PETSc with c++ compiler, using -with-clanguage=cxx as parameter to ./config/configure.py, so my PETSC_ARCH looks like "linux-gnu-cxx-debug"

o) the available PETSC_ARCH can be easily explored - in PETSc directory, do "cd lib; ls". This will list at least the configured ones.

Let me know.
Borek

Re: Open MPI library

Thank you for your help again, Borek.

Sorry, I still could not work it out. The running always says

Type in available PETSC_ARCH:
PETSC_ARCH  does not seem to be available.

I followed your step and ran the petsc test successfully. I used your "cd lib; ls", I only got some .a files and I could not recognize which one is the available PETSC_ARCH.

I am waiting  for your help. Thanks,

Xuejian

Re: Open MPI library

Dear Borek,

I think I solved the problem on PETSC_ARCH by setting the variable in .login file. Now I could run the ./configure correctly. But I have errors when I make the target. Could you check the output I pasted in the following? Thank you very much.

Xuejian

<65 [liuxueji@winry]:~/oofem-1.8 >./configure OOFEM_TARGET=poofem-release --enable-poofem --with-MPIDIR=/home/liuxueji/oofem-1.8/mpi --enable-petsc --with-PETSCDIR=/home/liuxueji/oofem-1.8/petsc-2.3.3-p15 --with-PETSCARCH=linux-gnu-cxx-debug --enable-dss --enable-oofeg --with-CKITDIR=/home/liuxueji/oofem-1.8/Ckit --with-ELIXIRDIR=/home/liuxueji/oofem-1.8/Elixir
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking OOFEM_DIR... /home/liuxueji/oofem-1.8
configuring OOFEM_TARGET=poofem-release
existing target directory found
target directory configured
checking for g++... g++
checking for C++ compiler default output file name... a.out
checking whether the C++ compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
checking for a BSD-compatible install... /usr/bin/install -c
checking for ranlib... ranlib
checking whether make sets $(MAKE)... yes
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for strings.h... (cached) yes
checking for unistd.h... (cached) yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking sys/resource.h usability... yes
checking sys/resource.h presence... yes
checking for sys/resource.h... yes
checking whether time.h and sys/time.h may both be included... yes
checking for strstr... yes
checking for strtod... yes
checking for strtol... yes
checking for strncasecmp... yes
checking for access... yes
checking for nearest... no
checking FALSE, TRUE enum/constants... no
checking typedef bool... yes
checking M_PI constant... yes
checking M_LN2 constant... yes
checking STL support... yes
checking Macro __VA_ARGS__ support... yes
checking Function Templates support... yes
checking for sqrt in -lm... yes
checking dss module location... src/dss
checking MPI location... /home/liuxueji/oofem-1.8/mpi
checking for mpicxx... /home/liuxueji/oofem-1.8/mpi/bin/mpicxx
checking PETSc module location... /home/liuxueji/oofem-1.8/petsc-2.3.3-p15
checking PETSC_ARCH... linux-gnu-cxx-debug
checking for X... libraries , headers
checking for gethostbyname... yes
checking for connect... yes
checking for remove... yes
checking for shmat... yes
checking for IceConnectionNumber in -lICE... yes
checking for XtNameToWidget in -lXt... yes
checking ckitcfg.h usability... no
checking ckitcfg.h presence... no
checking for ckitcfg.h... no
checking guess of Ckit includes location... /home/liuxueji/oofem-1.8/Ckit/include
checking for _get_list_next in -lckit... no
checking guess of libckit.a location... /home/liuxueji/oofem-1.8/Ckit/src
checking Ecfg.h usability... no
checking Ecfg.h presence... no
checking for Ecfg.h... no
checking guess of Elixir includes location... /home/liuxueji/oofem-1.8/Elixir/include
checking for ElixirInit in -lelixir... no
checking guess of libelixir.a location... /home/liuxueji/oofem-1.8/Elixir/src
checking for ESIBuildInterface in -lesi... no
checking guess of libesi.a location... /home/liuxueji/oofem-1.8/Elixir/src
checking Determining hostname... winry
checking OOFEM version... 1.8
configure: creating ./config.status
config.status: creating makefile
config.status: creating targets/poofem-release/include/oofemdef.h
config.status: creating targets/poofem-release/makefile
config.status: creating targets/poofem-release/lib/oofemlib/makefile
config.status: creating targets/poofem-release/lib/sm/makefile
config.status: creating targets/poofem-release/lib/tm/makefile
config.status: creating targets/poofem-release/lib/fm/makefile
config.status: creating targets/poofem-release/lib/dss/makefile
config.status: creating targets/poofem-release/lib/main/makefile
config.status: creating tools/makefile
config.status: creating targets/poofem-release/include/oofemcfg.h
config.status: targets/poofem-release/include/oofemcfg.h is unchanged

Modules to be included: oofemlib sm tm fm dss petsc
Target configured: poofem-release
PROG_NAME: oofeg
<66 [liuxueji@winry]:~/oofem-1.8 >cd targets/poofem-release
<67 [liuxueji@winry]:~/oofem-1.8/targets/poofem-release >make
make -C lib/oofemlib all
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib'
makefile:80: warning: overriding commands for target `clean'
/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/common/rules:95: warning: ignoring old commands for target `clean'
set -e; \
        /home/liuxueji/oofem-1.8/mpi/bin/mpicxx -MM -I/home/liuxueji/oofem-1.8/targets/poofem-release/include   -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15 -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/linux-gnu-cxx-debug -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/include  -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/externalpackages/mpich2-1.0.5p4/linux-gnu-cxx-debug/include      -I/home/liuxueji/oofem-1.8/src/dss -I/home/liuxueji/oofem-1.8/Ckit/include -I/home/liuxueji/oofem-1.8/Elixir/include -I/home/liuxueji/oofem-1.8/mpi/include -D__OOFEM -D__PARALLEL_MODE -DMPICH_IGNORE_CXX_SEEK -D__OOFEMLIB_MODULE -D__SM_MODULE -D__TM_MODULE -D__FM_MODULE -D__DSS_MODULE -D__PETSC_MODULE /home/liuxueji/oofem-1.8/src/oofemlib/engngm.C | sed 's|\(engngm\)\.o[ :]*|\1.o engngm.d : |g' > engngm.d; \
        [ -s engngm.d ] || rm -f engngm.d
g++: -lmpi_cxx: linker input file unused because linking not done
g++: -lmpi: linker input file unused because linking not done
g++: -lopen-rte: linker input file unused because linking not done
g++: -lopen-pal: linker input file unused because linking not done
g++: -ldl: linker input file unused because linking not done
g++: --export-dynamic: linker input file unused because linking not done
g++: -lnsl: linker input file unused because linking not done
g++: -lutil: linker input file unused because linking not done
g++: -lm: linker input file unused because linking not done
g++: -ldl: linker input file unused because linking not done
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib'
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib'
makefile:80: warning: overriding commands for target `clean'
/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/common/rules:95: warning: ignoring old commands for target `clean'
make[1]: Warning: File `engngm.d' has modification time 0.012 s in the future
/home/liuxueji/oofem-1.8/mpi/bin/mpicxx -c -O -D__OOFEM -D__PARALLEL_MODE -DMPICH_IGNORE_CXX_SEEK -D__OOFEMLIB_MODULE -D__SM_MODULE -D__TM_MODULE -D__FM_MODULE -D__DSS_MODULE -D__PETSC_MODULE  -I/home/liuxueji/oofem-1.8/targets/poofem-release/include   -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15 -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/linux-gnu-cxx-debug -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/include  -I/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/externalpackages/mpich2-1.0.5p4/linux-gnu-cxx-debug/include      -I/home/liuxueji/oofem-1.8/src/dss -I/home/liuxueji/oofem-1.8/Ckit/include -I/home/liuxueji/oofem-1.8/Elixir/include -I/home/liuxueji/oofem-1.8/mpi/include /home/liuxueji/oofem-1.8/src/oofemlib/engngm.C -o engngm.o
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C: In member function 'void EngngModel::petsc_assembleVectorFromDofManagers(_p_Vec*, TimeStep*, EquationID, CharType, ValueModeType, Domain*)':
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C:1212: error: cannot convert 'double*' to 'const PetscScalar*' for argument '4' to 'PetscErrorCode VecSetValues(_p_Vec*, PetscInt, const PetscInt*, const PetscScalar*, InsertMode)'
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C: In member function 'void EngngModel::petsc_assemblePrescribedVectorFromDofManagers(_p_Vec*, TimeStep*, EquationID, CharType, ValueModeType, Domain*)':
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C:1256: error: cannot convert 'double*' to 'const PetscScalar*' for argument '4' to 'PetscErrorCode VecSetValues(_p_Vec*, PetscInt, const PetscInt*, const PetscScalar*, InsertMode)'
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C: In member function 'void EngngModel::petsc_assembleVectorFromElements(_p_Vec*, TimeStep*, EquationID, CharType, ValueModeType, Domain*)':
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C:1301: error: cannot convert 'double*' to 'const PetscScalar*' for argument '4' to 'PetscErrorCode VecSetValues(_p_Vec*, PetscInt, const PetscInt*, const PetscScalar*, InsertMode)'
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C: In member function 'void EngngModel::petsc_assemblePrescribedVectorFromElements(_p_Vec*, TimeStep*, EquationID, CharType, ValueModeType, Domain*)':
/home/liuxueji/oofem-1.8/src/oofemlib/engngm.C:1347: error: cannot convert 'double*' to 'const PetscScalar*' for argument '4' to 'PetscErrorCode VecSetValues(_p_Vec*, PetscInt, const PetscInt*, const PetscScalar*, InsertMode)'
make[1]: *** [engngm.o] Error 1
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib'
make: *** [oofemlib_mod] Error 2

7

Re: Open MPI library

Hi Xuejian,

I thought that I figured out the problem - a recent release of PETSc has made some changes in its directory structure (that's why PETSC_ARCH could not be properly identified) and also the interface has changed. I am working on that at the moment, and will post an update soon (into repository version). If you are using release version, or if you don;t want to wait, please download and install PETSc  ver 2.3.3, This should work.

From your last post it is obvious, that it does not work - since you are using petsc-2.3.3-p15. How the PETSc has been configured? (can you post here all configuration parameters, that you have used for PETSc). It seems to me that you have used some options, that can change PetscScalar type (-with-scalar-type, --with-precision).

Borek

Re: Open MPI library

Dear Borek,

You are exactly right. I used a --with-scalar-type=complex, which was copied from somewhere and caused the problem. I ran out of that drop in another one. Please see the following output and help me. Thanks,

Xuejian

/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `MPI_Register_datarep'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_signed_char'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_lb'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_double_int'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_errors_are_fatal'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_op_replace'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_datatype_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_info_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_op_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_group_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_group_empty'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_errors_return'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_mpi_unsigned_long_long'
/home/liuxueji/oofem-1.8/openmpi-1.3/openmpi-1.3/lib/libmpi_cxx.so: undefined re                                                                                            ference to `ompi_attr_create_keyval'
collect2: ld returned 1 exit status
make[1]: *** [/home/liuxueji/oofem-1.8/targets/poofem-release/bin/oofeg] Error 1
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/                                                                                            main'
make: *** [all] Error 2

Re: Open MPI library

Complete output after make:

make -C lib/oofemlib all
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib'
makefile:80: warning: overriding commands for target `clean'
/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/common/rules:95: warning: ignoring old commands for target `clean'
echo all target done
all target done
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib'
make -C lib/sm all
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm'
makefile:79: warning: overriding commands for target `clean'
/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/common/rules:95: warning: ignoring old commands for target `clean'
echo all target done
all target done
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm'
make -C lib/tm all
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/tm'
makefile:77: warning: overriding commands for target `clean'
/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/common/rules:95: warning: ignoring old commands for target `clean'
echo all target done
all target done
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/tm'
make -C lib/fm all
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/fm'
makefile:75: warning: overriding commands for target `clean'
/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/common/rules:95: warning: ignoring old commands for target `clean'
echo all target done
all target done
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/fm'
make -C lib/dss all
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/dss'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/dss'
cd lib/main; make all
make[1]: Entering directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/main'
makefile:100: warning: overriding commands for target `clean'
/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/bmake/common/rules:95: warning: ignoring old commands for target `clean'
/home/liuxueji/oofem-1.8/openmpi-1.3/bin/mpicxx -O2 -D__OOFEM -D__PARALLEL_MODE -DMPICH_IGNORE_CXX_SEEK -D__OOFEMLIB_MODULE -D__SM_MODULE -D__TM_MODULE -D__FM_MODULE -D__IML_MODULE -D__DSS_MODULE -D__PETSC_MODULE -D__PARMETIS_MODULE   -o /home/liuxueji/oofem-1.8/targets/poofem-release/bin/oofeg main.o oofeg.o usrdefsub.o /home/liuxueji/oofem-1.8/targets/poofem-release/lib/fm/fmlib.a /home/liuxueji/oofem-1.8/targets/poofem-release/lib/tm/tmlib.a /home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a /home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a /home/liuxueji/oofem-1.8/targets/poofem-release/lib/dss/dsslib.a /home/liuxueji/oofem-1.8/Elixir/src/libesi.a /home/liuxueji/oofem-1.8/Elixir/src/libelixir.a /home/liuxueji/oofem-1.8/Ckit/src/libckit.a -Wl,-rpath,/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/lib/linux-gnu-cxx-debug -Wl,-rpath,/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/lib/linux-gnu-cxx-debug -L/home/liuxueji/oofem-1.8/petsc-2.3.3-p15/lib/linux-gnu-cxx-debug -lpetscts -lpetscsnes -lpetscksp -lpetscdm -lpetscmat -lpetscvec -lpetsc        -lX11  -llapack -lblas -Wl,-rpath,/usr/lib64 -L/usr/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../lib64 -Wl,-rpath,/lib/../lib64 -L/lib/../lib64 -Wl,-rpath,/usr/lib/../lib64 -L/usr/lib/../lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../.. -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../.. -ldl -llammpio -llamf77mpi -lmpi -llam -lutil -lgcc_s -lpthread -lgfortranbegin -lgfortran -lm -Wl,-rpath,/usr/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../lib64 -Wl,-rpath,/lib/../lib64 -Wl,-rpath,/usr/lib/../lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../.. -lm -Wl,-rpath,/usr/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../lib64 -Wl,-rpath,/lib/../lib64 -Wl,-rpath,/usr/lib/../lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../.. -llammpi++ -lstdc++ -Wl,-rpath,/usr/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../lib64 -Wl,-rpath,/lib/../lib64 -Wl,-rpath,/usr/lib/../lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../.. -llammpi++ -lstdc++ -Wl,-rpath,/usr/lib64 -L/usr/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../lib64 -Wl,-rpath,/lib/../lib64 -L/lib/../lib64 -Wl,-rpath,/usr/lib/../lib64 -L/usr/lib/../lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/lib -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../../../x86_64-pc-linux-gnu/lib -Wl,-rpath,/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../.. -L/usr/lib/gcc/x86_64-pc-linux-gnu/4.1.2/../../.. -ldl -llammpio -llamf77mpi -lmpi -llam -lutil -lgcc_s -lpthread -ldl  /home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a /home/liuxueji/oofem-1.8/ParMetis-3.1.1/libmetis.a  -lm   -lSM -lICE  -lXaw -lXmu -lXt -lXext -lX11  -lm   
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(engngm.o): In function `EngngModel::giveOutputStream()':
engngm.C:(.text+0x2e4): warning: the use of `tmpnam' is dangerous, better use `mkstemp'
main.o: In function `main':
main.C:(.text+0x237): undefined reference to `ompi_mpi_comm_world'
main.o: In function `MPI::Grequest::Complete()':
main.C:(.text._ZN3MPI8Grequest8CompleteEv[MPI::Grequest::Complete()]+0x5): undefined reference to `MPI_Grequest_complete'
main.o: In function `MPI::Request::Get_status() const':
main.C:(.text._ZNK3MPI7Request10Get_statusEv[MPI::Request::Get_status() const]+0x10): undefined reference to `MPI_Request_get_status'
main.o: In function `MPI::Request::Get_status(MPI::Status&) const':
main.C:(.text._ZNK3MPI7Request10Get_statusERNS_6StatusE[MPI::Request::Get_status(MPI::Status&) const]+0x1d): undefined reference to `MPI_Request_get_status'
main.o: In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)':
main.C:(.text._ZN3MPI2Op4InitEPFvPKvPviRKNS_8DatatypeEEb[MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)]+0x3e): undefined reference to `ompi_op_set_cxx_callback'
main.o: In function `MPI::Status::Set_cancelled(bool)':
main.C:(.text._ZN3MPI6Status13Set_cancelledEb[MPI::Status::Set_cancelled(bool)]+0x9): undefined reference to `MPI_Status_set_cancelled'
main.o: In function `MPI::Win::Call_errhandler(int) const':
main.C:(.text._ZNK3MPI3Win15Call_errhandlerEi[MPI::Win::Call_errhandler(int) const]+0x5): undefined reference to `MPI_Win_call_errhandler'
main.o: In function `MPI::Win::Unlock(int) const':
main.C:(.text._ZNK3MPI3Win6UnlockEi[MPI::Win::Unlock(int) const]+0x9): undefined reference to `MPI_Win_unlock'
main.o: In function `MPI::Win::Test() const':
main.C:(.text._ZNK3MPI3Win4TestEv[MPI::Win::Test() const]+0xe): undefined reference to `MPI_Win_test'
main.o: In function `MPI::Win::Lock(int, int, int) const':
main.C:(.text._ZNK3MPI3Win4LockEiii[MPI::Win::Lock(int, int, int) const]+0x11): undefined reference to `MPI_Win_lock'
main.o: In function `MPI::Status::Set_elements(MPI::Datatype const&, int)':
main.C:(.text._ZN3MPI6Status12Set_elementsERKNS_8DatatypeEi[MPI::Status::Set_elements(MPI::Datatype const&, int)]+0x9): undefined reference to `MPI_Status_set_elements'
main.o: In function `MPI::Datatype::Create_indexed_block(int, int, int const*) const':
main.C:(.text._ZNK3MPI8Datatype20Create_indexed_blockEiiPKi[MPI::Datatype::Create_indexed_block(int, int, int const*) const]+0x1b): undefined reference to `MPI_Type_create_indexed_block'
main.o: In function `MPI::Graphcomm::Clone() const':
main.C:(.text._ZNK3MPI9Graphcomm5CloneEv[MPI::Graphcomm::Clone() const]+0x54): undefined reference to `ompi_mpi_comm_null'
main.C:(.text._ZNK3MPI9Graphcomm5CloneEv[MPI::Graphcomm::Clone() const]+0x6f): undefined reference to `ompi_mpi_comm_null'
main.o: In function `MPI::Intracomm::Create_graph(int, int const*, int const*, bool) const':
main.C:(.text._ZNK3MPI9Intracomm12Create_graphEiPKiS2_b[MPI::Intracomm::Create_graph(int, int const*, int const*, bool) const]+0x59): undefined reference to `ompi_mpi_comm_null'
main.C:(.text._ZNK3MPI9Intracomm12Create_graphEiPKiS2_b[MPI::Intracomm::Create_graph(int, int const*, int const*, bool) const]+0x74): undefined reference to `ompi_mpi_comm_null'
main.o: In function `MPI::Intracomm::Clone() const':
main.C:(.text._ZNK3MPI9Intracomm5CloneEv[MPI::Intracomm::Clone() const]+0x52): undefined reference to `ompi_mpi_comm_null'
main.o:main.C:(.text._ZNK3MPI9Intracomm5CloneEv[MPI::Intracomm::Clone() const]+0x6b): more undefined references to `ompi_mpi_comm_null' follow
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(nlinearstatic.o): In function `NonLinearStatic::estimateMaxPackSize(IntArray&, CommunicationBuffer&, int)':
nlinearstatic.C:(.text+0x182f): undefined reference to `ompi_mpi_double'
nlinearstatic.C:(.text+0x18e2): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(nlinearstatic.o): In function `int Communicator::unpackAllData<StructuralEngngModel, FloatArray>(StructuralEngngModel*, FloatArray*, int (StructuralEngngModel::*)(FloatArray*, ProcessCommunicator&))':
nlinearstatic.C:(.text._ZN12Communicator13unpackAllDataI20StructuralEngngModel10FloatArrayEEiPT_PT0_MS3_FiS6_R19ProcessCommunicatorE[int Communicator::unpackAllData<StructuralEngngModel, FloatArray>(StructuralEngngModel*, FloatArray*, int (StructuralEngngModel::*)(FloatArray*, ProcessCommunicator&))]+0x204): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(nlinearstatic.o): In function `int Communicator::unpackAllData<StructuralEngngModel>(StructuralEngngModel*, int (StructuralEngngModel::*)(ProcessCommunicator&))':
nlinearstatic.C:(.text._ZN12Communicator13unpackAllDataI20StructuralEngngModelEEiPT_MS2_FiR19ProcessCommunicatorE[int Communicator::unpackAllData<StructuralEngngModel>(StructuralEngngModel*, int (StructuralEngngModel::*)(ProcessCommunicator&))]+0x1f4): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(pnldeidynamic.o): In function `PNlDEIDynamic::estimateMaxPackSize(IntArray&, CommunicationBuffer&, int)':
pnldeidynamic.C:(.text+0xea1): undefined reference to `ompi_mpi_double'
pnldeidynamic.C:(.text+0xf60): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(pnldeidynamic.o): In function `PNlDEIDynamic::computeMassMtrx(FloatArray&, double&, TimeStep*)':
pnldeidynamic.C:(.text+0x1772): undefined reference to `ompi_mpi_comm_world'
pnldeidynamic.C:(.text+0x1778): undefined reference to `ompi_mpi_op_max'
pnldeidynamic.C:(.text+0x177d): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(pnldeidynamic.o): In function `PNlDEIDynamic::solveYourselfAt(TimeStep*)':
pnldeidynamic.C:(.text+0x1e58): undefined reference to `ompi_mpi_comm_world'
pnldeidynamic.C:(.text+0x1e5e): undefined reference to `ompi_mpi_op_sum'
pnldeidynamic.C:(.text+0x1e63): undefined reference to `ompi_mpi_double'
pnldeidynamic.C:(.text+0x2099): undefined reference to `ompi_mpi_comm_world'
pnldeidynamic.C:(.text+0x209f): undefined reference to `ompi_mpi_op_sum'
pnldeidynamic.C:(.text+0x20a4): undefined reference to `ompi_mpi_double'
pnldeidynamic.C:(.text+0x2823): undefined reference to `ompi_mpi_comm_world'
pnldeidynamic.C:(.text+0x2829): undefined reference to `ompi_mpi_op_sum'
pnldeidynamic.C:(.text+0x282e): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(pnldeidynamic.o): In function `int Communicator::unpackAllData<PNlDEIDynamic>(PNlDEIDynamic*, int (PNlDEIDynamic::*)(ProcessCommunicator&))':
pnldeidynamic.C:(.text._ZN12Communicator13unpackAllDataI13PNlDEIDynamicEEiPT_MS2_FiR19ProcessCommunicatorE[int Communicator::unpackAllData<PNlDEIDynamic>(PNlDEIDynamic*, int (PNlDEIDynamic::*)(ProcessCommunicator&))]+0x1f4): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(idmnl1.o): In function `IDNLMaterial::estimatePackSize(CommunicationBuffer&, GaussPoint*)':
idmnl1.C:(.text+0x6c): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(mazarsmodelnl.o): In function `MazarsNLMaterial::estimatePackSize(CommunicationBuffer&, GaussPoint*)':
mazarsmodelnl.C:(.text+0x7c): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(linearstatic.o): In function `LinearStatic::estimateMaxPackSize(IntArray&, CommunicationBuffer&, int)':
linearstatic.C:(.text+0x763): undefined reference to `ompi_mpi_double'
linearstatic.C:(.text+0x80f): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(adaptnlinearstatic.o): In function `AdaptiveNonLinearStatic::solveYourselfAt(TimeStep*)':
adaptnlinearstatic.C:(.text+0x9d0): undefined reference to `ompi_mpi_comm_world'
adaptnlinearstatic.C:(.text+0x9d6): undefined reference to `ompi_mpi_op_max'
adaptnlinearstatic.C:(.text+0x9e3): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(huertaerrorestimator.o): In function `HuertaErrorEstimator::estimateError(EE_ErrorMode, TimeStep*)':
huertaerrorestimator.C:(.text+0x6d00): undefined reference to `ompi_mpi_comm_world'
huertaerrorestimator.C:(.text+0x6d06): undefined reference to `ompi_mpi_op_sum'
huertaerrorestimator.C:(.text+0x6d0b): undefined reference to `ompi_mpi_double'
huertaerrorestimator.C:(.text+0x70fd): undefined reference to `ompi_mpi_comm_world'
huertaerrorestimator.C:(.text+0x7103): undefined reference to `ompi_mpi_op_sum'
huertaerrorestimator.C:(.text+0x7108): undefined reference to `ompi_mpi_double'
huertaerrorestimator.C:(.text+0x759a): undefined reference to `ompi_mpi_comm_world'
huertaerrorestimator.C:(.text+0x75a0): undefined reference to `ompi_mpi_op_sum'
huertaerrorestimator.C:(.text+0x75ad): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `FETISolver::estimateMaxPackSize(IntArray&, CommunicationBuffer&, int&)':
fetisolver.C:(.text+0x624): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `FETISolver::setUpCommunicationMaps()':
fetisolver.C:(.text+0x17f4): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x1879): undefined reference to `ompi_mpi_int'
fetisolver.C:(.text+0x18bc): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x18e2): undefined reference to `ompi_mpi_int'
fetisolver.C:(.text+0x1999): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x19a3): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `FETISolver::solve(SparseMtrx*, FloatArray*, FloatArray*)':
fetisolver.C:(.text+0x2de8): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x2efc): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x2f83): undefined reference to `ompi_mpi_int'
fetisolver.C:(.text+0x311d): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x3414): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x3456): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x3474): undefined reference to `ompi_mpi_int'
fetisolver.C:(.text+0x3548): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x35b3): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x375a): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x37c0): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x37d9): undefined reference to `ompi_mpi_int'
fetisolver.C:(.text+0x38f9): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x3977): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x3c53): undefined reference to `ompi_mpi_int'
fetisolver.C:(.text+0x3d26): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x3d8c): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x401d): undefined reference to `ompi_mpi_comm_world'
fetisolver.C:(.text+0x4760): undefined reference to `ompi_mpi_int'
fetisolver.C:(.text+0x47ed): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::unpackArray(char*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer11unpackArrayEPci[StaticCommunicationBuffer::unpackArray(char*, int)]+0xf): undefined reference to `ompi_mpi_char'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::unpackArray(double*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer11unpackArrayEPdi[StaticCommunicationBuffer::unpackArray(double*, int)]+0xf): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::unpackArray(unsigned long*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer11unpackArrayEPmi[StaticCommunicationBuffer::unpackArray(unsigned long*, int)]+0xf): undefined reference to `ompi_mpi_unsigned_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::unpackArray(long*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer11unpackArrayEPli[StaticCommunicationBuffer::unpackArray(long*, int)]+0xf): undefined reference to `ompi_mpi_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::unpackArray(int*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer11unpackArrayEPii[StaticCommunicationBuffer::unpackArray(int*, int)]+0xf): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::packArray(char const*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer9packArrayEPKci[StaticCommunicationBuffer::packArray(char const*, int)]+0xf): undefined reference to `ompi_mpi_char'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::packArray(double const*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer9packArrayEPKdi[StaticCommunicationBuffer::packArray(double const*, int)]+0xf): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::packArray(unsigned long const*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer9packArrayEPKmi[StaticCommunicationBuffer::packArray(unsigned long const*, int)]+0xf): undefined reference to `ompi_mpi_unsigned_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::packArray(long const*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer9packArrayEPKli[StaticCommunicationBuffer::packArray(long const*, int)]+0xf): undefined reference to `ompi_mpi_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `StaticCommunicationBuffer::packArray(int const*, int)':
fetisolver.C:(.text._ZN25StaticCommunicationBuffer9packArrayEPKii[StaticCommunicationBuffer::packArray(int const*, int)]+0xf): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(fetisolver.o): In function `int Communicator::unpackAllData<FETISolver>(FETISolver*, int (FETISolver::*)(ProcessCommunicator&))':
fetisolver.C:(.text._ZN12Communicator13unpackAllDataI10FETISolverEEiPT_MS2_FiR19ProcessCommunicatorE[int Communicator::unpackAllData<FETISolver>(FETISolver*, int (FETISolver::*)(ProcessCommunicator&))]+0x1f7): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/sm/smlib.a(feticommunicator.o): In function `FETICommunicator::setUpCommunicationMaps(EngngModel*)':
feticommunicator.C:(.text+0x61a): undefined reference to `ompi_mpi_comm_world'
feticommunicator.C:(.text+0x6f0): undefined reference to `ompi_mpi_int'
feticommunicator.C:(.text+0x7a1): undefined reference to `ompi_mpi_comm_world'
feticommunicator.C:(.text+0x8ad): undefined reference to `ompi_mpi_int'
feticommunicator.C:(.text+0xef1): undefined reference to `ompi_mpi_comm_world'
feticommunicator.C:(.text+0x107b): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(engngm.o): In function `EngngModel::initParallel()':
engngm.C:(.text+0x151): undefined reference to `ompi_mpi_comm_world'
engngm.C:(.text+0x162): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(flotarry.o): In function `FloatArray::givePackSize(CommunicationBuffer&) const':
flotarry.C:(.text+0x265): undefined reference to `ompi_mpi_int'
flotarry.C:(.text+0x27d): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(flotmtrx.o): In function `FloatMatrix::givePackSize(CommunicationBuffer&)':
flotmtrx.C:(.text+0x17a): undefined reference to `ompi_mpi_int'
flotmtrx.C:(.text+0x192): undefined reference to `ompi_mpi_int'
flotmtrx.C:(.text+0x1af): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(intarray.o): In function `IntArray::givePackSize(CommunicationBuffer&)':
intarray.C:(.text+0x275): undefined reference to `ompi_mpi_int'
intarray.C:(.text+0x28d): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(calmls.o): In function `CylindricalALM::computeDeltaLambda(double&, FloatArray&, FloatArray&, FloatArray&, FloatArray&, double, double, double, double, int)':
calmls.C:(.text+0x1270): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x1276): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x128c): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x13eb): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x13f1): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x1407): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x1610): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x1616): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x161b): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x18c6): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x18cc): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x18d1): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x19f0): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x19f6): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x1a04): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(calmls.o): In function `CylindricalALM::solve(SparseMtrx*, FloatArray*, FloatArray*, FloatArray*, FloatArray*, FloatArray*, FloatArray*, double&, double, SparseNonLinearSystemNM::referenceLoadInputModeType, int&, TimeStep*)':
calmls.C:(.text+0x20bc): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x20c2): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x20c7): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x219b): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x21a1): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x21a6): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x2244): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x224a): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x224f): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x2326): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x232c): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x2331): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x271a): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x2720): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x2725): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x288a): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x2890): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x2895): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x2a6d): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x2a73): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x2a78): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x2b74): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x2b7a): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x2b7f): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x2fc7): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x2fcd): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x2fd2): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x3141): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x3147): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x314c): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x347f): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x3485): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x348a): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x376d): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x3773): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x3780): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x382b): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x3831): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x3847): undefined reference to `ompi_mpi_double'
calmls.C:(.text+0x3a05): undefined reference to `ompi_mpi_comm_world'
calmls.C:(.text+0x3a0b): undefined reference to `ompi_mpi_op_sum'
calmls.C:(.text+0x3a10): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(nrsolver.o): In function `NRSolver::solve(SparseMtrx*, FloatArray*, FloatArray*, FloatArray*, FloatArray*, FloatArray*, FloatArray*, double&, double, SparseNonLinearSystemNM::referenceLoadInputModeType, int&, TimeStep*)':
nrsolver.C:(.text+0x12d0): undefined reference to `ompi_mpi_comm_world'
nrsolver.C:(.text+0x12d6): undefined reference to `ompi_mpi_op_sum'
nrsolver.C:(.text+0x12db): undefined reference to `ompi_mpi_double'
nrsolver.C:(.text+0x154a): undefined reference to `ompi_mpi_comm_world'
nrsolver.C:(.text+0x1550): undefined reference to `ompi_mpi_op_sum'
nrsolver.C:(.text+0x1555): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petscsparsemtrx.o): In function `PetscSparseMtrx::printYourself() const':
petscsparsemtrx.C:(.text+0x215): undefined reference to `ompi_mpi_comm_self'
petscsparsemtrx.C:(.text+0x21a): undefined reference to `PETSC_VIEWER_STDOUT_(ompi_communicator_t*)'
petscsparsemtrx.C:(.text+0x229): undefined reference to `ompi_mpi_comm_self'
petscsparsemtrx.C:(.text+0x22e): undefined reference to `PETSC_VIEWER_STDOUT_(ompi_communicator_t*)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petscsparsemtrx.o): In function `PetscSparseMtrx::printStatistics() const':
petscsparsemtrx.C:(.text+0x245): undefined reference to `ompi_mpi_comm_self'
petscsparsemtrx.C:(.text+0x24a): undefined reference to `PETSC_VIEWER_STDOUT_(ompi_communicator_t*)'
petscsparsemtrx.C:(.text+0x25c): undefined reference to `ompi_mpi_comm_self'
petscsparsemtrx.C:(.text+0x261): undefined reference to `PETSC_VIEWER_STDOUT_(ompi_communicator_t*)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petscsparsemtrx.o): In function `PetscSparseMtrx::buildInternalStructure(EngngModel*, int, EquationID)':
petscsparsemtrx.C:(.text+0x9e6): undefined reference to `MatCreate(ompi_communicator_t*, _p_Mat**)'
petscsparsemtrx.C:(.text+0xda7): undefined reference to `MatCreate(ompi_communicator_t*, _p_Mat**)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petscsolver.o): In function `PetscSolver::petsc_solve(PetscSparseMtrx*, _p_Vec*, _p_Vec*)':
petscsolver.C:(.text+0x102): undefined reference to `KSPCreate(ompi_communicator_t*, _p_KSP**)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petsccontext.o): In function `PetscContext::scatterN2G(_p_Vec*, _p_Vec*, InsertMode)':
petsccontext.C:(.text+0x192): undefined reference to `ISCreateGeneral(ompi_communicator_t*, int, int const*, _p_IS**)'
petsccontext.C:(.text+0x1ad): undefined reference to `ISCreateStride(ompi_communicator_t*, int, int, int, _p_IS**)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petsccontext.o): In function `PetscContext::scatterG2N(_p_Vec*, _p_Vec*, InsertMode)':
petsccontext.C:(.text+0x283): undefined reference to `ISCreateGeneral(ompi_communicator_t*, int, int const*, _p_IS**)'
petsccontext.C:(.text+0x29e): undefined reference to `ISCreateStride(ompi_communicator_t*, int, int, int, _p_IS**)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petsccontext.o): In function `PetscContext::createVecGlobal(_p_Vec**)':
petsccontext.C:(.text+0x3a0): undefined reference to `VecCreate(ompi_communicator_t*, _p_Vec**)'
petsccontext.C:(.text+0x3f9): undefined reference to `ompi_mpi_comm_self'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petsccontext.o): In function `PetscContext::scatterN2G(FloatArray*, _p_Vec*, InsertMode)':
petsccontext.C:(.text+0x5d5): undefined reference to `ompi_mpi_comm_self'
petsccontext.C:(.text+0x5da): undefined reference to `VecCreateSeq(ompi_communicator_t*, int, _p_Vec**)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petsccontext.o): In function `PetscContext::scatterG2N(_p_Vec*, FloatArray*, InsertMode)':
petsccontext.C:(.text+0x694): undefined reference to `ompi_mpi_comm_self'
petsccontext.C:(.text+0x69b): undefined reference to `VecCreateSeq(ompi_communicator_t*, int, _p_Vec**)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petsccontext.o): In function `PetscContext::createVecGlobal(_p_Vec**)':
petsccontext.C:(.text+0x402): undefined reference to `VecCreateSeq(ompi_communicator_t*, int, _p_Vec**)'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(processcomm.o): In function `ProcessCommunicatorBuff::ProcessCommunicatorBuff(CommBuffType)':
processcomm.C:(.text+0x1c2): undefined reference to `ompi_mpi_comm_world'
processcomm.C:(.text+0x1fc): undefined reference to `ompi_mpi_comm_world'
processcomm.C:(.text+0x238): undefined reference to `ompi_mpi_comm_world'
processcomm.C:(.text+0x258): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(processcomm.o): In function `ProcessCommunicatorBuff::ProcessCommunicatorBuff(CommBuffType)':
processcomm.C:(.text+0x2c2): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(processcomm.o):processcomm.C:(.text+0x2fc): more undefined references to `ompi_mpi_comm_world' follow
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(problemcomm.o): In function `ProblemCommunicator::setUpCommunicationMapsForRemoteElementMode(EngngModel*, bool)':
problemcomm.C:(.text+0xc15): undefined reference to `ompi_mpi_op_max'
problemcomm.C:(.text+0xc22): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(problemcomm.o): In function `ProblemCommunicator::setUpCommunicationMapsForElementCut(EngngModel*, bool)':
problemcomm.C:(.text+0x12be): undefined reference to `ompi_mpi_comm_world'
problemcomm.C:(.text+0x1330): undefined reference to `ompi_mpi_comm_world'
problemcomm.C:(.text+0x1336): undefined reference to `ompi_mpi_op_max'
problemcomm.C:(.text+0x1343): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::MPIBuffer(bool)':
combuff.C:(.text+0x25): undefined reference to `ompi_request_null'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::MPIBuffer(bool)':
combuff.C:(.text+0x55): undefined reference to `ompi_request_null'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::init()':
combuff.C:(.text+0x6b): undefined reference to `ompi_request_null'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::bcast(ompi_communicator_t*, int)':
combuff.C:(.text+0x160): undefined reference to `ompi_mpi_packed'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::iSend(ompi_communicator_t*, int, int)':
combuff.C:(.text+0x18f): undefined reference to `ompi_mpi_packed'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::iRecv(ompi_communicator_t*, int, int, int)':
combuff.C:(.text+0x39d): undefined reference to `ompi_mpi_packed'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::MPIBuffer(int, bool)':
combuff.C:(.text+0x4d4): undefined reference to `ompi_request_null'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(combuff.o): In function `MPIBuffer::MPIBuffer(int, bool)':
combuff.C:(.text+0x504): undefined reference to `ompi_request_null'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(petscordering.o): In function `PetscNatural2GlobalOrdering::init(EngngModel*, EquationID, int, ApplicationOrdering::EquationType)':
petscordering.C:(.text+0x574): undefined reference to `ompi_mpi_int'
petscordering.C:(.text+0x592): undefined reference to `ompi_mpi_comm_world'
petscordering.C:(.text+0xa50): undefined reference to `ompi_mpi_comm_world'
petscordering.C:(.text+0xa85): undefined reference to `ompi_mpi_int'
petscordering.C:(.text+0xd1d): undefined reference to `ompi_mpi_comm_world'
petscordering.C:(.text+0xd52): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::unpackHeader(ompi_communicator_t*)':
dyncombuff.C:(.text+0x6): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::packHeader(ompi_communicator_t*)':
dyncombuff.C:(.text+0x69): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::iSend(ompi_communicator_t*, int, int)':
dyncombuff.C:(.text+0xee): undefined reference to `ompi_mpi_packed'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::iRecv(ompi_communicator_t*, int, int, int)':
dyncombuff.C:(.text+0x39d): undefined reference to `ompi_mpi_packed'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::init(ompi_communicator_t*)':
dyncombuff.C:(.text+0x425): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::CommunicationPacket(ompi_communicator_t*, int)':
dyncombuff.C:(.text+0x4cf): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::CommunicationPacket(ompi_communicator_t*, int)':
dyncombuff.C:(.text+0x6df): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::CommunicationPacket(ompi_communicator_t*, int, int)':
dyncombuff.C:(.text+0x76a): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `CommunicationPacket::CommunicationPacket(ompi_communicator_t*, int, int)':
dyncombuff.C:(.text+0x87a): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::unpackArray(char*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPci[DynamicCommunicationBuffer::unpackArray(char*, int)]+0x4b): undefined reference to `ompi_mpi_char'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPci[DynamicCommunicationBuffer::unpackArray(char*, int)]+0x7a): undefined reference to `ompi_mpi_char'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::unpackArray(double*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPdi[DynamicCommunicationBuffer::unpackArray(double*, int)]+0x4b): undefined reference to `ompi_mpi_double'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPdi[DynamicCommunicationBuffer::unpackArray(double*, int)]+0x7a): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::unpackArray(unsigned long*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPmi[DynamicCommunicationBuffer::unpackArray(unsigned long*, int)]+0x4b): undefined reference to `ompi_mpi_unsigned_long'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPmi[DynamicCommunicationBuffer::unpackArray(unsigned long*, int)]+0x7a): undefined reference to `ompi_mpi_unsigned_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::unpackArray(long*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPli[DynamicCommunicationBuffer::unpackArray(long*, int)]+0x4b): undefined reference to `ompi_mpi_long'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPli[DynamicCommunicationBuffer::unpackArray(long*, int)]+0x7a): undefined reference to `ompi_mpi_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::unpackArray(int*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPii[DynamicCommunicationBuffer::unpackArray(int*, int)]+0x4b): undefined reference to `ompi_mpi_int'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer11unpackArrayEPii[DynamicCommunicationBuffer::unpackArray(int*, int)]+0x79): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::packArray(double const*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKdi[DynamicCommunicationBuffer::packArray(double const*, int)]+0x79): undefined reference to `ompi_mpi_double'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKdi[DynamicCommunicationBuffer::packArray(double const*, int)]+0xa7): undefined reference to `ompi_mpi_double'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::packArray(long const*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKli[DynamicCommunicationBuffer::packArray(long const*, int)]+0x79): undefined reference to `ompi_mpi_long'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKli[DynamicCommunicationBuffer::packArray(long const*, int)]+0xa7): undefined reference to `ompi_mpi_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::packArray(int const*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKii[DynamicCommunicationBuffer::packArray(int const*, int)]+0x79): undefined reference to `ompi_mpi_int'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKii[DynamicCommunicationBuffer::packArray(int const*, int)]+0xa7): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::packArray(unsigned long const*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKmi[DynamicCommunicationBuffer::packArray(unsigned long const*, int)]+0x79): undefined reference to `ompi_mpi_unsigned_long'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKmi[DynamicCommunicationBuffer::packArray(unsigned long const*, int)]+0xa7): undefined reference to `ompi_mpi_unsigned_long'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(dyncombuff.o): In function `DynamicCommunicationBuffer::packArray(char const*, int)':
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKci[DynamicCommunicationBuffer::packArray(char const*, int)]+0x79): undefined reference to `ompi_mpi_char'
dyncombuff.C:(.text._ZN26DynamicCommunicationBuffer9packArrayEPKci[DynamicCommunicationBuffer::packArray(char const*, int)]+0xa7): undefined reference to `ompi_mpi_char'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(loadbalancer.o): In function `WallClockLoadBalancerMonitor::decide(TimeStep*)':
loadbalancer.C:(.text+0x1082): undefined reference to `ompi_mpi_double'
loadbalancer.C:(.text+0x10cd): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(loadbalancer.o): In function `int Communicator::unpackAllData<LoadBalancer, Domain>(LoadBalancer*, Domain*, int (LoadBalancer::*)(Domain*, ProcessCommunicator&))':
loadbalancer.C:(.text._ZN12Communicator13unpackAllDataI12LoadBalancer6DomainEEiPT_PT0_MS3_FiS6_R19ProcessCommunicatorE[int Communicator::unpackAllData<LoadBalancer, Domain>(LoadBalancer*, Domain*, int (LoadBalancer::*)(Domain*, ProcessCommunicator&))]+0x204): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(parmetisloadbalancer.o): In function `ParmetisLoadBalancer::initGlobalParmetisElementNumbering()':
parmetisloadbalancer.C:(.text+0x1f5): undefined reference to `ompi_mpi_int'
parmetisloadbalancer.C:(.text+0x210): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(parmetisloadbalancer.o): In function `ParmetisLoadBalancer::calculateLoadTransfer()':
parmetisloadbalancer.C:(.text+0xaf2): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(parmetisloadbalancer.o): In function `int Communicator::unpackAllData<ParmetisLoadBalancer>(ParmetisLoadBalancer*, int (ParmetisLoadBalancer::*)(ProcessCommunicator&))':
parmetisloadbalancer.C:(.text._ZN12Communicator13unpackAllDataI20ParmetisLoadBalancerEEiPT_MS2_FiR19ProcessCommunicatorE[int Communicator::unpackAllData<ParmetisLoadBalancer>(ParmetisLoadBalancer*, int (ParmetisLoadBalancer::*)(ProcessCommunicator&))]+0x1f9): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(nonlocalmatwtp.o): In function `NonlocalMaterialWTP::migrate()':
nonlocalmatwtp.C:(.text+0x121b): undefined reference to `ompi_mpi_comm_world'
nonlocalmatwtp.C:(.text+0x1463): undefined reference to `ompi_mpi_comm_world'
nonlocalmatwtp.C:(.text+0x1469): undefined reference to `ompi_mpi_op_max'
nonlocalmatwtp.C:(.text+0x1476): undefined reference to `ompi_mpi_int'
nonlocalmatwtp.C:(.text+0x14af): undefined reference to `ompi_mpi_int'
/home/liuxueji/oofem-1.8/targets/poofem-release/lib/oofemlib/oofemlib.a(nonlocalmatwtp.o): In function `int Communicator::unpackAllData<NonlocalMaterialWTP, Domain>(NonlocalMaterialWTP*, Domain*, int (NonlocalMaterialWTP::*)(Domain*, ProcessCommunicator&))':
nonlocalmatwtp.C:(.text._ZN12Communicator13unpackAllDataI19NonlocalMaterialWTP6DomainEEiPT_PT0_MS3_FiS6_R19ProcessCommunicatorE[int Communicator::unpackAllData<NonlocalMaterialWTP, Domain>(NonlocalMaterialWTP*, Domain*, int (NonlocalMaterialWTP::*)(Domain*, ProcessCommunicator&))]+0x202): undefined reference to `ompi_mpi_comm_world'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_real'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_byte'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_dblprec'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_float_int'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_lor'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_bxor'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_errcode_intern_lastused'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_cxx_bool'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_errhandler_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_cxx_ldblcplex'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_2real'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_logic'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_cxx_cplex'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_win_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_lxor'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_min'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_land'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_unsigned'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_registered_datareps'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_long_int'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_float'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_errhandler_create'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_long_double'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_short'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_character'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_integer'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_cplex'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_bor'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_long_long_int'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_2integer'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_unsigned_char'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_errhandler_invoke'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_2int'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_short_int'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_errcodes_intern'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_minloc'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_cxx_dblcplex'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_file_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_ub'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_errors_throw_exceptions'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_2dblprec'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_prod'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_maxloc'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_band'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_unsigned_short'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `MPI_File_close'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_wchar'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `MPI_Register_datarep'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_signed_char'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_lb'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_double_int'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_errors_are_fatal'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_replace'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_datatype_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_info_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_op_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_group_null'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_group_empty'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_errors_return'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_mpi_unsigned_long_long'
/home/liuxueji/oofem-1.8/openmpi-1.3/lib/libmpi_cxx.so: undefined reference to `ompi_attr_create_keyval'
collect2: ld returned 1 exit status
make[1]: *** [/home/liuxueji/oofem-1.8/targets/poofem-release/bin/oofeg] Error 1
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/main'
make: *** [all] Error 2

10

Re: Open MPI library

Hi Xuejian,
from the output you have provided I see at least two issues:
- the compiler used (mpicxx) comes from openmpi-1.3. This means that oofem was configured to use openmpi. On the other hand, the libraries listed in linker include also LAM/MPI libraries - seems that oofem was configured to use and compile with openmpi, while petsc using LAM MPI. This will not work, one should use only one MPI library in all packages that are linked together.
If you compiled PETSc by yourself, you have to configure to use openmpi. Seems that the configuration found another mpi implementation (LAM MPI) that was used to compile PETSc.

- Besides, the oofem looks like not to be configured correctly, since both main.o oofeg.o are inlcuded. You should omit --enable-oofeg when configuring poofem targets.

Borek

Re: Open MPI library

Hi Borek,

My installtion is described like the following.

o) Install MPI
./config --prefix=/home/liuxueji/oofem-1.8/openmpi-1.3
make all install

o) Install PETSc

setenv PETSC_DIR /home/liuxueji/oofem-1.8/petsc-2.3.3-p15
./config/configure.py (I also tried with --with-mpi-dir=/home/liuxueji/oofem-1.8/openmpi-1.3 or setenv LD_LIBRARY_PATH  /home/liuxueji/oofem-1.8/openmpi-1.3/bin)
make all test

The test ran successfully.
o) Install ParMetis3.1.1
make
o) I go to home/liuxueji/oofem-1.8,
./configure OOFEM_TARGET=poofem-release CXXFLAGS=-O2 --enable-iml --enable-poofem --with-MPIDIR=/home/liuxueji/oofem-1.8/openmpi-1.3 --enable-petsc --with-PETSCDIR=/home/liuxueji/oofem-1.8/petsc-2.3.3-p15 --with-PETSCARCH=linux-gnu-cxx-debug --enable-dss --enable-parmetis --with-PARMETISDIR=/home/liuxueji/oofem-1.8/ParMetis-3.1.1
Go to /targets/poofem-release. Make.
The same error occurs as my last post, which is really out of my mind now. Please help.

Xuejian

Re: Open MPI library

I also have another question, Borek,

The compiler used mpicc for ParMetis is also from open mpi? I specified the INCDIR = /home/liuxueji/oofem-1.8/openmpi-1.3/bin, where mpicc is. Is this okay? I do not know what changes needed for makefile.in. I got the message with error.

/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(mesh.o): In function `ParMETIS_V3_Mesh2Dual':
mesh.c:(.text+0x489): undefined reference to `lam_mpi_int'
mesh.c:(.text+0x5f0): undefined reference to `lam_mpi_int'
mesh.c:(.text+0x61f): undefined reference to `lam_mpi_int'
mesh.c:(.text+0x943): undefined reference to `lam_mpi_int'
mesh.c:(.text+0xc79): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(mesh.o):mesh.c:(.text+0xc82): more undefined references to `lam_mpi_int' follow
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(weird.o): In function `CheckInputs__':
weird.c:(.text+0x2c7): undefined reference to `lam_mpi_max'
weird.c:(.text+0x2cc): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(weird.o): In function `PartitionSmallGraph__':
weird.c:(.text+0x7e1): undefined reference to `lam_mpi_minloc'
weird.c:(.text+0x7e6): undefined reference to `lam_mpi_2int'
weird.c:(.text+0x8b8): undefined reference to `lam_mpi_int'
weird.c:(.text+0x8d8): undefined reference to `lam_mpi_int'
weird.c:(.text+0x9ba): undefined reference to `lam_mpi_sum'
weird.c:(.text+0x9cd): undefined reference to `lam_mpi_float'
weird.c:(.text+0xa3b): undefined reference to `lam_mpi_int'
weird.c:(.text+0xa72): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `GlobalSESumFloat__':
comm.c:(.text+0xd): undefined reference to `lam_mpi_sum'
comm.c:(.text+0x1c): undefined reference to `lam_mpi_float'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `GlobalSEMinFloat__':
comm.c:(.text+0x4d): undefined reference to `lam_mpi_min'
comm.c:(.text+0x5c): undefined reference to `lam_mpi_float'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `GlobalSEMaxFloat__':
comm.c:(.text+0x8d): undefined reference to `lam_mpi_max'
comm.c:(.text+0x9c): undefined reference to `lam_mpi_float'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `GlobalSESum__':
comm.c:(.text+0xcd): undefined reference to `lam_mpi_sum'
comm.c:(.text+0xe0): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `GlobalSEMin__':
comm.c:(.text+0x10d): undefined reference to `lam_mpi_min'
comm.c:(.text+0x120): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `GlobalSEMaxDouble__':
comm.c:(.text+0x14d): undefined reference to `lam_mpi_max'
comm.c:(.text+0x15a): undefined reference to `lam_mpi_double'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `GlobalSEMax__':
comm.c:(.text+0x18d): undefined reference to `lam_mpi_max'
comm.c:(.text+0x1a0): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o): In function `CommChangedInterfaceData__':
comm.c:(.text+0x277): undefined reference to `lam_mpi_int'
comm.c:(.text+0x2d9): undefined reference to `lam_mpi_int'
comm.c:(.text+0x314): undefined reference to `lam_mpi_int'
comm.c:(.text+0x49e): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(comm.o):comm.c:(.text+0x575): more undefined references to `lam_mpi_int' follow
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(grsetup.o): In function `ComputeMoveStatistics__':
grsetup.c:(.text+0x2a9): undefined reference to `lam_mpi_sum'
grsetup.c:(.text+0x2ae): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(timer.o): In function `PrintTimer__':
timer.c:(.text+0xcf): undefined reference to `lam_mpi_sum'
timer.c:(.text+0xd4): undefined reference to `lam_mpi_double'
timer.c:(.text+0xff): undefined reference to `lam_mpi_max'
timer.c:(.text+0x104): undefined reference to `lam_mpi_double'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(timer.o): In function `PrintTimingInfo__':
timer.c:(.text+0x197): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x19c): undefined reference to `lam_mpi_double'
timer.c:(.text+0x1f4): undefined reference to `lam_mpi_max'
timer.c:(.text+0x1f9): undefined reference to `lam_mpi_double'
timer.c:(.text+0x245): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x24a): undefined reference to `lam_mpi_double'
timer.c:(.text+0x273): undefined reference to `lam_mpi_max'
timer.c:(.text+0x278): undefined reference to `lam_mpi_double'
timer.c:(.text+0x2c9): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x2ce): undefined reference to `lam_mpi_double'
timer.c:(.text+0x2f7): undefined reference to `lam_mpi_max'
timer.c:(.text+0x2fc): undefined reference to `lam_mpi_double'
timer.c:(.text+0x34d): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x352): undefined reference to `lam_mpi_double'
timer.c:(.text+0x37b): undefined reference to `lam_mpi_max'
timer.c:(.text+0x380): undefined reference to `lam_mpi_double'
timer.c:(.text+0x3d1): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x3d6): undefined reference to `lam_mpi_double'
timer.c:(.text+0x3ff): undefined reference to `lam_mpi_max'
timer.c:(.text+0x404): undefined reference to `lam_mpi_double'
timer.c:(.text+0x455): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x45a): undefined reference to `lam_mpi_double'
timer.c:(.text+0x483): undefined reference to `lam_mpi_max'
timer.c:(.text+0x488): undefined reference to `lam_mpi_double'
timer.c:(.text+0x4d9): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x4de): undefined reference to `lam_mpi_double'
timer.c:(.text+0x507): undefined reference to `lam_mpi_max'
timer.c:(.text+0x50c): undefined reference to `lam_mpi_double'
timer.c:(.text+0x55d): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x562): undefined reference to `lam_mpi_double'
timer.c:(.text+0x58b): undefined reference to `lam_mpi_max'
timer.c:(.text+0x590): undefined reference to `lam_mpi_double'
timer.c:(.text+0x5e1): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x5e6): undefined reference to `lam_mpi_double'
timer.c:(.text+0x60f): undefined reference to `lam_mpi_max'
timer.c:(.text+0x614): undefined reference to `lam_mpi_double'
timer.c:(.text+0x667): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x66c): undefined reference to `lam_mpi_double'
timer.c:(.text+0x695): undefined reference to `lam_mpi_max'
timer.c:(.text+0x69a): undefined reference to `lam_mpi_double'
timer.c:(.text+0x6ed): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x6f2): undefined reference to `lam_mpi_double'
timer.c:(.text+0x71b): undefined reference to `lam_mpi_max'
timer.c:(.text+0x720): undefined reference to `lam_mpi_double'
timer.c:(.text+0x773): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x778): undefined reference to `lam_mpi_double'
timer.c:(.text+0x7a1): undefined reference to `lam_mpi_max'
timer.c:(.text+0x7a6): undefined reference to `lam_mpi_double'
timer.c:(.text+0x7f9): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x7fe): undefined reference to `lam_mpi_double'
timer.c:(.text+0x82a): undefined reference to `lam_mpi_max'
timer.c:(.text+0x82f): undefined reference to `lam_mpi_double'
timer.c:(.text+0x87d): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x882): undefined reference to `lam_mpi_double'
timer.c:(.text+0x8ae): undefined reference to `lam_mpi_max'
timer.c:(.text+0x8b3): undefined reference to `lam_mpi_double'
timer.c:(.text+0x901): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x906): undefined reference to `lam_mpi_double'
timer.c:(.text+0x92e): undefined reference to `lam_mpi_double'
timer.c:(.text+0x934): undefined reference to `lam_mpi_max'
timer.c:(.text+0x985): undefined reference to `lam_mpi_sum'
timer.c:(.text+0x98a): undefined reference to `lam_mpi_double'
timer.c:(.text+0x9b8): undefined reference to `lam_mpi_max'
timer.c:(.text+0x9bd): undefined reference to `lam_mpi_double'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(initpart.o): In function `Mc_InitPartition_RB__':
initpart.c:(.text+0x87b): undefined reference to `lam_mpi_sum'
initpart.c:(.text+0x884): undefined reference to `lam_mpi_int'
initpart.c:(.text+0x960): undefined reference to `lam_mpi_max'
initpart.c:(.text+0x965): undefined reference to `lam_mpi_int'
initpart.c:(.text+0x993): undefined reference to `lam_mpi_min'
initpart.c:(.text+0x998): undefined reference to `lam_mpi_float'
initpart.c:(.text+0xa08): undefined reference to `lam_mpi_minloc'
initpart.c:(.text+0xa0d): undefined reference to `lam_mpi_float_int'
initpart.c:(.text+0xa2b): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(match.o): In function `Mc_Global_CreateCoarseGraph__':
match.c:(.text+0x19a): undefined reference to `lam_mpi_int'
match.c:(.text+0x6d4): undefined reference to `lam_mpi_int'
match.c:(.text+0x738): undefined reference to `lam_mpi_int'
match.c:(.text+0x8f9): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(match.o):match.c:(.text+0xb3a): more undefined references to `lam_mpi_int' follow
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(kwayrefine.o): In function `Mc_ComputePartitionParams__':
kwayrefine.c:(.text+0x2e0): undefined reference to `lam_mpi_sum'
kwayrefine.c:(.text+0x2ea): undefined reference to `lam_mpi_float'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(kwayrefine.o): In function `Mc_ProjectPartition__':
kwayrefine.c:(.text+0x688): undefined reference to `lam_mpi_int'
kwayrefine.c:(.text+0x72c): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(kwaybalance.o): In function `Mc_KWayBalance__':
kwaybalance.c:(.text+0x4f9): undefined reference to `lam_mpi_int'
kwaybalance.c:(.text+0x833): undefined reference to `lam_mpi_int'
kwaybalance.c:(.text+0x93a): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(kwaybalance.o):kwaybalance.c:(.text+0x999): more undefined references to `lam_mpi_int' follow
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(kwaybalance.o): In function `Mc_KWayBalance__':
kwaybalance.c:(.text+0xc79): undefined reference to `lam_mpi_sum'
kwaybalance.c:(.text+0xc8e): undefined reference to `lam_mpi_float'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(remap.o): In function `ParallelTotalVReMap__':
remap.c:(.text+0x1a4): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(stat.o): In function `Mc_ComputeRefineStats__':
stat.c:(.text+0x262): undefined reference to `lam_mpi_max'
stat.c:(.text+0x267): undefined reference to `lam_mpi_int'
stat.c:(.text+0x2e5): undefined reference to `lam_mpi_sum'
stat.c:(.text+0x2ea): undefined reference to `lam_mpi_int'
stat.c:(.text+0x51d): undefined reference to `lam_mpi_sum'
stat.c:(.text+0x522): undefined reference to `lam_mpi_int'
stat.c:(.text+0x552): undefined reference to `lam_mpi_int'
stat.c:(.text+0x558): undefined reference to `lam_mpi_sum'
stat.c:(.text+0x5fb): undefined reference to `lam_mpi_sum'
stat.c:(.text+0x600): undefined reference to `lam_mpi_int'
/home/liuxueji/oofem-1.8/ParMetis-3.1.1/libparmetis.a(stat.o): In function `Mc_ComputeParallelBalance__':
stat.c:(.text+0xdae): undefined reference to `lam_mpi_sum'
stat.c:(.text+0xdb3): undefined reference to `lam_mpi_float'
stat.c:(.text+0xdd2): undefined reference to `lam_mpi_min'
stat.c:(.text+0xde6): undefined reference to `lam_mpi_float'
collect2: ld returned 1 exit status
make[1]: *** [/home/liuxueji/oofem-1.8/targets/poofem-release/bin/poofem] Error 1
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/targets/poofem-release/lib/main'
make: *** [all] Error 2

It seems the error is related to the compiler again. I am really confused.

Xuejian

13

Re: Open MPI library

Hi Xuejian,

the problem with Parmetis seems to be the same as with OOFEM. From output you have provided: missing "lam_mpi_int" means, that the compiler used comes from LAM/MPI or header files are used from LAM/MPI. When compiling Parmetis (as well as OOFEM) the compiler used should be the one that comes with opemmpi. So you should modify Parmetis configuration (makefile) to use the mpic++ compiler and provide correct path to openmpi installation directory, where includes and libraries are.

Still seems to me, that in certain step, LAM/MPI comes into the play.
The optional workaround would consists in deleting your openmpi installation and use LAM/MPI installation, that is probably already available on your system.

Borek

Re: Open MPI library

Hi Borek,

Yeah, the system I am using has already installed LAM/MPI and I am using it. But the interesting is the in LAM/MPI, there is not mpicxx compiler, it does have mpic++. When I am configuring oofem, the compiler can not be found. Please advise me on it. Thanks,

Xuejian

Re: Open MPI library

The following is the error I got when I was compiling ParMetis with openMPI mpicxx

(cd METISLib ; make )
make[1]: Entering directory `/home/liuxueji/oofem-1.8/ParMetis-3.1.1/METISLib'
/home/liuxueji/oofem-1.8/openmpi-1.3/bin/mpicxx -DNDEBUG -O3  -I. -I/home/liuxueji/oofem-1.8/openmpi-1.3/include -c coarsen.c
/home/liuxueji/oofem-1.8/openmpi-1.3/bin/mpicxx -DNDEBUG -O3  -I. -I/home/liuxueji/oofem-1.8/openmpi-1.3/include -c fm.c
fm.c: In function 'void __FM_2WayEdgeRefine(CtrlType*, GraphType*, int*, int)':
fm.c:44: warning: converting to 'int' from 'double'
/home/liuxueji/oofem-1.8/openmpi-1.3/bin/mpicxx -DNDEBUG -O3  -I. -I/home/liuxueji/oofem-1.8/openmpi-1.3/include -c initpart.c
initpart.c: In function 'void __GrowBisection(CtrlType*, GraphType*, int*, float)':
initpart.c:107: warning: converting to 'int' from 'float'
initpart.c:108: warning: converting to 'int' from 'float'
initpart.c:109: warning: converting to 'int' from 'double'
initpart.c:110: warning: converting to 'int' from 'double'
initpart.c:202: error: '__GKfree' was not declared in this scope
initpart.c: In function 'void __GrowBisectionNode(CtrlType*, GraphType*, float)':
initpart.c:233: warning: converting to 'int' from 'float'
initpart.c:234: warning: converting to 'int' from 'float'
initpart.c:235: warning: converting to 'int' from 'double'
initpart.c:236: warning: converting to 'int' from 'double'
initpart.c:340: error: '__GKfree' was not declared in this scope
initpart.c: In function 'void __RandomBisection(CtrlType*, GraphType*, int*, float)':
initpart.c:369: warning: converting to 'int' from 'float'
initpart.c:370: warning: converting to 'int' from 'float'
initpart.c:371: warning: converting to 'int' from 'double'
initpart.c:372: warning: converting to 'int' from 'double'
initpart.c:420: error: '__GKfree' was not declared in this scope
make[1]: *** [initpart.o] Error 1
make[1]: Leaving directory `/home/liuxueji/oofem-1.8/ParMetis-3.1.1/METISLib'
make: *** [default] Error 2

Re: Open MPI library

Hi Borek,

Good news! Finally I got it through. I installed openmpi1.3, petsc-2.3.3-p15 and ParMetis-3.1.1 (compiled with mpicc).

Thank you for your continuous help. I wish to get your help when I am using your oofem program.

Xuejian

17

Re: Open MPI library

Hi Xuejian,

I am sorry for not replying for some time, I have been on vacations. Anyway, congratulation to successful installation. I hope that you will enjoy. 
And of course, I ready to help when needed.

With best regards,
Borek

Re: Open MPI library

Good work on that... It helps me also