Topic: compile with SLEPc under windows

Hi all,
I compiled PETSc and SLECP for Windows, and trying now to compiled oofem with them in vs2013.
All fine with petsc 3.5.2, but for slepc 3.5.2 I obtain the following errors in vs:

Errore    40    error C2027: utilizzo di tipo non definito 'oofem::Domain'    C:\oofem\src\oofemlib\slepcsolver.C    70    1    core
Errore    41    error C2227: l'elemento a sinistra di '->giveEngngModel' deve puntare a un tipo classe, struttura, unione o generico    C:\oofem\src\oofemlib\slepcsolver.C    70    1    core
Errore    42    error C2227: l'elemento a sinistra di '->giveOutputStream' deve puntare a un tipo classe, struttura, unione o generico    C:\oofem\src\oofemlib\slepcsolver.C    70    1    core
Errore    39    error C2664: 'PetscErrorCode EPSDestroy(EPS *)': impossibile convertire l'argomento 1 da 'EPS' a 'EPS *'    C:\oofem\src\oofemlib\slepcsolver.C    58    1    core
Errore    59    error C2664: 'PetscErrorCode VecDestroy(Vec *)': impossibile convertire l'argomento 1 da 'Vec' a 'Vec *'    C:\oofem\src\oofemlib\slepcsolver.C    205    1    core

sorry if they are in Italian, if this is not clear I can translate them.
Any clue?

Thanks,
Giovanni

Re: compile with SLEPc under windows

This is probably because a version change in SLEPC, and OOFEM hasn't upgraded to that version yet.
I remember some changes in PETSc that switches some function arguments from Vec* to Vec.
You can try if it is that simple. There isn't many function calls to SLEPC, so it shouldn't be that hard to get it working with a newer version.

Re: compile with SLEPc under windows

Yes, the last 2 errors can be corrected with & before the argument:

EPSDestroy(&eps);
...
ierr = VecDestroy(& Vr);

but the first three errors are not related to that. The first in particular said that the 'oofem::Domain' is a type not defined. This happens in the line:

outStream = domain->giveEngngModel()->giveOutputStream();

in SLEPcSolver :: solve, I don't know what happened here.

Re: compile with SLEPc under windows

Probably a missing include
Try adding

#include "domain.h"

in the beginning of slepcsolver.C

5 (edited by johnnyontheweb 14-11-2014 17:33:35)

Re: compile with SLEPc under windows

Thanks, it worked fine. I was not sure to add the reference to domain.C.
In attachment the patch for slepcsolver.c (compatibility with SLEPc ver.3.5.2 - SLEPc developers told me that there will be soon a 3.5.3 because there are still some other errors compiling in cygwin).

I compiled OOFEM for windows in its parallel version with Intel Fortran compiler (v.11 and/or 12) and VS2013 express with:
PETSc, SLEPc, ParMETIS, METIS, ScalaPack, MUMPS and MPICH2 unofficial version 1.4.1p1. I'll post some detailed instructions on my old post on Windows parallel version after tests.

Let me say that, unless it seems that nobody would use windows for MPI jobs, it is exactly my case (my department doesn't wanto to adopt linux, also for compatibility of other commerical software) and I think that also other university or institution have a lot pc configured with windows ready to be used for parallel jobs.

Thanks,
Giovanni

Post's attachments

slepcsolver.c.patch 984 b, 3 downloads since 2014-11-14 

You don't have the permssions to download the attachments of this post.

Re: compile with SLEPc under windows

We have all intention of supporting windows, but none of the developers are using it, and you are probably the first that properly sets up PETSc under windows.
Also, for SLEPC and PETSc, they make incompatible changes with almost every release, so we sometimes aren't keeping up.

PETSc is quite difficult to use under windows, and so I had plans on supporting shared memory solvers instead (which we are currently missing).
I believe there are a few options in PARDISO, and we should add support for those solvers in OOFEM.
This would probably be ideal for all users that are just running on a local machine (desktop PC).  MPI would only be necessary when shared memory is not possible, such as running on multiple *nodes* in a cluster.
But, even on clusters, a lot of simulations usually fits on a single node (in our cluster, that would still be 16 cores, and plenty of RAM), in which case there is no need to go through and decompose the domain for with MPI.

I know there is Pastix which works on shared memory in PETSc, but it still requires MPI to compile with PETSc.

Re: compile with SLEPc under windows

Also, for SLEPC and PETSc, they make incompatible changes with almost every release, so we sometimes aren't keeping up.

I think OOFEM misses a clear list of supported packages with their versions (or simply, I didn't find it): in this way you can state clearly which is the version of PETSc, SLEPc (and others...) to be used.

Re: compile with SLEPc under windows

You are perhaps hitting the two core problems with documentation (in general);
1. Finding what you need. There is just so much possible information to give, so not everything can be given a front row seat.

2. Keeping it correct/updated. I find it's better to say nothing than to say something incorrect.

Here, unfortunately, both of these have failed.
The README file contains the information, although it is not correct, or up to date. The README-file says PETSc 3.3 and SLEPC 3.3 or higher is required.
This isn't accurate anymore. I believe PETSc 3.4 and 3.5 will work, and of course, when a newer version of PETSc comes out, it might break again.
And clearly, SLEPC 3.5 didn't work without changes.

This should of course be updated, but the problem remains that 1. Developers doesn't remember everywhere we have written documentation (wiki, manuals, readmes, forums), and keeping it updated is therefore a big chore. 2. Whenever a problem occurs, it's rarely possible for users to find the relevant information, as they most of the time doesn't know what to even search for.

Re: compile with SLEPc under windows

I understand perfectly what you mean, I develop in a team and it's normal to have such situations.

If it helps I can write directly in the OOFEM Wiki the instruction to compile in Windows and maybe also create a "requirements" page if I had an author account.
I think it is important to have a lot of documentation, especially to cover different OSes.

Re: compile with SLEPc under windows

That would be great. Only Borek can approve the wiki accounts, so please ask him about that.

11 (edited by johnnyontheweb 18-11-2014 22:52:11)

Re: compile with SLEPc under windows

I sent an email to bp, thanks.

Regarding the tests with the parallel binary for windows, I encountered an error running:

C:\oofem\Debug>mpiexec -n 2 oofem -l 4 -p -f dyn_bar01.oofem.in
[0]PETSC ERROR: [1]PETSC ERROR: ------------------------------------------------
------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably m
emory access out of range
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgr
ind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
 find memory corruption errors
[1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
[1]PETSC ERROR: to get more information on the crash.
[1]PETSC ERROR: --------------------- Error Message ----------------------------
----------------------------------
[1]PETSC ERROR: Signal received
[1]PETSC ERROR: ----------------------------------------------------------------
--------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably m
emory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgr
ind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
 find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message ----------------------------
----------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trou
ble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014
[0]PETSC ERROR: C:\oofem\Debug\oofem.exe on a arch-mswin-c-opt named DIA-STC-063
 by 10228 Tue Nov 18 21:42:47 2014
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --download-fblaslapack --with-debugging=0 --with-mpi-
dir=/cygdrive/c/MPICH2 --with-shared-libraries=0 --with-mpiexec="mpiexec --local
only" --useThreads=0
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014
[1]PETSC ERROR: C:\oofem\Debug\oofem.exe on a arch-mswin-c-opt named DIA-STC-063
 by 10228 Tue Nov 18 21:42:47 2014
[1]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --download-fblaslapack --with-debugging=0 --with-mpi-
dir=/cygdrive/c/MPICH2 --with-shared-libraries=0 --with-mpiexec="mpiexec --local
only" --useThreads=0
[1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1

job aborted:
rank: node: exit code[: error message]
0: ...: 59: process 0 exited without calling finalize
1: ...: 59: process 1 exited without calling finalize

C:\oofem\Debug>oofem -l 4 -p -f dyn_bar01.oofem.in
____________________________________________________
           OOFEM - Finite Element Solver
        Copyright (C) 1994-2014 Borek Patzak
____________________________________________________
Reading all data from input file dyn_bar01.oofem.in.0
Total number of solution steps     3
Instanciating domain                1
Instanciated nodes & sides          2
Instanciated elements               1
Instanciated cross sections         1
Instanciated materials              1
Instanciated BCs                    2
Instanciated ICs                    0
Instanciated load-time fncts        1
Consistency check:  OK
_______________________________________________________
Error: (intarray.C:108)
In oofem::IntArray::checkBounds:
array error on index : 2 > 1
_______________________________________________________
No backtrace available
Total 1 error(s) and 0 warning(s) reported
oofem exit code 1

May I have to compile PETSc with "--useThreads=0" or the error is inside intarray.c (as outlined in the second command)?
Please note that I previously compiled OOFEM parallel without MUMPS and ScalaPack inside PETSc (see http://www.oofem.org/forum/viewtopic.ph … 5&p=2) and I never encountered this error performing this test. Do you have any clue?

Final question: why the input deck without ending ".0" or ".1" (in this case, "dyn_bar01.oofem.in") must be in the same folder?

Re: compile with SLEPc under windows

Having a PETSc version without threads might be easier to debug when bad things happen, but hopefully, it shouldn't be necessary.
Just knowing that, some code calling intarray outside it's bounds is very very hard to debug. Running this through a debugger should help tremendously when debugging.




Final question;
The file "dyn_bar01.oofem.in" is neither necessary, or used here. It's just a sequential version of the same problem, so that it is easier to verify that the results are correct when debugging parallel stuff.

Edit: I forgot about the error checking module. The decomposed files are pointing towards the "dyn_bar01.oofem.in" file so that I won't have to repeat the test data, so that's probably what you meant. For normal analysis, you wouldn't use the error checking module, and this wouldn't be necessary.

13 (edited by johnnyontheweb 18-11-2014 23:55:47)

Re: compile with SLEPc under windows

Thanks, I`ll give a try with no threads.

Final question: I get an error when running without that file, it seems oofem reads something from it. The error I receive is related to a missing file with  that name. Edit: ok,thanks.

Re: compile with SLEPc under windows

For the PETSc error, nothing to do with useThreads=0 and debugging activated:

C:\oofem\Debug>mpiexec -n 2 oofem -l 4 -p -f dyn_bar01.oofem.in
[1]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------
------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably m
emory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgr
ind[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
 find memory corruption errors
[0]PETSC ERROR: likely location of problem given in stack below
[0]PETSC ERROR: ---------------------  Stack Frames ----------------------------
--------
[0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[0]PETSC ERROR:       INSTEAD the line number of the start of the function
------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably m
emory access out of range
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgr
ind[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to
 find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: ---------------------  Stack Frames ----------------------------
--------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR:       INSTEAD the line number of the start of the function
[1]PETSC ERROR:       is given.
[1]PETSC ERROR: --------------------- Error Message ----------------------------
----------------------------------
[1]PETSC ERROR: Signal received
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trou
ble shooting.
[1]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014
[1]PETSC ERROR: C:\oofem\Debug\oofem.exe on a arch-mswin-c-debug named DIA-STC-0
63 by 10228 Wed Nov 19 12:13:02 2014
[0]PETSC ERROR:       is given.
[0]PETSC ERROR: --------------------- Error Message ----------------------------
----------------------------------
[0]PETSC ERROR: Signal received
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trou
ble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.2, Sep, 08, 2014
[0]PETSC ERROR: C:\oofem\Debug\oofem.exe on a arch-mswin-c-debug named DIA-STC-0
63 by 10228 Wed Nov 19 12:13:02 2014
[0]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --download-fblaslapack --with-debugging=1 --download-
mumps --with-mpi-dir=/cygdrive/c/MPICH2 --with-shared-libraries=0 --with-mpiexec
="mpiexec --localonly" --useThreads=0 --download-scalapack --with-parmetis-inclu
de=/home/10228/parmetis-4.0.3/include --with-parmetis-lib="/home/10228/libs/parm
etis.lib /home/10228/libs/metis.lib" --with-metis-include=/home/10228/metis-5.1.
0/include --with-metis-lib=/home/10228/libs/metis.lib --useThreads=0
[0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
[1]PETSC ERROR: Configure options --with-cc="win32fe cl" --with-fc="win32fe ifor
t" --with-cxx="win32fe cl" --download-fblaslapack --with-debugging=1 --download-
mumps --with-mpi-dir=/cygdrive/c/MPICH2 --with-shared-libraries=0 --with-mpiexec
="mpiexec --localonly" --useThreads=0 --download-scalapack --with-parmetis-inclu
de=/home/10228/parmetis-4.0.3/include --with-parmetis-lib="/home/10228/libs/parm
etis.lib /home/10228/libs/metis.lib" --with-metis-include=/home/10228/metis-5.1.
0/include --with-metis-lib=/home/10228/libs/metis.lib --useThreads=0
[1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0

job aborted:
rank: node: exit code[: error message]

I'll ask to the PETSc support how to proceed under windows.

15

Re: compile with SLEPc under windows

Giovanni, I have created a wiki account for you. Yopu should receive a separate mail form wiki engine.
Sorry for not being able to perticipate in the discussion, but at the moment there is no time on my side.
Borek

Re: compile with SLEPc under windows

Thanks Borek, I'm waiting for the e-mail.

I contacted the PETSc support, and, since Valgrind cannot be used in Windows, the debug must be done in MS devenv.

Before doing that, I cleaned and rebuild the solution, and it worked fine:

____________________________________________________
           OOFEM - Finite Element Solver
        Copyright (C) 1994-2014 Borek Patzak
____________________________________________________
Reading all data from input file dyn_bar01.oofem.in.0
Total number of solution steps     3
Instanciating domain                1
Instanciated nodes & sides          2
Instanciated elements               1
Instanciated cross sections         1
Instanciated materials              1
Instanciated BCs                    2
Instanciated ICs                    0
Instanciated load-time fncts        1
Consistency check:  OK
Renumbering dofs in all domains
Number of equations 1
Assembling mass matrix


Solving [Step number        0, Time    0.000000e+00]
Updating domain                     1
Updated nodes                       2
Updated Elements                    1
Checking rules...
EngngModel info: user time consumed by solution step 0: 0.00s
Number of equations 1
Assembling right hand side


Solving [Step number        1, Time    1.000000e+00]
Updating domain                     1
Updated nodes                       2
Updated Elements                    1
Checking rules...
EngngModel info: user time consumed by solution step 1: 0.00s
Number of equations 1
Assembling right hand side


Solving [Step number        2, Time    2.000000e+00]
Updating domain                     1
Updated nodes                       2
Updated Elements                    1
Checking rules...
EngngModel info: user time consumed by solution step 2: 0.00s


ANALYSIS FINISHED


Real time consumed: 000h:00m:00s
User time consumed: 000h:00m:00s
CommunicationPacketPool: allocated 0 packets
(packet size: 40960, 0 leased, 0 free)
Total 0 error(s) and 0 warning(s) reported

I'm sorry I cannot give an explanation, the only things I changed are in the CMakeLists.txt file as shown in the patch attached.
Finally, you have to tell to the linker to include the lib directory of the Intel Compiler but to exclude LICMT.lib.

Post's attachments

CMakeLists_Windows.patch 3.63 kb, 2 downloads since 2014-11-19 

You don't have the permssions to download the attachments of this post.

Re: compile with SLEPc under windows

Even though I would like to help this become easier under windows, I can not apply that patch.
The PETScConfig.cmake file in "${MY_PETSC_DIR}/${MY_PETSC_ARCH}/conf" is not meant for use like this. It is missing specifying it's main library, and some other shortfalls.
I asked the PETSc developers to correct this file but they didn't want to supply such a file (though the identical content formatted in a pkgconfig file was OK), which means it just becomes way more inconvenient for windows users.

The PETScConfig.cmake is only meant to be used internally, and is not included when you "make install", so it really can't be relied upon.

Re: compile with SLEPc under windows

Yes, I imagined that this patch would be harmful for *nix users. I posted just for windows users and to show you what is missing on windows platform in CMakeLists.txt. I think the best solution is to publish the CMaleLists.txt seprately in the wiki page I'd like to do, only, of course, if you agree. The main issue in CMakeLists is to ensure that all the path are included and to avoid pkg_config (which is available under windows but it is completely useless here, at least in my experience).

I already read somewhere of your request to the PETSc developers; I agree with you and I would say that PETSc usage under windows is discourage since installation.

Re: compile with SLEPc under windows

Actually, I think, more and more, you will have a much better experience if there was support for PARDISO, and using shared memory parallelization.
The only need for MPI solvers would be if you have actual windows clusters (which I think is a bit crazy).

Adding support for pardiso and a matching matrix type shouldn't be that difficult.

Re: compile with SLEPc under windows

I have a windows cluster of 22 machines (some with windows xp 32bit, others with win7 64). It supports MPICH2 and MPI.NET and it was already been used for OpenSees.

PETSc development, as answer to my request about the access violation, said:

You might still have some memory corruption errors. They might not
show up on some builds - but show up on others.
[valgrind on linux is a good way to track such errors. Perhaps there
are similar tools on windows - but I don't know much about such
developer tools on windows..]

I agree that PARDISO will be a good choice (see for example OpenSeesSP, which do not run on separate machines).

Re: compile with SLEPc under windows

If you can get a hold of Clang for windows, then it has compilation flags that are just as good as valgrind (in my experience).
cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_CXX_FLAGS="-fsanitize=address"
It warns whenever there are leaks, or memory access violations, during runtime.

(I believe GCC also supports address sanitizer as well, but I haven't tried it yet)

Re: compile with SLEPc under windows

The Windows version of Clang can be downloaded from:
http://llvm.org/releases/download.html

and the LLVM plugin for VS2013 from:
http://llvm.org/builds/

I advise to the get them here, because compliling the latest version from SVN under windows is difficult (the main difficulty is that some files are missing, are reported also here http://llvm.org/bugs/show_bug.cgi?id=11232 , old bug but still present in my installation...).

Anyway, I recompiled PETSc for release and it seems to work fine, at least with dyn_bar01.oofem.in test.