Since revision 67 in SVN branch inst_procedure it is possible to use MPI kernels with MUSCLE.

Limits & requirements

MPI implementation

Currently OpenMPI does not support binding with JNI (see  this bug)

The following implementation have been tested so far:

•  Intel MPI
•  MPICH2

Running practices

When using MPI in some kernel, this kernel must be run as the only in this muscle instance. Other kernels, as well as main, must be run from other MUSCLE instances.

Preparing source code

Java

A new method, public void executeDirectly(), is available in CaController class. Only this method is called instead of normal MUSCLE routines on the processes with non-zero rank. Process with rank 0 is started in the usual way. Portals cannot be attached to slave processes (i.e. to the processes with non-zero rank). The executeDirectly method calls by default execute().

User may (and should) overload the executeDirectly` to better fit ones needs.

C/C++

As in Java, only process with Rank 0 may use MUSCLE routines, such as portals or the kernel.willStop() method. User is responsible for stopping all the processes once the kernel should stop.

MPI should be used in the natural way, i.e. the program should start with MPI::Initialize() and finish with MPI::Finalize().

Running MPI kernels

To run a MPI kernel, one must distribute the processes with mpirun/mpiexec utility. One kernel per one MUSCLE instance allowed. It is crucial to add the --mpi parameter for the muscle command - otherwise the simulation will fail, as multiple identical MUSCLE instances will arise.

mpirun -n 5 muscle --mpi --cxa_file src/cxa/Test.cxa.rb mpi

Of course, in another process the main must be started, as well as other kernels required by the cxa file, for example

muscle --main plumber --cxa_file src/cxa/Test.cxa.rb --autoquit

An example (started with the commands above) has been added to the SVN in revision 68.