Changes between Version 16 and Version 17 of C++ API

Timestamp:

09/12/13 12:05:30 (11 years ago)

Author:

jorisborgdorff

Comment:

--

C++ API

@@ -16 +16 @@
 muscle_error_t MUSCLE_Send(const char *exit_name, void *array, size_t size, muscle_datatype_t type);
 void* MUSCLE_Receive(const char *entrance_name, void *array, size_t *size, muscle_datatype_t type);
+
+int MUSCLE_Barrier_Init(char **barrier, int *len, int num_mpi_ranks);
+int MUSCLE_Barrier(const char *barrier);
+int MUSCLE_Barrier_Destroy(char *barrier);
 }}}
 The {{{MUSCLE_Init}}} function must be called before any of the other MUSCLE functions is used, and also before MPI_Init is called, if MPI is used. The usage is the same of MPI_Init: to initialize MUSCLE given the current arguments of main. After all MUSCLE calls have been made, MUSCLE_Finalize should be called. Usually:
@@ -80 +84 @@
 }}}
 
+=== MUSCLE and MPI ===
+
+When using MPI, MUSCLE_Send and MUSCLE_Receive only do their operations in rank 0, calls from other ranks are ignored. This means that the data should be gathered with MPI before a MUSCLE_Send and broadcasted after a MUSCLE_Receive.
+
+The `MUSCLE_Barrier` set of functions ease the integration of MUSCLE with MPI. Most MPI functions (including barrier, broadcast and gather) use a polling mechanism when they wait for communication to happen. This will use all the available CPU power but will somewhat reduce the latency of the operation. However, with MUSCLE, often other submodels than the MPI submodel should do some computing, and while the MPI operation waits this will slow down other submodels immensely. Therefore, MUSCLE has its own barrier operation, which has a higher latency than MPI_Barrier, but will not use any CPU resources. Since only rank 0 of the process ever receives data from MUSCLE, and a receive must wait for another submodel to send the message, that is a good point for calling a barrier. If multiple receives follow each other, barrier only needs to be called after the last one.
+
+The MUSCLE_Barrier API has three parts: `MUSCLE_Barrier_Init(char **barrier, size_t *len, int num_mpi_procs)`,
+`MUSCLE_Barrier(const char *barrier)` and `MUSCLE_Barrier_Destroy(char *barrier)`. In Init the barrier is created. It needs the number of MPI processes since MUSCLE itself does not do any MPI calls and has no other way to find out. The barrier data structure should then be broadcasted with MPI by the user so that each process uses the same barrier. Each time MUSCLE_Barrier is called, it waits until all ranks have called it. `MUSCLE_Barrier_Init` returns -1 in rank 0 if it fails, and 0 otherwise. `MUSCLE_Barrier` returns -1 in any rank that it fails in, but that does not guarantee that it failed in other ranks. In Destroy the resources of the barrier are cleaned up.
+
+A typical piece of code could look as follows:
+{{{
+int mpi_size;
+MPI_Comm_size(&mpi_size, MPI_COMM_WORLD);
+
+char *barrier;
+size_t barrier_len;
+MUSCLE_Barrier_Init(&barrier, &barrier_len, mpi_size);
+MPI_Bcast(barrier, barrier_len, MPI_CHAR, 0, MPI_COMM_WORLD);
+
+while (!EC) {
+    // Only receives in rank 0, it is ignored for other ranks
+    MUSCLE_Receive(data, ...);
+    MUSCLE_Barrier(barrier);
+    MPI_Bcast(data, ...)
+
+    // do something
+
+    // Only sends in rank 0, it is ignored in other ranks
+    MUSCLE_Send(data, ...)
+}
+MUSCLE_Barrier_Destroy(barrier);
+}}}
+This paradigm is used in `src/cpp/examples/simplempi/sender.c`.
+
 == C++ API ==
 
@@ -102 +140 @@
 std::string muscle::cxa::get_properties(void);
 std::string muscle::cxa::kernel_name(void);
+
+int muscle::util::barrier_init(char **barrier, size_t *len, int num_mpi_procs);
+int muscle::util::barrier(const char *barrier);
+void muscle::util::barrier(char *barrier);
 
 }}}