Introduction

This page ilustrates MUSCLization of real astrophysics code  PIERNIK ("Gingerbread").

The orginal code

• monolitic
• fortran
• MPI/HDF5 dependency
• high scalability of the main MHD module
• propotype particule simulation Monte Carlo module
• coupling done via global variables

The goal

• couple using MUSCLE, why?
  • the coupling is done one way (from MHD to MC), this brings a potential of introducing new level of parallelism (in curent code MHD and MC simulations are called one after another, sequentially). TODO: figure
  • the MC is in process of GPU-enabling, one may want to run both modules on different heterogenous resources (i.e. MC on GPU cluster, base MHD code on Intel Nehalem cluster)

Step By Step Guide

linking with MUSCLE

PIERNIK uses own build system based on Python scripts, all set of flags used for compilation are stored in plain configuration files like this one:

PROG      = piernik
USE_GNUCPP = yes
F90       = mpif90
F90FLAGS  =  -ggdb -fdefault-real-8 -ffree-form -std=gnu -fimplicit-none -ffree-line-length-none 
F90FLAGS += -Ofast -funroll-loops
F90FLAGS += -I/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/include
LDFLAGS   = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib 

In order to link with MUSCLE 2.0 we have to alter the last lines of the file:

...
LDFLAGS   = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib -L/mnt/lustre/scratch/groups/plggmuscle/2.0/devel-debug/lib 
LIBS      = -lmuscle2

Then we build the code with the following command:

# load MUSCLE
module load muscle2/devel-debug
# load PIERN dependencies (HDF5, OpenMPI, newest GNU compiler)
module load plgrid/libs/hdf5/1.8.9-gnu-4.7.2-ompi
./setup mc_collisions_test -c gnufast -d HDF5,MUSCLE,MHD_KERNEL

The MUSCLE and MHD_KERNEL stands for preprocessor defines as we want to keep the MUSCLE dependency conditional, we will use them later.

First Step - MUSCLE init