Version 12 (modified by mmamonski, 11 years ago) (diff)

--

Introduction

This page ilustrates MUSCLization of real astrophysics code  PIERNIK ("Gingerbread").

The orginal code

  • monolitic
  • fortran
  • MPI/HDF5 dependency
  • high scalability of the main MHD module
  • propotype particule simulation Monte Carlo module
  • coupling done via global variables

The goal

  • couple using MUSCLE, why?
    • the coupling is done one way (from MHD to MC), this brings a potential of introducing new level of parallelism (in curent code MHD and MC simulations are called one after another, sequentially). TODO: figure
    • the MC is in process of GPU-enabling, one may want to run both modules on different heterogenous resources (i.e. MC on GPU cluster, base MHD code on Intel Nehalem cluster)

Step By Step Guide

linking with MUSCLE

PIERNIK uses own build system based on Python scripts, all set of flags used for compilation are stored in plain configuration files like this one:

PROG      = piernik
USE_GNUCPP = yes
F90       = mpif90
F90FLAGS  =  -ggdb -fdefault-real-8 -ffree-form -std=gnu -fimplicit-none -ffree-line-length-none 
F90FLAGS += -Ofast -funroll-loops
F90FLAGS += -I/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/include
LDFLAGS   = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib 

In order to link with MUSCLE 2.0 we have to alter the last lines of the file:

...
LDFLAGS   = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib -L/mnt/lustre/scratch/groups/plggmuscle/2.0/devel-debug/lib 
LIBS      = -lmuscle2

Then we build the code with the following command:

# load MUSCLE
module load muscle2/devel-debug
# load PIERN dependencies (HDF5, OpenMPI, newest GNU compiler)
module load plgrid/libs/hdf5/1.8.9-gnu-4.7.2-ompi
./setup mc_collisions_test -c gnufast -d HDF5,MUSCLE,MHD_KERNEL

The MUSCLE and MHD_KERNEL stands for preprocessor defines as we want to keep the MUSCLE dependency conditional, we will use them later.

Preparation - adding MUSCLE_Init and MUSCLE _finalize calls

Using this [Fortran API|MUSCLE Fortran tuturial] as reference it was relatively easy to add the following code to the main PIERNIK file (piernik.F90):

#ifdef MUSCLE
   call muscle_fortran_init
#endif
   call init_piernik

...

   call cleanup_piernik
#ifdef MUSCLE
   call MUSCLE_Finalize
#endif 

...

#ifdef MUSCLE
subroutine muscle_fortran_init()
      implicit none
      integer :: argc, i, prevlen, newlen
      character(len=25600) :: argv
      character(len=255) :: arg

      prevlen = 0
      argc = command_argument_count()

      do i = 0, argc
        call get_command_argument(i, arg)
        newlen = len_trim(arg)
        argv = argv(1:prevlen) // arg(1:newlen) // char(0)
        prevlen = prevlen + newlen + 1
    end do

    call MUSCLE_Init(argc, argv(1:prevlen))
end subroutine muscle_fortran_init
#endif

end program piernik

First try - run the NOP kernel in MUSCLE environment

The MUSCLE_Init assumes that application is called with MUSCLE environment so it will always fail if called directly:

$./piernik 
(12:29:26       ) MUSCLE port not given. Starting new MUSCLE instance.
(12:29:26       ) ERROR: Could not instantiate MUSCLE: no command line arguments given.
(12:29:26       ) Program finished

At first we need to prepare a simplistic CxA file which describes the simulation, we starts from single kernel and no conduits:

# configure cxa properties
cxa = Cxa.LAST

# declare kernels and their params
cxa.add_kernel('mhd', 'muscle.core.standalone.NativeKernel')
cxa.env["mhd:command"] = "./piernik"
cxa.env["mhd:dt"] = 1

# global params
cxa.env["max_timesteps"] = 4
cxa.env["cxa_path"] = File.dirname(__FILE__)

# configure connection scheme
cs = cxa.cs

Now we are ready to run PIERNIK MHD module in MUSCLE:

$muscle2 --main --cxa piernik.cxa.rb mhd
Running both MUSCLE2 Simulation Manager and the Simulation
=== Running MUSCLE2 Simulation Manager ===
[12:39:05 muscle] Started the connection handler, listening on 10.3.1.22:5000
=== Running MUSCLE2 Simulation ===
[12:39:06 muscle] Using directory </scratch/26934481.batch.grid.cyf-kr.edu.pl/n3-1-22.local_2013-03-09_12-39-05_23596>
[12:39:06 muscle] mhd: connecting...
[12:39:06 muscle] Registered ID mhd
[12:39:06 muscle] mhd conduit entrances (out): []
		  mhd conduit exits (in): []
[12:39:06 muscle] mhd: executing
(12:39:06    mhd) Spawning standalone kernel: [./piernik]
[n3-1-22.local:23649] mca: base: component_find: unable to open /software/local/OpenMPI/1.6.3/ib/gnu/4.1.2/lib/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored)

Start of the PIERNIK code. No. of procs =     1

Warning @    0: [units:init_units] PIERNIK will use 'cm', 'sek', 'gram' defined in problem.par
[units:init_units] cm   =  1.3459000E-11 [user unit]
[units:init_units] sek  =  3.1688088E-08 [user unit]
[units:init_units] gram =  1.0000000E-22 [user unit]
   Starting problem : mctest :: tst

Info    @    0:  Working with            2  fluid.
Info    @    0:  Number of cells:            1
Info    @    0:  Cell volume:    4.1016785411997372E+032
Info    @    0:  Monomer mass:    4.2893211697012652E-013
Info    @    0:  Temperature:    200.02221228956296
Info    @    0:  Number of monomers per one representative particle:    1.3865676291650172E+030
Info    @    0:  Dust density [g/cm3]:    2.9000000001269637E-013
Warning @    0: [initfluids:sanitize_smallx_checks] adjusted smalld to  1.1895E-04
Warning @    0: [initfluids:sanitize_smallx_checks] adjusted smallp to  1.7705E-01
Info    @    0:  Timesteps:    2.0416940798311732E+038   50.000000000000000
Info    @    0:  Timesteps:    50.000000000000000        50.000000000000000
Info    @    0:  Timesteps:    2.0416940798311732E+038   50.000000000000000
Info    @    0:  Timesteps:    50.000000000000000        50.000000000000000
[MC] nstep =       1   dt =  1.5778800002006178E+09 s   t =  9.9998257934644172E+01 yr   dWallClock =    0.04 s
[MC] Writing output     1 time =  9.9998257934644172E+01 yr =  3.1557600004012356E+09 s
Info    @    0:  Timesteps:    2.0416940798311732E+038   50.000000000000000
Info    @    0:  Timesteps:    50.000000000000000        50.000000000000000
[MC] nstep =       2   dt =  1.5778800002006178E+09 s   t =  1.9999651586928834E+02 yr   dWallClock =    0.04 s
[MC] Writing output     2 time =  1.9999651586928834E+02 yr =  6.3115200008024712E+09 s
Info    @    0:  Timesteps:    2.0416940798311732E+038   50.000000000000000
Info    @    0:  Timesteps:    50.000000000000000        50.000000000000000
[MC] nstep =       3   dt =  1.5778800002006178E+09 s   t =  2.9999477380393250E+02 yr   dWallClock =    0.11 s
[MC] Writing output     3 time =  2.9999477380393250E+02 yr =  9.4672800012037067E+09 s
Info    @    0:  Timesteps:    2.0416940798311732E+038   50.000000000000000
Info    @    0:  Timesteps:    50.000000000000000        50.000000000000000
[MC] nstep =       4   dt =  1.5778800002006178E+09 s   t =  3.9999303173857669E+02 yr   dWallClock =    0.17 s
[MC] Writing output     4 time =  3.9999303173857669E+02 yr =  1.2623040001604942E+10 s
Info    @    0:  Timesteps:    2.0416940798311732E+038   50.000000000000000
Info    @    0:  Timesteps:    50.000000000000000        50.000000000000000
[MC] nstep =       5   dt =  1.5778800002006178E+09 s   t =  4.9999128967322088E+02 yr   dWallClock =    0.18 s
[MC] Writing output     5 time =  4.9999128967322088E+02 yr =  1.5778800002006178E+10 s
Info    @    0:  Timesteps:    2.0416940798311732E+038   50.000000000000000
Info    @    0:  Timesteps:    50.000000000000000        50.000000000000000
[MC] nstep =       6   dt =  1.5778800002006178E+09 s   t =  5.9998954760786501E+02 yr   dWallClock =    0.77 s
[MC] Writing output     6 time =  5.9998954760786501E+02 yr =  1.8934560002407413E+10 s
Info    @    0: Simulation has reached final time t = 600.000
Finishing ..........
(12:39:08    mhd) Program finished.
(12:39:08    mhd) Command [./piernik] finished.
[12:39:08 muscle] mhd: finished
[12:39:08 muscle] All ID's have finished, quitting MUSCLE now.
[12:39:08 muscle] All local submodels have finished; exiting.


	Executed in </scratch/26934481.batch.grid.cyf-kr.edu.pl/n3-1-22.local_2013-03-09_12-39-05_23596>

Adding second kernel: MC

At first we need to create separate PIERNIK build for the MC kernel

/setup -o mc mc_collisions_test -c gnufast -d HDF5,MUSCLE,MC_KERNEL

Please note that we use -o mc (use suffix for obj directory) and MC_KERNEL instead of MHD_KERNEL. This will create another build in ./obj_mc/piernik. Now we are ready to add another kernel definition in the CxA file:

cxa.add_kernel('mc', 'muscle.core.standalone.NativeKernel')
cxa.env["mc:command"] = "./piernik" #here we assume that the other kernel is started in obj_mc
cxa.env["mc:dt"] = 1;

Because we need to run every kernel in separate directory, we need to start two MUSCLE instances.

cd obj
time muscle2 --main --bindaddr 127.0.0.1 --bindport 1234 --cxa ../scripts/piernik.cxa.rb mhd &
cd ..
cd obj_mc
time muscle2 --manager 127.0.0.1:1234 --cxa ../scripts/piernik.cxa.rb mc &
wait

Coupling

So we have now both kernels running with MUSCLE, but get scientific relevant results we must finish the coupling: exchanging gas density and momentum for each cell. At first we have two define conduits in the CxA file:

cs.attach('mhd' => 'mc') {
	tie('rho_gas', 'rho_gas')
	tie('m_gas', 'm_gas')
}

}}}