| Version 6 (modified by mmamonski, 12 years ago) (diff) |
|---|
Introduction
This page ilustrates MUSCLization of real astrophysics code PIERNIK ("Gingerbread").
The orginal code
- monolitic
- fortran
- MPI/HDF5 dependency
- high scalability of the main MHD module
- propotype particule simulation Monte Carlo module
- coupling done via global variables
The goal
- couple using MUSCLE, why?
- the coupling is done one way (from MHD to MC), this brings a potential of introducing new level of parallelism (in curent code MHD and MC simulations are called one after another, sequentially). TODO: figure
- the MC is in process of GPU-enabling, one may want to run both modules on different heterogenous resources (i.e. MC on GPU cluster, base MHD code on Intel Nehalem cluster)
Step By Step Guide
linking with MUSCLE
PIERNIK uses own build system based on Python scripts, all set of flags used for compilation are stored in plain configuration files like this one:
PROG = piernik USE_GNUCPP = yes F90 = mpif90 F90FLAGS = -ggdb -fdefault-real-8 -ffree-form -std=gnu -fimplicit-none -ffree-line-length-none F90FLAGS += -Ofast -funroll-loops F90FLAGS += -I/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/include LDFLAGS = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib
In order to link with MUSCLE 2.0 we have to alter the last lines of the file:
... LDFLAGS = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib -L/mnt/lustre/scratch/groups/plggmuscle/2.0/devel-debug/lib LIBS = -lmuscle2
Then we build the code with the following command:
# load MUSCLE module load muscle2/devel-debug # load PIERN dependencies (HDF5, OpenMPI, newest GNU compiler) module load plgrid/libs/hdf5/1.8.9-gnu-4.7.2-ompi ./setup mc_collisions_test -c gnufast -d HDF5,MUSCLE,MHD_KERNEL
The MUSCLE and MHD_KERNEL stands for preprocessor defines as we want to keep the MUSCLE dependency conditional, we will use them later.
Preparation - adding MUSCLE_Init and MUSCLE _finalize calls
Using this [Fortran API|MUSCLE Fortran tuturial] as reference it was relatively easy to add the following code to the main PIERNIK file (piernik.F90):
#ifdef MUSCLE
call muscle_fortran_init
#endif
call init_piernik
...
call cleanup_piernik
#ifdef MUSCLE
call MUSCLE_Finalize
#endif
...
#ifdef MUSCLE
subroutine muscle_fortran_init()
implicit none
integer :: argc, i, prevlen, newlen
character(len=25600) :: argv
character(len=255) :: arg
prevlen = 0
argc = command_argument_count()
do i = 0, argc
call get_command_argument(i, arg)
newlen = len_trim(arg)
argv = argv(1:prevlen) // arg(1:newlen) // char(0)
prevlen = prevlen + newlen + 1
end do
call MUSCLE_Init(argc, argv(1:prevlen))
end subroutine muscle_fortran_init
#endif
end program piernik
First try - run the NOP kernel in MUSCLE environment
The MUSCLE_Init assumes that application is called with MUSCLE environment so it will always fail if called directly:
$./piernik (12:29:26 ) MUSCLE port not given. Starting new MUSCLE instance. (12:29:26 ) ERROR: Could not instantiate MUSCLE: no command line arguments given. (12:29:26 ) Program finished
At first we need to prepare a simplistic CxA file which describes the simulation, we starts from single kernel and no conduits:
# configure cxa properties
cxa = Cxa.LAST
# declare kernels and their params
cxa.add_kernel('mhd', 'muscle.core.standalone.NativeKernel')
cxa.env["mhd:command"] = "./piernik"
cxa.env["mhd:dt"] = 1
# global params
cxa.env["max_timesteps"] = 4
cxa.env["cxa_path"] = File.dirname(__FILE__)
# configure connection scheme
cs = cxa.cs
Now we are ready to run PIERNIK MHD module in MUSCLE:
$muscle2 --main --cxa piernik.cxa.rb mhd Running both MUSCLE2 Simulation Manager and the Simulation === Running MUSCLE2 Simulation Manager === [12:39:05 muscle] Started the connection handler, listening on 10.3.1.22:5000 === Running MUSCLE2 Simulation === [12:39:06 muscle] Using directory </scratch/26934481.batch.grid.cyf-kr.edu.pl/n3-1-22.local_2013-03-09_12-39-05_23596> [12:39:06 muscle] mhd: connecting... [12:39:06 muscle] Registered ID mhd [12:39:06 muscle] mhd conduit entrances (out): [] mhd conduit exits (in): [] [12:39:06 muscle] mhd: executing (12:39:06 mhd) Spawning standalone kernel: [./piernik] [n3-1-22.local:23649] mca: base: component_find: unable to open /software/local/OpenMPI/1.6.3/ib/gnu/4.1.2/lib/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) Start of the PIERNIK code. No. of procs = 1 Warning @ 0: [units:init_units] PIERNIK will use 'cm', 'sek', 'gram' defined in problem.par [units:init_units] cm = 1.3459000E-11 [user unit] [units:init_units] sek = 3.1688088E-08 [user unit] [units:init_units] gram = 1.0000000E-22 [user unit] Starting problem : mctest :: tst Info @ 0: Working with 2 fluid. Info @ 0: Number of cells: 1 Info @ 0: Cell volume: 4.1016785411997372E+032 Info @ 0: Monomer mass: 4.2893211697012652E-013 Info @ 0: Temperature: 200.02221228956296 Info @ 0: Number of monomers per one representative particle: 1.3865676291650172E+030 Info @ 0: Dust density [g/cm3]: 2.9000000001269637E-013 Warning @ 0: [initfluids:sanitize_smallx_checks] adjusted smalld to 1.1895E-04 Warning @ 0: [initfluids:sanitize_smallx_checks] adjusted smallp to 1.7705E-01 Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 1 dt = 1.5778800002006178E+09 s t = 9.9998257934644172E+01 yr dWallClock = 0.04 s [MC] Writing output 1 time = 9.9998257934644172E+01 yr = 3.1557600004012356E+09 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 2 dt = 1.5778800002006178E+09 s t = 1.9999651586928834E+02 yr dWallClock = 0.04 s [MC] Writing output 2 time = 1.9999651586928834E+02 yr = 6.3115200008024712E+09 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 3 dt = 1.5778800002006178E+09 s t = 2.9999477380393250E+02 yr dWallClock = 0.11 s [MC] Writing output 3 time = 2.9999477380393250E+02 yr = 9.4672800012037067E+09 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 4 dt = 1.5778800002006178E+09 s t = 3.9999303173857669E+02 yr dWallClock = 0.17 s [MC] Writing output 4 time = 3.9999303173857669E+02 yr = 1.2623040001604942E+10 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 5 dt = 1.5778800002006178E+09 s t = 4.9999128967322088E+02 yr dWallClock = 0.18 s [MC] Writing output 5 time = 4.9999128967322088E+02 yr = 1.5778800002006178E+10 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 6 dt = 1.5778800002006178E+09 s t = 5.9998954760786501E+02 yr dWallClock = 0.77 s [MC] Writing output 6 time = 5.9998954760786501E+02 yr = 1.8934560002407413E+10 s Info @ 0: Simulation has reached final time t = 600.000 Finishing .......... (12:39:08 mhd) Program finished. (12:39:08 mhd) Command [./piernik] finished. [12:39:08 muscle] mhd: finished [12:39:08 muscle] All ID's have finished, quitting MUSCLE now. [12:39:08 muscle] All local submodels have finished; exiting. Executed in </scratch/26934481.batch.grid.cyf-kr.edu.pl/n3-1-22.local_2013-03-09_12-39-05_23596>
Adding second kernel: MC
At first we need to create separate PIERNIK build for the MC kernel
/setup -o mc mc_collisions_test -c gnufast -d HDF5,MUSCLE,MC_KERNEL
Please note that we use -o mc (use suffix for obj directory) and MC_KERNEL instead of MHD_KERNEL. This will create another build in ./obj_mc/piernik. Now we are ready to add another kernel definition in the CxA file:
cxa.add_kernel('mc', 'muscle.core.standalone.NativeKernel')
cxa.env["mc:command"] = "../obj_mc/piernik"
cxa.env["mc:dt"] = 1;
