= Introduction = This page ilustrates MUSCLization of real astrophysics code [http://piernik.astri.umk.pl/ PIERNIK] ("Gingerbread"). = The orginal code = - monolitic - fortran - MPI/HDF5 dependency - high scalability of the main MHD module - propotype particule simulation Monte Carlo module - coupling done via global variables = The goal = - couple using MUSCLE, why? * the coupling is done one way (from MHD to MC), this brings a potential of introducing new level of parallelism (in curent code MHD and MC simulations are called one after another, sequentially). TODO: figure * the MC is in process of GPU-enabling, one may want to run both modules on different heterogenous resources (i.e. MC on GPU cluster, base MHD code on Intel Nehalem cluster) = Step By Step Guide = == linking with MUSCLE == PIERNIK uses own build system based on Python scripts, all set of flags used for compilation are stored in plain configuration files like this one: {{{ PROG = piernik USE_GNUCPP = yes F90 = mpif90 F90FLAGS = -ggdb -fdefault-real-8 -ffree-form -std=gnu -fimplicit-none -ffree-line-length-none F90FLAGS += -Ofast -funroll-loops F90FLAGS += -I/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/include LDFLAGS = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib }}} In order to link with MUSCLE 2.0 we have to alter the last lines of the file: {{{ ... LDFLAGS = -Wl,--as-needed -Wl,-O1 -L/software/local/libs/hdf5/1.8.9-pre1/gnu-4.7.2-ompi/lib -L/mnt/lustre/scratch/groups/plggmuscle/2.0/devel-debug/lib LIBS = -lmuscle2 }}} Then we build the code with the following command: {{{ # load MUSCLE module load muscle2/devel-debug # load PIERN dependencies (HDF5, OpenMPI, newest GNU compiler) module load plgrid/libs/hdf5/1.8.9-gnu-4.7.2-ompi ./setup mc_collisions_test -c gnufast -d HDF5,MUSCLE,MHD_KERNEL }}} The `MUSCLE` and `MHD_KERNEL` stands for preprocessor defines as we want to keep the MUSCLE dependency conditional, we will use them later. == Preparation - adding MUSCLE_Init and MUSCLE _finalize calls == Using this [Fortran API|MUSCLE Fortran tuturial] as reference it was relatively easy to add the following code to the main PIERNIK file (piernik.F90): {{{ #ifdef MUSCLE call muscle_fortran_init #endif call init_piernik ... call cleanup_piernik #ifdef MUSCLE call MUSCLE_Finalize #endif ... #ifdef MUSCLE subroutine muscle_fortran_init() implicit none integer :: argc, i, prevlen, newlen character(len=25600) :: argv character(len=255) :: arg prevlen = 0 argc = command_argument_count() do i = 0, argc call get_command_argument(i, arg) newlen = len_trim(arg) argv = argv(1:prevlen) // arg(1:newlen) // char(0) prevlen = prevlen + newlen + 1 end do call MUSCLE_Init(argc, argv(1:prevlen)) end subroutine muscle_fortran_init #endif end program piernik }}} == First try - run the NOP kernel in MUSCLE environment == The MUSCLE_Init assumes that application is called with MUSCLE environment so it will always fail if called directly: {{{ $./piernik (12:29:26 ) MUSCLE port not given. Starting new MUSCLE instance. (12:29:26 ) ERROR: Could not instantiate MUSCLE: no command line arguments given. (12:29:26 ) Program finished }}} At first we need to prepare a simplistic CxA file which describes the simulation, we starts from single kernel and no conduits: {{{ # configure cxa properties cxa = Cxa.LAST # declare kernels and their params cxa.add_kernel('mhd', 'muscle.core.standalone.NativeKernel') cxa.env["mhd:command"] = "./piernik" cxa.env["mhd:dt"] = 1 # global params cxa.env["max_timesteps"] = 4 cxa.env["cxa_path"] = File.dirname(__FILE__) # configure connection scheme cs = cxa.cs }}} Now we are ready to run PIERNIK MHD module in MUSCLE: {{{ $muscle2 --main --cxa piernik.cxa.rb mhd Running both MUSCLE2 Simulation Manager and the Simulation === Running MUSCLE2 Simulation Manager === [12:39:05 muscle] Started the connection handler, listening on 10.3.1.22:5000 === Running MUSCLE2 Simulation === [12:39:06 muscle] Using directory [12:39:06 muscle] mhd: connecting... [12:39:06 muscle] Registered ID mhd [12:39:06 muscle] mhd conduit entrances (out): [] mhd conduit exits (in): [] [12:39:06 muscle] mhd: executing (12:39:06 mhd) Spawning standalone kernel: [./piernik] [n3-1-22.local:23649] mca: base: component_find: unable to open /software/local/OpenMPI/1.6.3/ib/gnu/4.1.2/lib/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) Start of the PIERNIK code. No. of procs = 1 Warning @ 0: [units:init_units] PIERNIK will use 'cm', 'sek', 'gram' defined in problem.par [units:init_units] cm = 1.3459000E-11 [user unit] [units:init_units] sek = 3.1688088E-08 [user unit] [units:init_units] gram = 1.0000000E-22 [user unit] Starting problem : mctest :: tst Info @ 0: Working with 2 fluid. Info @ 0: Number of cells: 1 Info @ 0: Cell volume: 4.1016785411997372E+032 Info @ 0: Monomer mass: 4.2893211697012652E-013 Info @ 0: Temperature: 200.02221228956296 Info @ 0: Number of monomers per one representative particle: 1.3865676291650172E+030 Info @ 0: Dust density [g/cm3]: 2.9000000001269637E-013 Warning @ 0: [initfluids:sanitize_smallx_checks] adjusted smalld to 1.1895E-04 Warning @ 0: [initfluids:sanitize_smallx_checks] adjusted smallp to 1.7705E-01 Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 1 dt = 1.5778800002006178E+09 s t = 9.9998257934644172E+01 yr dWallClock = 0.04 s [MC] Writing output 1 time = 9.9998257934644172E+01 yr = 3.1557600004012356E+09 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 2 dt = 1.5778800002006178E+09 s t = 1.9999651586928834E+02 yr dWallClock = 0.04 s [MC] Writing output 2 time = 1.9999651586928834E+02 yr = 6.3115200008024712E+09 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 3 dt = 1.5778800002006178E+09 s t = 2.9999477380393250E+02 yr dWallClock = 0.11 s [MC] Writing output 3 time = 2.9999477380393250E+02 yr = 9.4672800012037067E+09 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 4 dt = 1.5778800002006178E+09 s t = 3.9999303173857669E+02 yr dWallClock = 0.17 s [MC] Writing output 4 time = 3.9999303173857669E+02 yr = 1.2623040001604942E+10 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 5 dt = 1.5778800002006178E+09 s t = 4.9999128967322088E+02 yr dWallClock = 0.18 s [MC] Writing output 5 time = 4.9999128967322088E+02 yr = 1.5778800002006178E+10 s Info @ 0: Timesteps: 2.0416940798311732E+038 50.000000000000000 Info @ 0: Timesteps: 50.000000000000000 50.000000000000000 [MC] nstep = 6 dt = 1.5778800002006178E+09 s t = 5.9998954760786501E+02 yr dWallClock = 0.77 s [MC] Writing output 6 time = 5.9998954760786501E+02 yr = 1.8934560002407413E+10 s Info @ 0: Simulation has reached final time t = 600.000 Finishing .......... (12:39:08 mhd) Program finished. (12:39:08 mhd) Command [./piernik] finished. [12:39:08 muscle] mhd: finished [12:39:08 muscle] All ID's have finished, quitting MUSCLE now. [12:39:08 muscle] All local submodels have finished; exiting. Executed in }}} == Adding second kernel: MC == At first we need to create separate PIERNIK build for the MC kernel {{{ /setup -o mc mc_collisions_test -c gnufast -d HDF5,MUSCLE,MC_KERNEL }}} Please note that we use `-o mc` (use suffix for obj directory) and MC_KERNEL instead of MHD_KERNEL. This will create another build in `./obj_mc/piernik`. Now we are ready to add another kernel definition in the CxA file: {{{ cxa.add_kernel('mc', 'muscle.core.standalone.NativeKernel') cxa.env["mc:command"] = "../obj_mc/piernik" cxa.env["mc:dt"] = 1; }}}