Version 24 (modified by mmamonski, 11 years ago) (diff)

--

The QCG-Computing service allows to map abstract application name to the absoluth path of the wrapper script. The mapping file is stored in file:

/opt/plgrid/qcg/etc/qcg-comp/application_mapfile

The file has the following syntax:

APPLICATION-NAME VERSION SCRIPT-PATH

where VERSION can be an asterix (*) which means "any version". There is no need to restart the qcg-compd service after updating the file. Example application_mapfile file:

MATLAB * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/matlab.qcg
NAMD * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/namd.qcg
bash * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/bash.qcg
R * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/R.qcg
CFX * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/cfx.qcg
fluent * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/fluent.qcg
nwchem * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/nwchem.qcg
g09	* /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/g09.qcg
qcg-ompi * /opt/exp_soft/plgrid/qcg-app-scripts/app-scripts/qcg-openmpi/qcg-openmpi.qcg

The example scripts can be checkouted from our svn:

svn co https://apps.man.poznan.pl/svn/qcg-computing/trunk/app-scripts

Please put the dir under location that is available from all worker nodes (e.g. /opt/exp_soft/plgrid/qcg-app-scripts/). Every script sources in the beginning its own, site specific, configuration file (e.g. matlab.qcg sources matlab.conf) which is responsible for initializing application environment (e.g. loading environment module, setting PATH). In the next sections we will shortly describe what should be provide in such file for every supported application. Make sure that all the files have execution and read rights for all.

common.conf

  • QCG_SCRATCH_DIR - the location of job temporary directory (e.g $TMPDIR) - OPTIONAL - default to $TMPDIR (if set) or "/tmp",
  • QCG_NO_DEBUG - if set do not create qcg.debug file - OPTIONAL,
  • QCG_GROUP_DIR_ROOT - The root of groups home directory (PL-Grid only). Default: PLG_GROUPS_SHARED - OPTIONAL,
  • MACHINE_FILE - the name of the environment variable pointing to the machine file (e.g. PBS_NODEFILE) - OPTIONAL.

bash.qcg

This application usually needs no configuration as it is assumed that bash binary cane be always found in the /bin directory. However you can request some extra argument to be passed to bash by specifying BASH_ARGS in common.conf, e.g.:

BASH_ARGS=--login

matlab.qcg

In matlab.conf simply load matlab module:

module load matlab/current

Or set the PATH variable:

export PATH=$PATH:/opt/exp_soft/matlab/bin/

namd.qcg

In namd.conf load the NAMD module

module load namd/current

Additionally if namd2 must be started using different command than mpiexec you can provide additional configuration variables:

MPIEXEC=charmrun
MPIEXEC_ARGS="++verbose ++local +p`cat $QCG_NODEFILE |  wc -l`"

g09.qcg

In g09.conf load the Gaussian module

module load gaussian/current

and set the following variables

#amount of memory in bytes allocated for the job
MEM_BYTES=`qstat -f $PBS_JOBID | grep Resource_List.mem | cut -f 2 -d = | tr "b" " " `
#primary, unlimited, scratch
GAUSSIAN_SCRATCH="/mnt/lustre/$USER"
#additional fast scratches, the value after colon denotes max used size in MB
GAUSSIAN_LIMITED_SCRATCHES="$TMPDIR:100"
#used as shred memory key (must be an integer!)
QCG_GAUSSIAN_SESSION_KEY=${PBS_JOBID%%.*}

moreover you have to compile the QCG gaussian helper library that is LD_PRELOADed before starting gaussian:

cd app-scripts/tools/qcg-gaussian
make
gcc -ggdb -Wall -fPIC -p -o libqcg-gaussian-dbg.so -shared qcg-gaussian.c -ldl
gcc -O3 -Wall -fPIC -p -o libqcg-gaussian.so -shared qcg-gaussian.c -ldl

abinit.qcg

in abinit.conf load the abinit module

module load abinit/current

alternatively you may set and extra mpiexec arguments, e.g.:

MPIEXEC_ARGS=" --mca orte_tmpdir_base $PWD "

or set additional variables

export MV2_ENABLE_AFFINITY=0

Application Scripts - Developers view

This section is target at QCG developers rather than administrators, its list environment variables which influence job life cycle:

Common Input Environment Variables

  • QCG_MODULES_LIST - list of environment modules to be loaded (separated by spaces)
  • QCG_PREPROCESS - file name of the pre process script
  • QCG_MONITOR - file name of the monitor script
  • QCG_MONITOR_INTERVAL - determines how frequently the QCG_MONITOR script should be called (in seconds). - TBD
  • QCG_NTF_CONSUMER_URL - the address of the Notification consumer interested in receiving notification about job output status change - TBD
  • QCG_NTF_WATCH_PATTERN - the regular expression that should trigger notification - TBD
  • QCG_NTF_WATCH_FILE - the file to be watched (default: stdout/err file) - TBD
  • QCG_NTF_WATCH_INTERVAL - determines how often the watched file should be checked (in seconds) - TBD
  • QCG_POSTPROCESS - file name of the post process script
  • QCG_OUTERR_FILE - target filename of the joined output/error stream file (default: output.log)
  • QCG_ZIPPED_INPUTS - set if the input files must be unziped first (e.g. QCG_ZIPPED_INPUTS=inputs.zip)
  • QCG_ZIP_OUTPUTS - whether to zip results into $QCG_ZIP_OUTPUTS.zip file
  • QCG_ZIP_OUTPUTS_FILTER - The wildcard pattern of files to be stored in the zip file (TBD)
  • QCG_COMP_PROCESSES_MAP - topology of hybrid application (e.g. QCG_COMP_PROCESSES_MAP="4:1:1:1")
  • QCG_FORCE_SCRATCH - force chdir to temporary directory

Application Scripts - Users view

Here we list environment variables available in any job submitted as QCG-Application (e.g. bash):

  • QCG_NODEFILE - the machinefile like file with list of nodes allocated for the job
  • QCG_PPN - number of processes per node
  • QCG_SPN - number of allocated slots per node (usually equal to QCG_PPN, differ only if job was submitted with: #QCG node=a:b:c where b>c)
  • QCG_PROCS - total number of proceses