Application mapping

The QCG-Computing service allows to map abstract application name to the absoluth path of the wrapper script. The mapping file is stored in file:

/etc/qcg/qcg-comp/application_mapfile

The file has the following syntax:

APPLICATION-NAME VERSION SCRIPT-PATH

where VERSION can be an asterix (*) which means "any version". The mapping file is periodically reloaded (5 minutes by default), so there is no need to restart the qcg-compd service after updating the file. Example application_mapfile file:

MATLAB * /opt/qcg-app-scripts/apps/matlab.app
NAMD * /opt/qcg-app-scripts/apps/namd.app
bash * /opt/qcg-app-scripts/apps/bash.app
R * /opt/qcg-app-scripts/apps/R.app
CFX * /opt/qcg-app-scripts/apps/cfx.app
fluent * /opt/qcg-app-scripts/apps/fluent.app
nwchem * /opt/qcg-app-scripts/apps/nwchem.app
g09	* /opt/qcg-app-scripts/apps/g09.app

The scripts for the most common applications are available in qcg-appscripts RPM package.

For the QCG-Computing to be usable you must provide mapping for at least the bash application, i.e.:

bash * /opt/qcg-app-scripts/apps/bash.app

Frequently Asked Question: Can I use simply mapping to /bin/bash

bash * /bin/bash

? The answer is "No". You must use the bash.qcg which does much more (e.g. setup environment variables like QCG_NODEFILE).

qcg-appscripts package

This package contains QCG application scripts for the most common applications. Role of the application scripts, apart from the application launch, is the:

• loads the appropriate modules,
• converts input files to UNIX character encoding,
• setup environment,
• launch user's helper scripts - preprocess, postprocess, assistent,
• monitor the execution of application through defined schemes or user scripts,
• handling interactive jobs,
• etc.

All files are installed in directory:

    /usr/share/qcg-appscripts

This directory contains following directories:

• apps - application configuration files,
• app-scripts - application scripts,
• config - QCG application scripts configuration,
• core - QCG scripts library,
• tools - QCG tools used by applications.

The administrator must create/edit application configuration files (apps) that contains settings specific to each cluster, such as:

• name of the application module,
• environment variables needed by application.

Files from apps directory should be referenced by QCG-Computing application mapping file (/etc/qcg/qcg-comp/application_mapfile).

The application scripts must be accessible by jobs running on cluster worker nodes. Thus directories apps, app-scripts, config, core and tools must be copied to directory shared by all worker nodes. This is done by:

/usr/sbin/qcg-appscripts-deploy

script included in the package. This script reads:

/etc/qcg/qcg-comp/app-scripts/config

configuration file for destination directory, where scripts should deployed.

Installation

To install QCG application scripts:

1. install qcg-appscripts RPM package
2. edit /etc/qcg/qcg-comp/app-scripts/config configuration file and set cluster_shared_path variable to point to a directory shared by all worker nodes
3. launch qcg-appscripts-deploy script
4. edit/create application configuration files in $cluster_shared_path/apps directory and refer to them in the QCG-Computing application mapping file (/etc/qcg/qcg-comp/application_mapfile)

Upgrade

After instalation of new QCG application scripts package:

1. launch qcg-appscripts-deploy script to update files in shared directory.

qcg-appscripts-deploy updates directories app-scripts, core, tools. apps directories (besides bash.app, common.app and template.app is not overwritten.

New application

To handle application, following elements must be provided:

• application configuration file (script in apps directory),
• application script (script in app-scripts directory),
• mapping in /etc/qcg/qcg-comp/application_mapfile file.

Application configuration file

The application configuration file is simple script that:

• loads the required modules or sets PATH/LD_LIBRARY_PATH variables,
• calls proper application script (from the app-scripts directory).

To create new application configuration apps/template.app file can be used as a base.

Example application configuration file for NAMD application:

#!/bin/bash
. $( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/common.app

module load namd/current
MPIEXEC=charmrun
MPIEXEC_ARGS="++verbose ++local +p`cat $QCG_NODEFILE |  wc -l`"

. $QCG_APP_ROOT_DIR/app-scripts/namd.qcg

Application script

The application script is responsible for:

• validation of input arguments,
• preparation of input data,
• application launch with proper parameters,
• optional post processing of output data/files.

Example application script file for NAMD application:

#!/bin/bash 

QCG_APP_NAME="namd"
. $QCG_APP_ROOT_DIR/core/core.qcg


if [ $# -ne 1 ]
then
    echo "Illegal number of arguments: $#"
    echo "Usage:"
    echo "namd.qcg NAMD-FILE"
    exit 1
fi

#QCG -> NAMD
MAIN_FILE="$1"
NAMD_PATH=`which namd2`

qcg_log_debug "main file     = $MAIN_FILE "
qcg_log_debug "using NAMD    = $NAMD_PATH "

qcg_app_wrapper ${MPIEXEC:-mpiexec} $MPIEXEC_ARGS $NAMD_PATH  $MAIN_FILE

Every application script should include core.qcg script which provides rich function library that can be used by application scripts. In the above simple example, application script only validates number of arguments and uses qcg_app_wrapper function to launch MPI version of the NAMD application. The qcg_app_wrapper function should always be used to launch application, as it also activates other QCG mechanism such as: user's pre-processing & post-processing scripts, application monitoring, output file notifications, mounting remote file systems etc.

Any modification/conversion of input file should be done in function qcg_pre_callback, for example:

function qcg_pre_callback {
    if [ -e "$BASH_SCRIPT" ]; then
            dos2unix $BASH_SCRIPT 2>> qcg.debug
    fi
}

if [ "$BASH_SCRIPT_ARGS" ]; then
    qcg_app_wrapper /bin/bash '$BASH_ARGS' -c \"'$BASH_SCRIPT $BASH_SCRIPT_ARGS'\"
else
    qcg_app_wrapper /bin/bash '$BASH_ARGS' -c '$BASH_SCRIPT'
fi

New application version

To create new version of already supported application, it is sufficient to:

1. copy existing application configuration file, eg:

   cp /opt/qcg-app-scripts/apps/namd.app /opt/qcg-app-scripts/apps/namd-6.2.app
   

2. change loaded module and/or environment variables in $cluster_shared_path/apps/namd-6.2.app
3. add new application version to the QCG application mapping file (/etc/qcg/qcg-comp/application_mapfile), eg:

   ...
   NAMD 6.2 /opt/qcg-app-scripts/apps/namd-6.2.app
   ...
   

Configuration

config/config.conf

• QCG_SCRATCH_DIR - the location of job temporary directory (e.g $TMPDIR) - OPTIONAL - default to $TMPDIR (if set) or "/tmp",
• QCG_NO_SCRATCH - if not set, application will be launched in QCG_SCRATCH_DIR
• QCG_DEBUG - if not set do not create qcg.debug file - OPTIONAL,
• MACHINE_FILE - the name of the environment variable pointing to the machine file (e.g. PBS_NODEFILE) - OPTIONAL.
• QCG_GROUP_DIR_ROOT - The root of groups home directory (PL-Grid only). Default: PLG_GROUPS_SHARED - OPTIONAL,
• QCG_NTF_URL - address of QCG-Notification broker service
• PLG_GROUPS_SHARED - same as QCG_GROUP_DIR_ROOT

{{{#!

Application Scripts - Developers view

This section is target at QCG developers rather than administrators, its list environment variables which influence job life cycle:

Common Input Environment Variables

• QCG_MODULES_LIST - list of environment modules to be loaded (separated by spaces)
• QCG_PREPROCESS - file name of the pre process script
• QCG_MONITOR - file name of the monitor script
• QCG_MONITOR_INTERVAL - determines how frequently the QCG_MONITOR script should be called (in seconds). - TBD
• QCG_NTF_CONSUMER_URL - the address of the Notification consumer interested in receiving notification about job output status change - TBD
• QCG_NTF_WATCH_PATTERN - the regular expression that should trigger notification - TBD
• QCG_NTF_WATCH_FILE - the file to be watched (default: stdout/err file) - TBD
• QCG_NTF_WATCH_INTERVAL - determines how often the watched file should be checked (in seconds) - TBD
• QCG_POSTPROCESS - file name of the post process script
• QCG_OUTERR_FILE - target filename of the joined output/error stream file (default: output.log)
• QCG_ZIPPED_INPUTS - set if the input files must be unziped first (e.g. QCG_ZIPPED_INPUTS=inputs.zip)
• QCG_ZIP_OUTPUTS - whether to zip results into $QCG_ZIP_OUTPUTS.zip file
• QCG_ZIP_OUTPUTS_FILTER - The wildcard pattern of files to be stored in the zip file (TBD)
• QCG_COMP_PROCESSES_MAP - topology of hybrid application (e.g. QCG_COMP_PROCESSES_MAP="4:1:1:1")
• QCG_FORCE_SCRATCH - force chdir to temporary directory

Application Scripts - Users view

Here we list environment variables available in any job submitted as QCG-Application (e.g. bash):

• QCG_NODEFILE - the machinefile like file with list of nodes allocated for the job
• QCG_PPN - number of processes per node
• QCG_SPN - number of allocated slots per node (usually equal to QCG_PPN, differ only if job was submitted with: #QCG node=a:b:c where b>c)
• QCG_PROCS - total number of proceses

}}}