Version 35 (modified by piontek, 10 years ago) (diff)

--

Application mapping

The QCG-Computing service allows to use abstract application names that are automatically mapped to the absolute paths of the wrapper scripts. The mapping is stored in the file:

/etc/qcg/qcg-comp/application_mapfile

The file has the following syntax:

APPLICATION-NAME VERSION SCRIPT-PATH

where VERSION can be an asterix (*) which means "any version". The mapping file is periodically reloaded (5 minutes by default), so there is no need to restart the qcg-compd service after updating the file. Example application_mapfile file:

MATLAB * /opt/qcg-app-scripts/apps/matlab.app
NAMD 6.2 /opt/qcg-app-scripts/apps/namd-6.2.app
bash * /opt/qcg-app-scripts/apps/bash.app
R * /opt/qcg-app-scripts/apps/R.app

The scripts for the most common applications are provided by the qcg-appscripts RPM package.

IMPORTANT: For the correct work QCG-Computing requires the mapping for the bash application, which is mandatory i.e.:

bash * /opt/qcg-app-scripts/apps/bash.app

Frequently Asked Question: Can I use simply mapping to /bin/bash

bash * /bin/bash

? The answer is "No". You must use the bash.qcg which does some additional work (e.g. setup environment variables like QCG_NODEFILE).

qcg-appscripts package

The qcg-appscripts package contains QCG application scripts for the most common applications. Role of the application scripts, apart from launching the proper application, is as follows:

  • loading the appropriate modules,
  • converting input files to UNIX character encoding,
  • setup environment,
  • launching user's helper scripts - preprocess, postprocess, assistent,
  • monitoring the execution of application through defined schemes or user scripts,
  • handling interactive jobs,
  • etc.

All files are installed in the directory:

    /usr/share/qcg-appscripts

The directory contains the following sub-directories:

  • apps - application configuration files,
  • app-scripts - application scripts,
  • config - QCG application scripts configuration,
  • core - QCG scripts library,
  • tools - QCG tools used by applications.

The administrator must create/edit application configuration files (apps) that contain specific to each cluster settings, such as:

  • name of the application module,
  • environment variables needed by application.

Files from apps directory should be referenced by QCG-Computing application mapping file (/etc/qcg/qcg-comp/application_mapfile).

The application scripts must be accessible by jobs running on cluster worker nodes. Thus directories apps, app-scripts, config, core and tools must be copied to directory shared by all worker nodes. This is done manually by the administrator by running:

/usr/sbin/qcg-appscripts-deploy

script included in the package. The script reads:

/etc/qcg/qcg-comp/app-scripts/config

configuration file for destination directory, where scripts should be deployed.

Installation

To install QCG application scripts:

  1. install (as root) the qcg-appscripts RPM package
    yum install qcg-appscripts
    
  2. edit /etc/qcg/qcg-comp/app-scripts/config configuration file and set cluster_shared_path variable to point to a directory shared by all worker nodes
  3. launch the qcg-appscripts-deploy script
  4. edit/create application configuration files in $cluster_shared_path/apps directory and refer to them in the QCG-Computing application mapping file (/etc/qcg/qcg-comp/application_mapfile)

Upgrade

After installation of new QCG application scripts package:

  1. launch qcg-appscripts-deploy script to update files in shared directory.

IMPORTANT: qcg-appscripts-deploy updates directories app-scripts, core, tools.

apps directories (besides bash.app, common.app and template.app) are not overwritten.

New application

To handle a application, the following elements must be provided:

  • application configuration file (script in apps directory),
  • application script (script in app-scripts directory),
  • mapping in /etc/qcg/qcg-comp/application_mapfile file.

Application configuration file

The application configuration file is a simple script that:

  • loads the required modules or sets PATH/LD_LIBRARY_PATH variables,
  • calls proper application script (from the app-scripts directory).

To create a new application configuration the apps/template.app file can be used as a base.

Example application configuration file for NAMD application:

#!/bin/bash
. $( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/common.app

module load namd/current
MPIEXEC=charmrun
MPIEXEC_ARGS="++verbose ++local +p`cat $QCG_NODEFILE |  wc -l`"

. $QCG_APP_ROOT_DIR/app-scripts/namd.qcg

Application script

The application script is responsible for:

  • validation of input arguments,
  • preparation of input data,
  • launching the application with proper parameters,
  • optional post processing of output data/files.

Example application script file for NAMD application:

#!/bin/bash 

QCG_APP_NAME="namd"
. $QCG_APP_ROOT_DIR/core/core.qcg


if [ $# -ne 1 ]
then
    echo "Illegal number of arguments: $#"
    echo "Usage:"
    echo "namd.qcg NAMD-FILE"
    exit 1
fi

#QCG -> NAMD
MAIN_FILE="$1"
NAMD_PATH=`which namd2`

qcg_log_debug "main file     = $MAIN_FILE "
qcg_log_debug "using NAMD    = $NAMD_PATH "

qcg_app_wrapper ${MPIEXEC:-mpiexec} $MPIEXEC_ARGS $NAMD_PATH  $MAIN_FILE

Every application script has to include the core.qcg script which provides rich function library that can be used by application scripts. In the above simple example, the application script only validates number of arguments and uses the qcg_app_wrapper function to launch MPI version of the NAMD application. The qcg_app_wrapper function should always be used to launch application, as it also activates other QCG mechanisms such as: user's pre-processing & post-processing scripts, application monitoring, output file notifications, mounting remote file systems etc.

Any modification/conversion of input files should be done in the qcg_pre_callback function, for example:

function qcg_pre_callback {
    if [ -e "$BASH_SCRIPT" ]; then
            dos2unix $BASH_SCRIPT 2>> qcg.debug
    fi
}

if [ "$BASH_SCRIPT_ARGS" ]; then
    qcg_app_wrapper /bin/bash '$BASH_ARGS' -c \"'$BASH_SCRIPT $BASH_SCRIPT_ARGS'\"
else
    qcg_app_wrapper /bin/bash '$BASH_ARGS' -c '$BASH_SCRIPT'
fi

New application version

To create a new version of the already supported application, it is sufficient:

  1. to copy the existing application configuration file, eg:
    cp /opt/qcg-app-scripts/apps/namd.app /opt/qcg-app-scripts/apps/namd-6.2.app
    
  2. to change the module to be loaded and/or environment variables in $cluster_shared_path/apps/namd-6.2.app
  3. to add a new application version to the QCG application mapping file (/etc/qcg/qcg-comp/application_mapfile), eg:
    ...
    NAMD 6.2 /opt/qcg-app-scripts/apps/namd-6.2.app
    ...
    

Configuration

config/config.conf

  • QCG_SCRATCH_DIR - the location of job temporary directory (e.g $TMPDIR) - OPTIONAL - default to $TMPDIR (if set) or "/tmp",
  • QCG_NO_SCRATCH - if not set, application will be launched in QCG_SCRATCH_DIR
  • QCG_DEBUG - if not set do not create qcg.debug file - OPTIONAL,
  • MACHINE_FILE - the name of the environment variable pointing to the machine file (e.g. PBS_NODEFILE) - OPTIONAL.
  • QCG_GROUP_DIR_ROOT - The root of groups home directory (PL-Grid only). Default: PLG_GROUPS_SHARED - OPTIONAL,
  • QCG_NTF_URL - address of QCG-Notification broker service
  • PLG_GROUPS_SHARED - same as QCG_GROUP_DIR_ROOT