Version 56 (modified by mmamonski, 11 years ago) (diff)

--

  1. Introduction
  2. Prerequisites
    1. GridFTP server
  3. Firewall configuration
  4. Related software
  5. Installation
  6. Service certificates
  7. DRMAA library
    1. Torque/PBS Professional
    2. SLURM
  8. Service configuration
  9. Creating applications' script space
  10. Note on the security model
  11. Starting the service
  12. Stopping the service
  13. Verifying the installation

* !!! Warning !!! *

This page has been obsoleted by the general Installation Guide.

Introduction

QCG-Computing service (the successor of the OpenDSP project) is an open source service acting as a computing provider exposing on demand access to computing resources and jobs over the HPC Basic Profile compliant Web Services interface. In addition the QCG-Computing offers remote interface for Advance Reservations management.

This document describes installation of the QCG-Computing service on Debian machines using binary packages. The service should be deployed on the machine (or virtual machine) that:

  • has at least 1GB of memory (recommended value: 2 GB)
  • has 10 GB of free disk space (most of the space will be used by the log files)
  • has any modern CPU (if you plan to use virtual machine you should dedicated to it one or two cores from the host machine)
  • runs DEBIAN 6.X

Prerequisites

We assume that you have the local batch systems already installed.

The QosCosGrid services do not require from you to install any QCG component on the worker nodes, however application wrapper scripts need the following software to be available on worker nodes:

  • bash,
  • rsync,
  • zip/unzip,
  • dos2unix,
  • nc,
  • python.

Which are usually available out of the box on most of the HPC systems.

GridFTP server

To be fully operable the QosCosGrid stack requires the GridFTP server to be installed. This requirements is usually fulfilled by most PRACE sites. If not it can be easily installed by issuing the following commands: 

# apt-get install xinetd globus-gridftp-server-progs 
# cat > /etc/xinetd.d/gsiftp << EOF
service gsiftp
{
 instances               = 100
 socket_type             = stream
 wait                    = no
 user                    = root
 env                     += GLOBUS_TCP_PORT_RANGE=20000,25000
 server = /usr/sbin/globus-gridftp-server
 server_args = -i -aa -l /var/log/globus-gridftp.log
 server_args += -d WARN
 log_on_success          += DURATION
 nice                    = 10
 disable                 = no
}
EOF
# /etc/init.d/xinetd reload
Reloading internet superserver configuration: xinetd.

Firewall configuration

In order to expose the QosCosGrid services externally you need to open the following incoming ports in the firewall:

  • 19000 (TCP) - QCG-Computing
  • 19001 (TCP) - QCG-Notification
  • 2811 (TCP) - GridFTP server
  • 20000-25000 (TCP) - GridFTP port-range

The following outgoing trafic should be allowed in general:

  • NTP, DNS, HTTP, HTTPS services
  • gridftp (TCP ports: 2811 and port-ranges: 9000-9500, 20000-25000)

Related software

  • Install database backend (PostgresSQL) - optional, only if you want to host the QCG-Computing database on the same machine. 
    apt-get install postgresql
  • UnixODBC and the PostgresSQL ODBC driver: 
    apt-get install unixodbc odbc-postgresql

Moreover we further assume that the X.509 host certificate and key are already installed in the following locations:

  • /etc/grid-security/hostcert.pem
  • /etc/grid-security/hostkey.pem

Most of the grid services and security infrastructures are sensitive to time skews. Thus we recommended to install a Network Time Protocol daemon or use any other solution that provides accurate clock synchronization.

Installation

The one who want to install QCG-Computing on Debian should follow these steps:

  • ensure that the qcg-comp user is present in a system, otherwise create it: 
    useradd -r -d /var/log/qcg-comp/ qcg-comp
  • ensure that the qcg-dev group is present in a system, otherwise create it: 
    groupadd -r qcg-dev
  • install the QosCosGrid Debian repository: 
    cat > /etc/apt/sources.list.d/qcg.unstable.list << EOF
deb http://fury.man.poznan.pl/qcg-packages/debian/ unstable main
EOF
  • add the public key of the QCG repository to your trusted keys in the apt configuration: 
    wget https://apps.man.poznan.pl/trac/qcg-notification/raw-attachment/wiki/InstallingUsingDeb/qcg.pub
apt-key add qcg.pub
  • refresh the packages list: 
    apt-get update
  • install QCG-Computing: 
    apt-get install qcg-comp qcg-comp-client qcg-comp-doc
  • setup the QCG-Computing database as described  here.

Service certificates

Copy the service certificate and key into the /etc/qcg-comp/certs/. Remember to set appropriate rights to the key file. 

cp /etc/grid-security/hostcert.pem /etc/qcg-comp/certs/qcgcert.pem
cp /etc/grid-security/hostkey.pem /etc/qcg-comp/certs/qcgkey.pem
chown qcg-comp /etc/qcg-comp/certs/qcgcert.pem
chown qcg-comp /etc/qcg-comp/certs/qcgkey.pem 
chmod 0600 /etc/qcg-comp/certs/qcgkey.pem

DRMAA library

Torque/PBS Professional

Install DRMAA for Torque/PBS Pro using source package available at  PBS DRMAA home page

SLURM

Install DRMAA for SLURM using source package available at  SLURM DRMAA home page. 

# install SLURM headers files
apt-get install libslurm21-dev
# get SLURM DRMAA
wget http://apps.man.poznan.pl/trac/slurm-drmaa/downloads/slurm-drmaa-1.0.5.tar.gz
tar -xzf slurm-drmaa-1.0.5.tar.gz
cd slurm-drmaa-1.0.5
# configure, make and install (by default DRMAA should be installed into /usr/local/
./configure
make 
make install
# test it!
 /usr/local/bin/drmaa-run /bin/hostname

Service configuration

Edit the preinstalled service configuration file (/etc/qcg-comp/qcg-compd.xml): 

<?xml version="1.0" encoding="UTF-8"?>
<sm:QCGCore
	xmlns:sm="http://schemas.qoscosgrid.org/core/2011/04/config"
	xmlns="http://schemas.qoscosgrid.org/comp/2011/04/config"
	xmlns:smc="http://schemas.qoscosgrid.org/comp/2011/04/config"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	
	<Configuration>
		<sm:ModuleManager>
			<sm:Directory>/usr/lib/qcg-core/modules/</sm:Directory>
			<sm:Directory>/usr/lib/qcg-comp/modules/</sm:Directory>
		</sm:ModuleManager>

		<sm:Service xsi:type="qcg-compd" description="QCG Computing">
			<sm:Logger>
				<sm:Filename>/var/log/qcg-comp/qcg-compd.log</sm:Filename>
				<sm:Level>INFO</sm:Level>
			</sm:Logger>

			<sm:Transport>
				<sm:Module xsi:type="sm:ecm_gsoap.service">
					<sm:Host>localhost</sm:Host>
					<sm:Port>19000</sm:Port>
				</sm:Module>
				<sm:Module xsi:type="smc:qcg-comp-service"/>
			</sm:Transport>
			
			<sm:Authentication>
        <sm:Module xsi:type="sm:atc_transport_gsi.service">
                         <sm:X509CertFile>/etc/qcg-comp/certs/qcgcert.pem</sm:X509CertFile>
                         <sm:X509KeyFile>/etc/qcg-comp/certs/qcgkey.pem</sm:X509KeyFile>
        </sm:Module>
			</sm:Authentication>

      <sm:Authorization>
        <sm:Module xsi:type="sm:atz_mapfile">
          <sm:Mapfile>/etc/grid-security/grid-mapfile</sm:Mapfile>
        </sm:Module>
      </sm:Authorization>


			<sm:Module xsi:type="submission_drmaa" path="/usr/local/lib/libdrmaa.so"/>

      <!-- The jsdl filter module - uncomment module appropriate for your batch system -->
      <!-- sm:Module xsi:type="pbs_jsdl_filter"/-->
      <!-- sm:Module xsi:type="sge_jsdl_filter"/-->
      <!-- sm:Module xsi:type="slurm_jsdl_filter"/-->
      <!-- sm:Module xsi:type="lsf_jsdl_filter"/-->

      <!-- The reservation module - uncomment module appropriate for your batch/scheduler system -->
			<!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_sge.py"/-->
			<!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_maui.py"/-->
			<!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_moab.py"/-->
			<!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_pbs.py"/-->
			<!--sm:Module xsi:type="reservation_python" path="/usr/lib/qcg-comp/modules/python/reservation_slurm.py"/-->
      <sm:Module xsi:type="atz_ardl_filter"/>

      <sm:Module xsi:type="sm:general_python" path="/usr/lib/qcg-comp/modules/python/monitoring.py"/>
			
			<sm:Module xsi:type="notification_wsn">
				<sm:Module xsi:type="sm:ecm_gsoap.client" >
						<sm:ServiceURL>http://localhost:19001/</sm:ServiceURL>
							<sm:Authentication>
								<sm:Module xsi:type="sm:atc_transport_http.client"/>
							</sm:Authentication>
						<sm:Module xsi:type="sm:ntf_client"/>
				</sm:Module>
			</sm:Module>
				
			<sm:Module xsi:type="application_mapper">
				<ApplicationMapFile>/etc/qcg-comp/application_mapfile</ApplicationMapFile>
			</sm:Module>

			<Database>
				<DSN>qcg-comp</DSN>
				<User>qcg-comp</User>
				<Password>qcg-comp</Password>
			</Database>

			<UnprivilegedUser>qcg-comp</UnprivilegedUser>

			<FactoryAttributes>
				<CommonName>IT cluster</CommonName>
				<LongDescription>IT department cluster for public use</LongDescription>
			</FactoryAttributes>
		</sm:Service>

	</Configuration>
</sm:QCGCore>

In most cases it should be enough to change only following elements:

Transport/Module/Host
the hostname of the machine where the service is deployed
Transport/Module/Authentication/Module/X509CertFile and Transport/Module/Authentication/Module/X509KeyFile
the service private key and X.509 certificate. Make sure that the key and certificate is owned by the qcg-comp user. If you installed cert and key file in the recommended location you do not need to edit these fields.
Module[type="smc:notification_wsn"]/PublishedBrokerURL
the external URL of the QCG-Notification service (You can do it later, i.e. after  installing the QCG-Notification service)
Module[type="smc:notification_wsn"]/Module/ServiceURL
the localhost URL of the QCG-Notification service (You can do it later, i.e. after  installing the QCG-Notification service)
Module[type="submission_drmaa"]/@path
path to the DRMAA library (the libdrmaa.so). Also, if you installed the DRMAA library using provided SRC RPM you do not need to change this path.
Database/Password
the qcg-comp database password
UseScratch
set this to true if you set QCG_SCRATCH_DIR_ROOT in sysconfig so any job will be started from scratch directory (instead of the default home directory)
FactoryAttributes/CommonName
a common name of the cluster (e.g. reef.man.poznan.pl). You can use any name that is unique among all systems (e.g. cluster name + domain name of your institution)
FactoryAttributes/LongDescription
a human readable description of the cluster

Moreover remember to uncomment jsdl_filter and reservation_python modules (appropriate for your batch system).

Creating applications' script space

A common case for the QCG-Computing service is that an application is accessed using abstract app name rather than specifying absolute executable path. The application name/version to executbale path mappings are stored in the file /etc/qcg-comp/application_mapfile: 

cat /etc/qcg-comp/application_mapfile
# ApplicationName ApplicationVersion Executable

date * /bin/date
LPSolve 5.5 /usr/local/bin/lp_solve

It is also common to provide here wrapper scripts rather than target executables. The wrapper script can handle such aspects of the application lifetime like: environment initialization, copying files from/to scratch storage and application monitoring. It is recommended to create separate directory for those wrapper scripts (e.g. the application partition) for an applications and add write permission to them to the QCG Developers group. This directory must be readable by all users and from every worker node (the application partition usually fullfils those requirements). 

mkdir /opt/exp_soft/qcg-app-scripts
chown :qcg-dev /opt/exp_soft/qcg-app-scripts
chmod g+rwx /opt/exp_soft/qcg-app-scripts

More on Application Scripts.

Note on the security model

The QCG-Computing can be configured with various authentication and authorization modules. However in the typical deployment we assume that the QCG-Computing is configured as in the above example, i.e.:

  • authentication is provided on basics of httpg protocol,
  • authorization is based on the local grid-mapfile mapfile.

Starting the service

As root type: 

/etc/init.d/qcg-comp start

The service logs can be found in: 

/var/log/qcg-comp/qcg-compd.log

Stopping the service

The service can be stopped using the following command: 

/etc/init.d/qcg-comp stop

Verifying the installation

  • Edit the QCG-Computing client configuration file (/etc/qcg-comp/qcg-comp.xml):
    • set the Host and Port to reflects the changes in the service configuration file (qcg-compd.xml). 
      <?xml version="1.0" encoding="UTF-8"?>
<sm:QCGCore
       xmlns:sm="http://schemas.qoscosgrid.org/core/2011/04/config"
       xmlns="http://schemas.qoscosgrid.org/comp/2011/04/config"
       xmlns:smc="http://schemas.qoscosgrid.org/comp/2011/04/config"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  
       <Configuration>
               <sm:ModuleManager>
                       <sm:Directory>/opt/qcg/lib/qcg-core/modules/</sm:Directory>
                       <sm:Directory>/opt/qcg/lib/qcg-comp/modules/</sm:Directory>
               </sm:ModuleManager>
 
               <sm:Client xsi:type="qcg-comp" description="QCG-Computing client">
                       <sm:Transport>
                               <sm:Module xsi:type="sm:ecm_gsoap.client">
                                       <sm:ServiceURL>httpg://frontend.example.com:19000/</sm:ServiceURL>
                                       <sm:Authentication>
                                               <sm:Module xsi:type="sm:atc_transport_gsi.client"/>
                                       </sm:Authentication>
                                       <sm:Module xsi:type="smc:qcg-comp-client"/>
                               </sm:Module>
                       </sm:Transport>
               </sm:Client>
       </Configuration>
</sm:qcgCore>
  • Initialize your credentials: 
    grid-proxy-init -rfc
Your identity: /O=Grid/OU=QosCosGrid/OU=PSNC/CN=Mariusz Mamonski
Enter GRID pass phrase for this identity:
Creating proxy .................................................................. Done
Your proxy is valid until: Wed Apr  6 05:01:02 2012
  • Query the QCG-Computing service: 
    qcg-comp -G | xmllint --format - # the xmllint is used only to present the result in more pleasant way
  
<bes-factory:FactoryResourceAttributesDocument xmlns:bes-factory="http://schemas.ggf.org/bes/2006/08/bes-factory">
    <bes-factory:IsAcceptingNewActivities>true</bes-factory:IsAcceptingNewActivities>
    <bes-factory:CommonName>IT cluster</bes-factory:CommonName>
    <bes-factory:LongDescription>IT department cluster for public   use</bes-factory:LongDescription>
    <bes-factory:TotalNumberOfActivities>0</bes-factory:TotalNumberOfActivities>
    <bes-factory:TotalNumberOfContainedResources>1</bes-factory:TotalNumberOfContainedResources>
    <bes-factory:ContainedResource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="bes-factory:BasicResourceAttributesDocumentType">
        <bes-factory:ResourceName>worker.example.com</bes-factory:ResourceName>
        <bes-factory:CPUArchitecture>
            <jsdl:CPUArchitectureName xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl">x86_32</jsdl:CPUArchitectureName>
        </bes-factory:CPUArchitecture>
        <bes-factory:CPUCount>4</bes-factory:CPUCount><bes-factory:PhysicalMemory>1073741824</bes-factory:PhysicalMemory>
    </bes-factory:ContainedResource>
    <bes-factory:NamingProfile>http://schemas.ggf.org/bes/2006/08/bes/naming/BasicWSAddressing</bes-factory:NamingProfile> 
    <bes-factory:BESExtension>http://schemas.ogf.org/hpcp/2007/01/bp/BasicFilter</bes-  factory:BESExtension>
    <bes-factory:BESExtension>http://schemas.qoscosgrid.org/comp/2011/04</bes-factory:BESExtension>
    <bes-factory:LocalResourceManagerType>http://example.com/SunGridEngine</bes-factory:LocalResourceManagerType>
    <smcf:NotificationProviderURL xmlns:smcf="http://schemas.qoscosgrid.org/comp/2011/04/factory">http://localhost:2211/</smcf:NotificationProviderURL>
</bes-factory:FactoryResourceAttributesDocument>
  • Submit a sample job: 
    qcg-comp -c -J /usr/share/doc/qcg-comp-doc/examples/date.xml
Activity Id: ccb6b04a-887b-4027-633f-412375559d73
  • Query it status: 
    qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73
status = Executing
qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73
status = Executing
qcg-comp -s -a ccb6b04a-887b-4027-633f-412375559d73
status = Finished
exit status = 0