QCG Broker Installation

The QosCosGrid Resource Management System (QCG-Broker) is an open source meta-scheduling system, which allows developers to build and deploy resource management systems for large scale distributed computing infrastructures. The QCG-Broker, based on dynamic resource selection, mapping and advanced scheduling methodology, combined with feedback control architecture, deals with dynamic Grid environment and resource management challenges, e.g. load-balancing among clusters, remote job control or file staging support. Therefore, the main goal of the QCG-Broker is to manage the whole process of remote job submission to various batch queuing systems, clusters or resources. It has been designed as an independent core component for resource management processes which can take advantage of various low-level Core Services and existing technologies. Finally, the QCG-Broker can be considered as a robust system which provides abstraction of the complex grid infrastructure as well as a toolbox which helps to form and adapts to distributing computing environments.

Requirements

• Package dependencies - All packages needed to install and run QCG-Broker service are installed automatically by the YUM Package Manager.
• certificates - set of CA certificates has to be installed to allow proper mutual authentication between client and service.
• range of opened ports - QCG-Broker needs set of opened ports to be able to receive notifications about changes of jobs and tasks statuses and to communicate with gridFTP server. Ports must be opened between broker site and sites where QCG-Notification services are installed. For the proper work just several opened ports are enough.

Preparation of the environment

Host configuration

•  Disable the IPv6 if you do not need it.

Installation of CA certificates

To allow mutual authentication between service and clients please install the set of certificates of trusted Certification Authorities. If the system does not have CA certificates already installed you can:

• install them globally following the   CA certificates installation guide
• configure QCG-Broker to use its own list of trusted CA certificates copping selected ones into the ~/.globus/certificates directory.

Software repositories

To install QCG-Broker service please configure the system with   QCG Software Repositories.

Installation

IMPORTANT: Perform the installation as the administrator (root user).

IMPORTANT: A new account (qcg-broker) will be created during the installation.

• install QCG-Broker service and all dependencies using YUM Package Manager:

  yum install qcg-broker
  

IMPORTANT For Scientific Linux 5.x: Due to a bug in this version of SL, it might be necessary to install the package xml-commons-apis by hand:

yum install xml-commons-apis

Configuration

Database setup

• For Scientific Linux 6.x: If PostgreSQL service has not been initialized (e.g. has been installed as dependency of qcg-broker package), perform its initialization before starting it:

  su -
  service postgresql initdb
  

• Start the PostgreSQL service:

  su -
  service postgresql start
  

• Create new database user (e.g. qcg-broker) authenticated via password (the same password must be later used to setup QCG-Broker deployment):

  su - postgres
  createuser -P qcg-broker
  
  Enter password for new role: 
  Enter it again: 
  Shall the new role be a superuser? (y/n) n
  Shall the new role be allowed to create databases? (y/n) y
  Shall the new role be allowed to create more new roles? (y/n) n
  CREATE ROLE
  
  exit
  

• Create new database (e.g. qcg-broker) owned by the qcg-broker user:

  su - qcg-broker
  createdb -U qcg-broker qcg-broker
  exit
  

  Note: The qcg-broker user must be allowed to create new databases.

• Depending on the local PostgreSQL configuration you may need to edit the pg_hba.conf file (host based authentication configuration file) to enable password authentication to the qcg-broker database for the qcg-broker user.

For SL5/SL6: edit the /var/lib/pgsql/data/pg_hba.conf file.

If the PostgreSQL installation is fully dedicated for QCG-Broker replace the default configuration with rules (lines) given below, otherwise insert them to the configuration file in a proper place:

local   qcg-broker   qcg-broker                         md5
host    qcg-broker   qcg-broker   127.0.0.1/32          md5

Note: Please reload the PostgreSQL server in order to make the changes visible e.g:

service postgresql reload

QCG Broker setup

Note: All changes should be made as 'qcg-broker' user:

su - qcg-broker

• configure the QCG-Broker instance - all configuration variables are placed in the /etc/qcg/qcg-broker/global-config.prop file
  • deploy.dir - directory where QCG-Broker was installed (/usr/share/qcg/qcg-broker/qcg-broker). Please do not modify this property.
  • var.dir - directory where QCG-Broker holds temporary files (/var/lib/qcg/qcg-broker). Please do not modify this property.
  • log.dir - directory where QCG-Broker holds log files (/var/log/qcg/qcg-broker). Please do not modify this property.
  • deploy.certs.dir - directory with QCG-Broker credentials (must contain qcg-brokercert.pem and qcg-brokerkey.pem files)
  • deploy.cert.dn distinguish name of QCG-Broker credential
  • deploy.db.user PostgreSQL user name
  • deploy.db.pwd Password for PostgreSQL user
  • deploy.db.host PostgreSQL database host name (127.0.0.1 by default. If PostgreSQL service is on the same host as QCG-Broker, this property should not be changed. Otherwise some changes in PostgreSQL configuration (pg_hba.conf file) may be needed).
  • deploy.db.name PostgreSQL database name
  • deploy.https_port the port which QCG-Broker will be listening for https requests on
  • deploy.rmi_port the port needed for internal communication between QCG-Broker components.
  • deploy.shutdown_port the port which the tomcat will be listening for shutdown requests on
  • deploy.port_range the range of opened ports
  • deploy.admin.email separated by commas list of email addresses of qcg-broker administrators. All notification messages produced by scripts checking if the service is working correctly will be sent to this list.

• deploy the configuration and setup the QCG-Broker database

  /usr/share/qcg/qcg-broker/config.sh initdb
  

  IMPORTANT: The initdb command of the /usr/share/qcg/qcg-broker/config.sh script modifies QCG-Broker configuration files as well as clears QCG-Broker database. It's not recommended to run this script with initdb command on already configured and running QCG-Broker instance.

Configuring the range of opened ports

QCG-Broker needs set of opened ports for receiving notifications and for communication with gridFTP services. To configure the range of opened ports please do changing the port range to correspond to configuration of your environment:

mkdir -p ~/.globus
echo "tcp.port.range=20000,25000" > ~/.globus/cog.properties

QCG Broker resource providers setup

Note: All changes should be made as 'qcg-broker' user:

su - qcg-broker

• Edit the /etc/qcg/qcg-broker/providers.prop file specifying list of resource providers (instances of QCG BES/AR services controlling resources and creating together virtual pool of resources)

providers.count=<n> - number of providers

For x=0 to n-1 define locations of domain providers:

provider.x.domain=<DOMAIN> - name of the administrative domain, which resources controlled by the given provider belong to,
provider.x.type=qcg - type of the provider, DO NOT CHANGE THIS VALUE,
provider.x.id=<PROVIDER ID> - identifier of the provider (must be globally unique),
provider.x.location=<LOCATION> - address of provider service.
provider.x.storage=<GRIDFTP> - address of the gridFTP server
provider.x.hostname=<LIST OF NAMES> - list of names of the site separated by comma
provider.x.props=<PROPERTIES> - list of properties separated by comma. For the PLGrid sites please set the "plgrid" property
provider.x.active=<STATE> - optional parameter (true|false) determining the availability of the provider

For example:

# how many providers should be used for testing
providers.count=1

provider.0.domain=PCSS
provider.0.id=reef
provider.0.type=qcg
provider.0.location=httpg://qcg.reef.man.poznan.pl:19000
provider.0.storage=qcg.reef.man.poznan.pl
provider.0.hostname=reef.man.poznan.pl,reef,pcss
provider.0.props=plgrid
provider.0.active=true

QCG Broker advance reservations setup (optional)

Note: All changes should be made as 'qcg-broker' user:

su - qcg-broker

To allow QCG-Broker to create and manage advance reservations on resources accessible via  QCG BES/AR service some additional configuration steps are needed on Administrative Domain level:

• the QCG-Broker credential must be mapped (in the grid-mapfile file) to the local user that is authorized to create and manage reservations. Usually the LRMS administrator user (like sgeadmin in SGE or lsfadmin in LSF) has sufficient rights, e.g.:

  "/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg.man.poznan.pl" sgeadmin
  

QCG Broker notifications setup (optional)

Note: All changes should be made as 'qcg-broker' user:

su - qcg-broker

To allow QCG-Broker to send notifications about user's task status change, edit /usr/share/qcg/qcg-broker/broker/etc/config.prop, and set variable:

broker.notificationbroker.url

to point to QCG-Notification service.

See  http://www.qoscosgrid.org/trac/qcg-notification#Installation for detailed information about installation of QCG-Notification service.

Firewall configuration

The following ports must be open in the firewall:

• QCG Broker listen port (8443 by default)
• GridFTP port - 2811
• defined port range

Starting the service

As the root user type:

service qcg-broker start

Checking the service status

As the qcg-broker user type:

service qcg-broker status

Stopping the service

As the qcg-broker user type:

service qcg-broker stop

Veryfing the installation

To verify the installation please use QCG-Broker Client User Guide.

License

For license details please visit the  QCG License page.