Version 35 (modified by piontek, 11 years ago) (diff)

--

QCG Broker Installation

The QosCosGrid Resource Management System (QCG-Broker) is an open source meta-scheduling system, which allows developers to build and deploy resource management systems for large scale distributed computing infrastructures. The QCG-Broker, based on dynamic resource selection, mapping and advanced scheduling methodology, combined with feedback control architecture, deals with dynamic Grid environment and resource management challenges, e.g. load-balancing among clusters, remote job control or file staging support. Therefore, the main goal of the QCG-Broker is to manage the whole process of remote job submission to various batch queuing systems, clusters or resources. It has been designed as an independent core component for resource management processes which can take advantage of various low-level Core Services and existing technologies. Finally, the QCG-Broker can be considered as a robust system which provides abstraction of the complex grid infrastructure as well as a toolbox which helps to form and adapts to distributing computing environments.

Requirements

  • Package dependencies - The postgres-server package must me installed prior to qcg-broker package. Rest of dependencies can be installed before the qcg-broker installation or will be automatically installed by YUM Package Manager.
    • Java (>= 1.5)

      For SL5.x

      yum install java-1.6.0-sun-compat.x86_64
      
    • PostgreSQL (>= 7.0) - configured to allow TCP/IP connections from localhost (default configuration)

      For SL5.x

      yum install postgresql-server.x86_64
      
    • Apache Ant (>= 1.6) ("Optional tasks for ant" are required. For SL5.x the ant-nodeps.x86_64 package must be installed)

      For SL5.x

      yum install ant.x86_64 ant-nodeps.x86_64
      
    • xml-commons-apis - installation of this package is not mandatory, but is recommended to avoid confusing information about not important errors.

      For SL5.x

      yum install xml-commons.x86_64 xml-commons-apis.x86_64
      
  • certificates - set of CA certificates has to be installed to allow proper mutual authentication between client and service.
  • range of opened ports

QCG-Broker needs set of opened ports to be able to receive notifications about changes of jobs and tasks statuses and to communicate with gridFTP server.

Preparing the environment

  • create the qcg-broker user (as a root)

IMPORTANT: Altought it's possible to run QCG-Broker service on any account (root, regular user) we strongly recommend to run QCG-Broker service on the unpriledged account designated specifically for this service (qcg-broker for example).

We recommend to create a new user (e.g. qcg-broker) that will be used by the service to run in unprivileged mode

useradd -m -s /bin/bash qcg-broker 
  • define the set of trusted Certificate Authorities [optional step]

If you want to configure QCG-Broker to use its own list of trusted CA certificates, please do not forget to copy them into ~/.globus/certificates directory.

  • specify the range of opened ports

The port range must be set in ~/.globus/cog.properties file.

su - qcg-broker
mkdir -p ~/.globus
echo "tcp.port.range=5000,5100" >> ~/.globus/cog.properties

Database setup

  • Start the PostgreSQL service:

    For SL5.x

    su -
    service postgresql start
    
  • Create new database user (e.g. qcg-broker) authenticated via password (the same password must be later used to setup QCG-Broker deployment):
    su - postgres
    createuser -P qcg-broker
    
    Enter password for new role: 
    Enter it again: 
    Shall the new role be a superuser? (y/n) n
    Shall the new role be allowed to create databases? (y/n) y
    Shall the new role be allowed to create more new roles? (y/n) n
    CREATE ROLE
    
    exit
    
  • Create new database (e.g. qcg-broker) owned by the qcg-broker user:
    su - qcg-broker
    createdb -U qcg-broker qcg-broker
    exit
    
    Note: The qcg-broker user must be allowed to create new databases.
  • Depending on the local PostgreSQL configuration you may need to edit the pg_hba.conf file (host based authentication configuration file) to enable password authentication to the qcg-broker database for the qcg-broker user.

For SL5.x: edit the /var/lib/pgsql/data/pg_hba.conf file.

If the PostgreSQL installation is fully dedicated for QCG-Broker replace the default configuration with rules (lines) given below, otherwise insert them to the configuration file in a proper place:

local   qcg-broker   qcg-broker                         md5
host    qcg-broker   qcg-broker   127.0.0.1/32          md5

Note: You must reload the PostgreSQL server in order to make the changes visible e.g:

service postgresql reload

Installation

IMPORTANT: Perform the installation as the administrator (root user).

To install QCG-Broker service please configure the system with   QCG Software Repositories.

  • install QCG-Broker service using YUM Package Manager:
    yum install qcg-broker
    
  • configure the QCG-Broker instance - all configuration variables are placed in the /opt/plgrid/qcg/share/qcg-broker/deploy.prop file
    • deploy.dir - directory where QCG-Broker was installed (/opt/plgrid/qcg/share/qcg-broker). Please do not modify this property.
    • deploy.certs.dir - directory with QCG-Broker credentials (must contain qcg-brokercert.pem and qcg-brokerkey.pem files)
    • deploy.cert.dn distinguish name of QCG-Broker credential
    • deploy.db.user PostgreSQL user name
    • deploy.db.pwd Password for PostgreSQL user
    • deploy.db.host PostgreSQL database host name (127.0.0.1 by default. If PostgreSQL service is on the same host as QCG-Broker, this property should not be changed. Otherwise some changes in PostgreSQL configuration (pg_hba.conf file) may be needed).
    • deploy.db.name PostgreSQL database name
    • deploy.https_port the port which QCG-Broker will be listening for https requests on
    • deploy.rmi_port the port needed for internal communication between QCG-Broker components.
    • deploy.shutdown_port the port which the tomcat will be listening for shutdown requests on
    • deploy.port_range the range of opened ports
  • deploy the configuration and setup the database
    /opt/plgrid/qcg/share/qcg-broker/config.sh
    

Configuration

  • Edit the QCG-BROKER-DEPLOY_DIR/broker/etc/providers.list file specyfying list of resource providers (instances of QCG BES/AR services controling resources and creating together virtual pool of resources)

providers.count=<n> - number of providers

For x=0 to n-1 define locations of domain providers:

provider.x.domain=<DOMAIN> - name of the administrative domain, which resources controlled by the given provider belong to,
provider.x.type=qcg - type of the provider, DO NOT CHANGE THIS VALUE,
provider.x.id=<PROVIDER ID> - identifier of the provider (must be globally unique),
provider.x.location=<LOCATION> - address of provider service.

For example:

# how many providers should be used for testing
providers.count=1

provider.0.domain=PSNC
provider.0.id=qcg_cluster
provider.0.type=qcg
provider.0.location=httpg://qcg.man.poznan.pl:19000
  • For cross-clusters job submission, edit QCG-BROKER_DEPLOY_DIR/broker/etc/config.prop file, and set following variables:
    • broker.qoscos.proactive.pncURL - ProActvie node coordinator address
    • broker.qoscos.ompi.pncURL - OpenMPI node coordinator address

See  QCG-ProActive Coordinator,  QCG-OpenMPI Coordinator for detailed information about installation of ProActive and OpenMPI node coordinators.

Resource Providers setup

To allow QCG-Broker to create and manage advance reservations on resources accessible via  QCG BES/AR service some additional configuration steps are needed on Administrative Domain level:

  • the QCG-Broker credential must be mapped (in the grid-mapfile file) to the local user that is authorized to create and manage reservations. Usually the LRMS administrator user (like sgeadmin in SGE or lsfadmin in LSF) has sufficient rights, e.g.:
    "/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg.man.poznan.pl" sgeadmin
    

Starting the service

As the qcg-broker user type:

cd /opt/plgrid/qcg/share/qcg-broker
./broker/bin/start-broker.sh
export JAVA_HOME=/usr/lib/jvm/java; ./service/bin/startup.sh

IMPORTANT: Do not forget to set JAVA_HOME or JRE_HOME environment variables.

Stopping the service

As the qcg-broker user type:

cd /opt/plgrid/qcg/share/qcg-broker
./broker/bin/stop-broker.sh
export JAVA_HOME=/usr/lib/jvm/java; ./service/bin/shutdown.sh

Veryfing the installation

To verify the installation please use QCG-Broker Client User Guide.