Version 87 (modified by bartek, 9 years ago) (diff)

--

QosCosGrid

The QosCosGrid (QCG) middleware is an integrated e-infrastructure offering advanced job and resource management capabilities to deliver to end-users supercomputer-like performance and structure. By connecting many computing clusters together, QosCosGrid offers easy-to-use mapping, execution and monitoring capabilities for variety of applications, such as parameter sweep, workflows, MPI or hybrid MPI-OpenMP. Thanks to QosCosGrid, large-scale and complex computing models written in Fortran, C, C++ or Java can be automatically distributed over a network of computing resources with guaranteed Quality of Service. Consequently, applications can be run at given periods of time, their execution time and waiting times can be reduced, and thus bigger problem instances can be considered.

News

QosCosGrid in Mapper

April 2011

It was decided to use the QosCosGrid middleware to multi-scale computations in the Mapper project. Installations of the QosCosGrid middleware at production sites in Europe, including EGI and PRACE machines, are currently being discussed.

QCG release 2.0

16 April 2011

We are happy to introduce a next relase of QosCosGrid. This is the first release after finishing the QosCosGrid project and takes a number 2.0.

QosCosGrid middleware deployed in PL-Grid

March - April 2011

QosCosGrid was successfully deployed on the PL-Grid infrastructure in Poland. Currently the QCG middleware is available in Poznan Supercomputing and Networking Center as well as in AGH Cyfronet in Krakow. Further deployment are planned.

QosCosGrid project page moved

March 2010

After the end of QosCosGrid project, the old www.qoscosgrid.org page was moved to the  archive. The new page is strictly related to the QosCosGrid middleware and further QosCosGrid sollutions.

QCG Middleware

Main features

  • the most efficient remote access to computational resources in a single cluster or many clusters in Poland and Europe,
  • automatic steering of various types of complex computing experiments ranging from multi-parameter sweep studies to cross-cluster executions of parallel applications,
  • fully transparent integration with parallel programming and execution environments like OpenMPI and ProActive located on many computing clusters,
  • support for Quality of Service (e.g. start time) based on advance reservation mechanisms,
  • shorter waiting times and improved resource utilization by hierarchical grid- and local-level job scheduling,
  • management of input and output files in distributed computing clusters,
  • efficient integration between services and queuing systems ensuring high performance and reliability of the overall system,
  • extensible, open and standard based architecture supporting OGF DRMAA, JSDL, BES and HPC Profile with pluggable modules,
  • secure communication channels using transport level (SSL/TLS, X.509) and message level (SAML2.0) mechanisms,
  • delivered together with command-line, graphical, web-based or even mobile phone tools for end users and administrators,
  • fast and reliable installation procedures.

Architecture and Main Components

The following diagram presents a general architecture of the QosCosGrid middleware. Information about individual components is available by clicking on the corresponding objects.

Cross-cluster communication QCG Science Gateways QCG Data Movement QCG Data Movement QCG Broker QCG BES/AR QCG BES/AR QCG-Notification QCG-Notification

In a nutshell, the QosCosGrid middleware consists of two logical levels: grid domain and administrative domain. Grid-level services control, schedule and generally supervise the execution of end-users applications, which are spread between independent administrative domains. The administrative domain represents a single resource provider (e.g. HPC or datacenter) participating in a certain Grid or Cloud environment by sharing its computational resources, e.g. computing clusters, with both local and external end-users. The logical separation of administrative domains corresponds with the fact that they are possessed by different institutions or resource owners. Each institution contributes its resources for the benefit of the entire Grid or Cloud, while controlling its own administrative domain and own resource allocation/sharing policies.

The key component of every administrative domain in QosCosGrid is the QCG-Computing, which gives the remote access to queuing systems resources. QCG-Computing supports advance reservations, parallel execution environments - OpenMPI, ProActive and MUSCLE with coordinators responsible for synchronization of cross-cluster executions (see Cross-cluster communication) and QCG Data Movement services for managing input and output data. The another relevant service at the administrative domain is in charge of notification mechanism and it is called QCG-Notification. These services are tightly integrated and connected to the Grid-level services. The critical service on that level is QCG-Broker, which is a meta-scheduling framework controlling executions of applicaitons on the top of queuing systems via QCG-Computing services.

End-user Information

The demanding nature of scientific simulations requires a new e-Infrastructure that is able to simultaneously manage heterogeneous resources, such as computing resources, storage and network to guarantee the level of Quality of Service (QoS) required by end-users for their applications, especially a large number of legacy applications designed to run in parallel. Addressing the need of end-users to run large scale complex simulations we have built a system capable to bring supercomputer-like performance for advanced applications, including sophisticated parameter sweep experiments, workflows, and more importantly cross-cluster parallel computations. For high-level end-users we provide a QCG Science Gateways, the components offering an easy access to concrete applications. If you are interested in using QosCosGrid as a user see End-user Information.

Developer & Admin Information

QosCosGrid provides a consistent environment for creating and deploying Grid applications. It is also relatively easy to install, configure and maintain. Developers of new applications and administrators should look at the Developer & Admin Information? section for more details.

Community

The QosCosGrid community is still rising. See the Community section and get involved!

Installation

For the QosCosGrid packages and installation instructions go to the Installation section.

Interoperability & standards supported

The QosCosGrid implementation is based on the open, widely accepted standards. In general, QosCosGrid supports OGF DRMAA, JSDL, BES, HPC Profile and OASIS WS-Notification.

Contacts

Application Department of  Poznan Supercomputing and Networking Center

  • krzysztof.kurowski(at)man.poznan.pl
  • piontek(at)man.poznan.pl
  • mamonski(at)man.poznan.pl
  • bbosak(at)man.poznan.pl