[[PageOutline]] The page describes the usage of the QCG-Broker Client, which is the command line text client for the QCG-Broker service. = QCG Submission host = * To submit and control their jobs in the QCG infrastructure users can use various tools. One of such tools is QCG-!BrokerClient which is the commandline client for the QCG-Broker service. '''IMPORTANT:''' * For the PLGrid users it is recommended to use one the "official" QCG Access Machines with pre-installed client (e.g. ''qcg.man.poznan.pl'') {{{ ssh plg@qcg.man.poznan.pl }}} The RPM installation procedure for SL5/6 is described [http://www.qoscosgrid.org/trac/qcg-broker/wiki/client_installation_guide_rpm here] = User Requirements = * '''User's credential''' To secure the communication and to authenticate the user to the service the X509 proxy certificate is needed. = Configuration = For its work the client needs user credentials to be present and properly configured. Additionally every user can overwrite some global settings to adapt the client behavior to its own preferences. For details please visit the [[qcg-client-config|QCG-Client Configuration]] page. = QCG Broker Client = The QCG-Client is a java based text client offering access to the QCG infrastructure from the command-line level. The QCG-Client is offered in two versions and the choice of the version depends on user's experience and needs: * QCG-!SimpleClient (recommended) is a set of tools (similar to ones offered by queuing systems) for managing of jobs. For user's convenience the simple format of description of jobs (based on #QCG directives) was proposed allowing in a simple way to describe the most popular use cases. * [[qcg-advanced-client | QCG-AdvancedClient]] - the client intended for more experienced users requiring more advanced features like workflows, parameter sweep, tasks consisting of many groups of processes with different requirements. The job has to be described by XML document. = QCG Simple Client = The QCG Simple client is recommended for all users who do not need advanced capabilities of the QCG middleware like workflows, parallel jobs with topologies or parameter sweep jobs. In return, it offers access to the most frequently used functionality in a very simple and intuitive way. The QCG Simple client consists of a set of command line tools modeled after the set of tools known from popular queueing systems. All tasks submitted to be executed by QCG-Broker must be described in a formal way using the concept of directives known also from queueing systems. == qcg-* Tools == Set of tools allowing to submit/control tasks and to reserve/control resources managed by the QCG infrastructure. [[qcg-tools|The detailed description of the qcg-* tools]] Submission and controlling of tasks: * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-cancel qcg-cancel] - cancel task(s) * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-clean qcg-clean] - clean the working directories of given tasks * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-connect qcg-connect] - connect with interactive session to the task * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-info qcg-info] - display detailed information about the given tasks * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-list qcg-list] - lists tasks in the system * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-peak qcg-peek] - display ending of (stdout, stderr) streams * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-proxy qcg-proxy] - create user proxy certificate * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-refetch qcg-refetch] - retry/repeat the transfer of output files/directories * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-refresh_proxy qcg-refresh_proxy] - refresh user proxy certificate for given tasks * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-resub qcg-resub] - resubmit the task to be processed once again * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-sub qcg-sub] - submit the task to be processes by QCG services Resources reservation and controll: * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-rcancel qcg-rcancel] - cancel reservation(s) * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-reserve qcg-reserve] - reserve resources * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-rinfo qcg-rinfo] - display information about the given reservation(s) * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-tools#qcg-rlist qcg-rlist] - list reservation in the system == QCG-Simple description format == Every task submitted to the system has to be described in a formal way. The current section describes the QCG-Simple format which is recommended and sufficient for most tasks. The format does not allow to describe more sophisticated scenarios like workflows, parameter sweep tasks, parallel tasks with topologies. Such scenarios are supported by the XML format called QCG-!JobProfile. The QCG-Simple description format is similar to the concept used in queueing systems. The document submitted to be processed by the system is an script with QCG directives and invocation of user code. The QCG directives are lines starting with the #QCG string and inform the system how to process the task (eg. about resource requirements and input/output files and directories). [[qcg-directives| The detailed description of the #QCG directives]] List of #QCG directives: * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#application application] - name of the application * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#argument argument] - argument of the application * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#assistant assistant] - auxiliary script started together with the main application * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#deadline deadline]- deadline for the task execution * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#environment environment] - environment variable * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#error error] - path for the standard error * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#executable executable] - path to the executable * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#grant grant]- name of the grant * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#host host] - name of the candidate host * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#include include] - include description file * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#input input] - path of the standard input file * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#memory memory] - memory requirements * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#memory-per-slot memory-per-slot] - memory-per-slot requirements * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#module module]- name of the required module * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#monitor monitor] - task monitoring script * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#name name]- human readable name of the task * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#native native] - parameters/requirements passed directly to the queueing system * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#nodes nodes] - requested topology of the parallel application * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#not-after not-after] - execute not later then * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#not-before not-before] - do not execute before * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#note note] - human readable description of the task * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#notify notify] - notify about changes of task status * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#output output] - location for the standard output file * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#parameter-sweep parameter-sweep] - parameter-sweep task definition * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#persistent persistent] - do not remove the working directory * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#postprocess postprocess] - task epilog * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#preprocess preprocess] - task prolog * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#procs procs] - number of requested slots/processes * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#properties properties] - node properties * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#queue queue] - selected queueu * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#reservation reservation] - reservation identifier * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#stage-in-dir stage-in-dir] - input directory * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#stage-in-file stage-in-file] - input file * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#stage-out-dir stage-out-dir] - output directory * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#stage-out-file stage-out-file] - output file * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#use-reservation use-reservation] - use the advance reservation mechanism * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#use-scratch use-scratch] - use the scratch space * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#walltime walltime] - execution duration * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#watch-output watch-output] - monitor the output and send notifications about the progress * [http://apps.man.poznan.pl/trac/qcg-broker/wiki/qcg-directives#variable variable] - description variable (replaced only in values of QCG directives) == Description variables == The following set of variables is allowed in descriptions: * '''HOSTNAME''' - name of the resource on which the task is running, * '''HOME''' - home directory of the user, * '''TASK_DIR''' - gsiftp:// lisk to the task working directory * '''JOB_ID''' - experiment identifier * '''TASK_ID''' - task identifier * '''USER_DN''' - User's Distinguished Name (in PEM format /C=PL/O=GRID/O=PSNC/CN=Tomasz Piontek) * '''PROCESS_GROUP''' - identifier of the group of processes * '''NOTE''' - value of the "note" directive. {{{ #QCG output={JOB_ID}.output #QCG stage-out-file=file.output->file.${NOTE} }}} == Remote files/directories == The following transfer protocols are supported in descriptins: * '''GridFTP''', * '''FTP''', * '''HTTP/HTTPS''', * '''iRods (authorization by user/password) Currently transfer of HTTP/HTTPS directories and output files is not supported. '''IMPORTANT: There is no HTTPS server's authorization. QCG-Broker does no validate remote server's certificates.''' For detailed description of remote file's syntax visit the [[qcg-transfer-syntax | QCG-Client transfer syntax]] page. == Tasks examples == Please visit the [[qcg-simple-examples | QCG-Simple examples]] page for description examples. = Contact = Please report all problems and send questions to the [mailto:qcg@plgrid.pl] address.