Context Navigation

General information
qcg-cancel
qcg-clean
qcg-connect
qcg-info
qcg-list
qcg-peak
qcg-proxy
qcg-refetch
qcg-refresh_proxy
qcg-resub
qcg-sub
qcg-rcancel
qcg-reserve
qcg-rinfo
qcg-rlist

General information

Invoked with the -h/--help option all qcg-* commands display the coherent information about the usage.

qcg-<command> -h
qcg-<command> --help

Many qcg-commands take as an argument list of task identifiers. The list consists of identifiers in the following form "JOBID/[TASKID]" separated by the space character. The JOBID is an identifier of the job and TASKID is an optional identifier of the task. If the TASKID is not given, the default one - "task" is assumed. For tasks described in the QCG-Simple format the JOBID identifier is enough.

qcg-cancel

The command allows to cancel processing of the list of tasks.

Cancel task(s)
 
usage: qcg-cancel [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
usage: qcg-client cancel_task [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
 
Options:
 -h,--help      display help message
 -v,--version   display version
 
Arguments:
 JOBID  identifier of the job
 TASKID optional identifier of the task
                The default identifier of task is 'task'

qcg-cancel J1363865127610__1345
qcg-cancel J1363865127610__1345/task123
qcg-cancel J1363865127610__1345 J1363865128119__5737

qcg-clean

The command removes working directories of the set of tasks.

Clean task(s). Remove working directory
 
usage: qcg-clean [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
usage: qcg-client clean_task [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
 
Options:
 -h,--help      display help message
 -v,--version   display version
 
Arguments:
 JOBID  identifier of the job
 TASKID optional identifier of the task
                The default identifier of task is 'task'

qcg-clean J1363865127610__1345
qcg-clean J1363865127610__1345/task
qcg-clean J1363865127610__1345 J1363865128119__5737

qcg-connect

The command opens an interactive shell session in the working directory of the task.

IMPORTANT: It is possible to connect only to the task in the RUNNING state.

Connect to the task. Open interactive terminal in the working directory of the given task
 
usage: qcg-connect [-h] [-v]   JOBID[/TASKID]
usage: qcg-client connect_to_task [-h] [-v]   JOBID[/TASKID]
 
Options:
 -h,--help      display help message
 -v,--version   display version
 
Arguments:
 JOBID  identifier of the job
 TASKID optional identifier of the task
                The default identifier of task is 'task'

qcg-connect J1363865127610__1345
qcg-connect J1363865127610__1345/task

qcg-info

The command displays comprehensive information about the given set of tasks.

Display information about task(s)
 
usage: qcg-info [-d] [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
usage: qcg-client task_info [-d] [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
 
Options:
 -d,--description   display user's job description
 -h,--help          display help message
 -v,--version       display version
 
Arguments:
 JOBID  identifier of the job
 TASKID optional identifier of the task
                The default identifier of task is 'task'

qcg-info J1363865127610__1345
qcg-info J1363865127610__1345/task

Below there is an example of the qcg-info command output

[plgpiontek@qcg host]$ qcg-info -d J1376043927718__9398
https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek
ProxyLifetime = 6 Days 21 Hours 50 Minutes 51 Seconds

J1376043927718__9398 : 
Note: 
UserDN: /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek
TaskType: SINGLE
SubmissionTime: Fri Aug 09 12:25:28 CEST 2013
FinishTime: Fri Aug 09 12:27:14 CEST 2013
ProxyLifetime: P6DT21H50M48S
Status: FINISHED
StatusDesc: 
StartTime: Fri Aug 09 12:25:28 CEST 2013
Purged: true
DescriptionType: QCG_SIMPLE
Description: 

#QCG procs=5
#QCG walltime=PT5M
#QCG output=p2.${JOB_ID}

cat $QCG_NODEFILE


# Provided as QCG-Broker Client argument

# Added by QCG-Broker Client 
#QCG  client-url=gsiftp://qcg.man.poznan.pl:2811/
#QCG  working-dir=/home/plgrid/plgpiontek/reef/EGI/
#QCG  script-file=./p2.qcg
#QCG  environment=QCG_TASK_INDEX->0


Allocation: 
HostName: zeus.cyfronet.pl
ProcessesCount: 5
ProcessesGroupId: qcg
Status: FINISHED
StatusDescription: 
SubmissionTime: Fri Aug 09 12:25:28 CEST 2013
FinishTime: Fri Aug 09 12:27:19 CEST 2013
LocalSubmissionTime: Fri Aug 09 12:25:31 CEST 2013
LocalStartTime: Fri Aug 09 12:26:54 CEST 2013
LocalFinishTime: Fri Aug 09 12:27:14 CEST 2013
Purged: true

qcg-list

The command displays table with information about tasks that meet the given criteria.

Display information about tasks

usage: qcg-list [-C <period>] [-c] [-F <format>] [-h] [-n <NOTE>] [-N] [-p <PURGED>] [-R] [-S
       <period>] [-s <status(es)>] [-T <format>] [-t] [-v]
usage: qcg-client get_tasks [-C <period>] [-c] [-F <format>] [-h] [-n <NOTE>] [-N] [-p <PURGED>]
       [-R] [-S <period>] [-s <status(es)>] [-T <format>] [-t] [-v]

Options:
 -C,--changed <period>      changed in the period of time
 -c,--color                 Colorize output
 -F,--format <format>       columns format
 -h,--help                  display help message
 -n,--note <NOTE>           list elements with the given note
 -N,--no-list               do not display list
 -p,--purged <PURGED>       display whether the working directory is purged or not
                            (PURGED=true|false).
 -R,--report                display summary report.
 -S,--submitted <period>    submited in the period of time. If the status is defined the default
                            value of the period is 1 day.
 -s,--status <status(es)>   status(es) of the job. Lack of this option is equivalent to requesting
                            unterminated activities.
 -T,--time <format>         time format
 -t,--truncated             display truncated identifiers
 -v,--version               display version

qcg-list
qcg-list -S 1d
qcg-list -s failed
qcg-list -S 1d -s all -R

The options -C/--changed and -S/--submitted accept definition of time period being a single value of days ("d"), hours ("h") or minutes ("m").
The Option -s/--status allows to limit the list to ones with concrete statuses. It is possible to specify list of statuses separated by commas.
The option -n/--note supports search patterns with "%" as a wildcard. (e.g. "experiment_%").

List of possible task statuses:

UNSUBMITTED – processing of the task postponed due to the precedence constraints,
UNCOMMITED - tasks waits to be committed,
QUEUED – tasks waits in the queue to be processed,
PREPROCESSING – the execution environment in prepared,
PENDING – application waits in the queuing system to be executed,
RUNNING – task (application) is running,
STOPPED – application has finished, system in ready to stage out results,
POSTPROCESSING – system stages out results or cleans the working directory
FINISHED – the whole task (application and all transfers) has finished
FAILED – processing of the task failed
CANCELED – the task has been canceled by the user or system

For the user convenience set of some constants was defined:

all - any status,
terminated - terminal statuses (FINISHED,FAILED,CANCELED)
unterminated - unterminal statuses (UNSUBMITED,UNCOMMITED,QUEUED,PREPROCESSING,PENDING,RUNNING,STOPPED,POSTPROCESSING).

Date display format

User can define his own format for dates:

YY - year (4 digits)
yy - year (2 digits)
m - month (01-12)
B - name of the month
b - short name of the month
A - week day
a - short week day
d - day (01-31)
e - day (1-31)
H - hour (00-23)
k - hour (0-23)
M - minutes (00-59)
S - seconds (00-59)

Default format is: "dd.MM.yy HH:mm"

Columns format

User can define his own format for columns.

In the format one can define list of columns, their widths and alignment using the following schema:

%[-]<WIDTH><COLUMN_ID> [ %[-]<WIDTH><COLUMN_ID> ] ...

where:

% - mandatory character starting
- optional character "minus" for the adjustment to the left
WIDTH - width of the column
COLUMN_ID - identifier of the column (see the table below).

The default format is: "%-22I %-20N %-15T %-15X %-15E %-16S %-8H %-5F %-20D"

List of displayed columns:

Column	Format Identifier	Meaning
IDENTIFIER	I	task identifier
NOTE	N	value of the #QCG note directive
SUBMISSION TIME	T	time when the task was submitted to the system
START TIME	X	estimated (if known) of real start time
FINISH TIME	E	time when the task has been finished
STATUS	S	task status
HOSTNAME	H	name of the cluster which the task is or was executed on
FLAGS	F	Some additional information about the task
STATUS DESCRIPTION	D	human readable description (reason) of the status. Should be present for FAILED status

qcg-list -F ""%-22I  %-20N  %-15T  %-15X  %-15E  %-16S  %-8H  %-5F  %-20D" -t "dd.MM.yy HH:mm"

FLAGS:

S - QCG-Simple dialect
X - QCG-JobProfile (XML) dialect
J - JSDL dialect
P - the working directory has been purged
UP - the working directory has not been purged.

[plgpiontek@qcg host]$ qcg-list -S 10h -s all -R -t
https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek
ProxyLifetime = 6 Days 20 Hours 49 Minutes 13 Seconds

JOB IDENTIFIER          NOTE                  SUBMISSION TIME  START TIME       FINISH TIME      STATUS            HOSTNAME  FLAGS  STATUS DESCRIPTION    

J1376032871771__3345                          09.08.13 09:21                    09.08.13 09:21   FAILED            galera    S UP   errorMessage = jobS*  
J1376033086383_TASK_9*                        09.08.13 09:24   09.08.13 09:32R  09.08.13 12:26   CANCELED          zeus no*  X UP                         
J1376033996852__3397                          09.08.13 09:39   09.08.13 09:41   09.08.13 09:41   FINISHED          zeus      S P                          
J1376034209604_TASK_8*                        09.08.13 09:43   09.08.13 09:52   09.08.13 10:22   FAILED            nova ze*  X UP   Application failed    
J1376034216649_TASK_4*                        09.08.13 09:43   09.08.13 09:49   09.08.13 09:58   FAILED            nova ze*  X UP   Application failed    
J1376034222276_TASK_6*                        09.08.13 09:43   09.08.13 09:53   09.08.13 10:05   FAILED            nova ze*  X UP   Application failed    
J1376035574438_TASK_8*                        09.08.13 10:06   09.08.13 10:21   09.08.13 12:26   CANCELED          nova ze*  X UP                         
J1376043152403__8834                          09.08.13 12:12                    09.08.13 12:17   CANCELED          inula     S P                          
J1376043474972__0601                          09.08.13 12:17   09.08.13 12:20   09.08.13 12:23   FINISHED          nova      S P                          
J1376043567862__3102                          09.08.13 12:19   09.08.13 12:20   09.08.13 12:23   FINISHED          hydra     S P                          
J1376043655456__7730                          09.08.13 12:20   09.08.13 12:21   09.08.13 12:23   FINISHED          hydra     S P                          
J1376043927254__5479                          09.08.13 12:25   09.08.13 12:27   09.08.13 12:27   FINISHED          hydra     S P                          
J1376043927718__9398                          09.08.13 12:25   09.08.13 12:26   09.08.13 12:27   FINISHED          zeus      S P                          
J1376047699549__5485                          09.08.13 13:28   09.08.13 13:28   09.08.13 13:28   FINISHED          inula     S P                          

IMPORTANT: * - the string was truncated to the width of the column

List of truncated job identifiers:
J1376033086383_TASK_9115
J1376034209604_TASK_8944
J1376034216649_TASK_4688
J1376034222276_TASK_6093
J1376035574438_TASK_8535


Number of tasks:
FINISHED : 7
CANCELED : 3
FAILED : 4
TOTAL: 14

qcg-peak

The command allows to watch the ending of the requested file. If the file is not specified the concatenation of stdout and stderr is displayed.

Peek the output file(s)
 
usage: qcg-peek [-c <NUMBER>] [-f <FILE>] [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
usage: qcg-client peek_output [-c <NUMBER>] [-f <FILE>] [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
 
Options:
 -c,--characters <NUMBER>   number of characters
 -f,--file <FILE>           file to peek
 -h,--help                  display help message
 -v,--version               display version
 
Arguments:
 JOBID  identifier of the job
 TASKID optional identifier of the task
                The default identifier of task is 'task'

qcg-peek J1363865127610__1345
qcg-peek -f output.txt J1363865127610__1345
qcg-peek -f output.txt -c 1000 J1363865127610__1345

qcg-proxy

The command creates user proxy certificate.

Create user proxy
 
usage: proxy_init [-h] [-v]
 
Options:
 -h,--help      display help message
 -v,--version   display version

qcg-refetch

The command allows to repeat once again the transfer of output files and directories. It can be useful when the first automatic attempt failed due to lack of valid user proxy certificate or due to temporal network problems. The user proxy certificate can be refreshed using the qcg-refresh_proxy command.

Refetch output files
 
usage: qcg-refetch [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
usage: qcg-client stage_out [-h] [-v]   JOBID[/TASKID] [JOBID[/TASKID]] ...
 
Options:
 -h,--help      display help message
 -v,--version   display version
 
Arguments:
 JOBID  identifier of the job
 TASKID optional identifier of the task
                The default identifier of task is 'task'

qcg-refetch J1363865127610__1345
qcg-refetch J1363865127610__1345/task

qcg-refresh_proxy

The command allows to refresh or prolongs the user proxy certificate for the set of experiments (jobs). The valid proxy certificate is needed for example to transfer output files and directories. The remaining life time of the proxy certificate can be checked using the "qcg-info" command.

Refresh proxy for the given job(s)
 
usage: qcg-client refresh_proxy [-h] [-v]   JOBID [JOBID] ...
 
Options:
 -h,--help      display help message
 -v,--version   display version
 
Arguments:
 JOBID  identifier of the job

qcg-resub

Resubmits the job to be processed once again. The description of the job is taken from the job to be "resubmited".

Resubmit job(s)

usage: qcg-client resubmit_job [-h] [-v]   JOBID [JOBID] ...
usage: qcg-resub [-h] [-v]   JOBID [JOBID] ...

Options:
 -h,--help      display help message
 -v,--version   display version

Arguments:
 JOBID  identifier of the job

qcg-sub

The command submits the task to be processed by QCG services and executed on the infrastructure. For every description the command returns the globally unique identifier which identifies the task in the system.

Submit job(s) to the QCG infrastructure
 
usage: qcg-sub [-D <DIRECTIVE>] [-h] [-I] [-J | -S | -X] [-R <NUMBER>]  [-v]   FILE [FILE] ...
usage: qcg-client submit_job [-D <DIRECTIVE>] [-h] [-I] [-J | -S | -X] [-R <NUMBER>]  [-v]   FILE [FILE] ...
 
Options:
 -D,--directive <DIRECTIVE>   adds #QCG directives to the description
 -h,--help                    display help message
 -I,--interactive             Interactive mode
 -J,--jsdl                    JSDL dialect
 -R,--repeat <NUMBER>         repeat submission (only for testing)
 -S,--simple                  QCG Simple dialect
 -v,--version                 display version
 -X,--xml                     QCG XML dialect

For every task the QCG_TASK_INDEX environment variable is set to the value of corresponding iteration.

IMPORTANT: QCG directives separated by comma can be defined also out of the description file using the -D/--directive option. This allows to change or adapt the description without necessity to modify the file.

qcg-sub experiment.qcg
qcg-sub -S file1.qcg file2.qcg file3.qcg
qcg-sub -D "host=nova,walltime=PT10M" cfd.qcg

[plgpiontek@qcg EGI]$ qcg-sub p1.qcg p2.qcg
https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek
ProxyLifetime = 6 Days 19 Hours 14 Minutes 20 Seconds

p1.qcg 0        jobId = J1376058593985__6195
p2.qcg 0        jobId = J1376058594454__6576

IMPORTANT: Using the qcg-sub command with the -I/--interactive option it is possible to run any interactive (textual) application. The especially often used case is to run the shell console what gives interactive access to clusters that do not offer ssh access.

#QCG host=inula.man.poznan.pl
 
/bin/bash

qcg-sub -I interactive.qcg

It is also possible to omit the host directive in the description file and to define as an argument of invocation of the qcg-sub tool.

qcg-sub -I -D "host=inula" interactive.qcg

[plgpiontek@qcg host]$ qcg-sub -I -D "host=nova" interactive.qcg 
https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek                                                                                                   
ProxyLifetime = 6 Days 18 Hours 46 Minutes 22 Seconds                                                                                                                          
                                                                                                                                                                               
interactive.qcg 0       jobId = J1376060272230__4664
Press Ctrl-C to cancel the task
Interactive session started. Use the 'exit' command to quit.
plgpiontek@wn732: ~/J1376060272230__4664_task_1376060272550_192 $ hostname
wn732

qcg-rcancel

The command cancels the list of reservations.

Cancel reservation(s)
 
usage: qcg-rcancel [-h] [-v]   RESID [RESID] ...
usage: qcg-client cancel_reservation [-h] [-v]   RESID [RESID] ...
 
Options:
 -h,--help      display help message
 -v,--version   display version
 
Arguments:
 RESID  identifier of the reservation

qcg-rcancel R1363180788689_RESERVATION_0172
qcg-rcancel R1363180788689_RESERVATION_0172 R1363252044299__9086

qcg-reserve

The commands reserves computational resources. For every description containing the resource requirements it returns the reservation identifier which identifies the reservation in the system. The command is similar to the qcg-sub one, but instead of executing the application on resources it only reserves them.

Create reservation(s)
 
usage: qcg-reserve [-h] [-J | -S | -X]  [-v]   FILE [FILE] ...
usage: qcg-client create_reservation [-h] [-J | -S | -X]  [-v]   FILE [FILE] ...
 
Options:
 -h,--help      display help message
 -J,--jsdl      JSDL dialect
 -S,--simple    QCG Simple dialect
 -v,--version   display version
 -X,--xml       QCG XML dialect
 
Arguments:
 FILE   Path to file with job description

[plgpiontek@qcg EGI]$ qcg-reserve p1.qcg p2.qcg
https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek
ProxyLifetime = 6 Days 18 Hours 33 Minutes 58 Seconds

p1.qcg : reservationId = R1376061016022__2486
p2.qcg : reservationId = R1376061016345__4556

The status of the reservation can be checked using the "qcg-rinfo" and "qcg-rlist" commands.

The reservation can be canceled using the "qcg-rcancel" command.

qcg-rinfo

The command displays the comprehensive information about the list of reservations.

Display information about the given reservation(s)
 
usage: qcg-rinfo [-d] [-D] [-h] [-v]   RESID [RESID] ...
usage: qcg-client reservation_info [-d] [-D] [-h] [-v]   RESID [RESID] ...
 
Options:
 -d,--description       display user's job description
 -D,--qcg_description   display QCG job description
 -h,--help              display help message
 -v,--version           display version
 
Arguments:
 RESID  identifier of the reservation

[plgpiontek@qcg EGI]$ qcg-rinfo -d R1376061016345__4556
https://qcg-broker.man.poznan.pl:8443/qcg/services/
/C=PL/O=GRID/O=PSNC/CN=qcg-broker/qcg-broker.man.poznan.pl
UserDN = /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek
ProxyLifetime = 6 Days 18 Hours 30 Minutes 11 Seconds

UserDN: /C=PL/O=PL-Grid/O=Uzytkownik/O=PCSS/CN=Tomasz Piontek/CN=plgpiontek
SubmissionTime: Fri Aug 09 17:10:16 CEST 2013
DescriptionType: QCG_SIMPLE
UserDescription: 

#QCG procs=5
#QCG walltime=PT5M
#QCG output=p2.${JOB_ID}

cat $QCG_NODEFILE

#QCG  client-url=gsiftp://qcg.man.poznan.pl:2811/
#QCG  working-dir=/home/plgrid/plgpiontek/reef/EGI/
#QCG  script-file=p2.qcg
#QCG  use-reservation
StartTime: Fri Aug 09 17:11:00 CEST 2013
EndTime: Fri Aug 09 17:17:00 CEST 2013
Status: ACTIVE
TotalSlotsCount: 5
InUse: false

HostName: zeus.cyfronet.pl
ProcessesGroupId: qcg
SlotsCount: 5
LocalReservationId: plgpiontek.2487057
Node: n14-4-29.local SlotsCount: 2
Node: n14-4-30.local SlotsCount: 3

qcg-rlist

The command displays the list of reservations that meet the required criteria.

Display information about reservations

usage: qcg-rlist [-C <period>] [-c] [-F <format>] [-h] [-n <NOTE>] [-N] [-R] [-S <period>] [-s
       <status(es)>] [-T <format>] [-t] [-v]
usage: qcg-client get_reservations [-C <period>] [-c] [-F <format>] [-h] [-n <NOTE>] [-N] [-R] [-S
       <period>] [-s <status(es)>] [-T <format>] [-t] [-v]

Options:
 -C,--changed <period>      changed in the period of time
 -c,--color                 Colorize output
 -F,--format <format>       columns format
 -h,--help                  display help message
 -n,--note <NOTE>           list elements with the given note
 -N,--no-list               do not display list
 -R,--report                display summary report.
 -S,--submitted <period>    submited in the period of time. If the status is defined the default
                            value of the period is 1 day.
 -s,--status <status(es)>   status(es) of the job. Lack of this option is equivalent to requesting
                            unterminated activities.
 -T,--time <format>         time format
 -t,--truncated             display truncated identifiers
 -v,--version               display version

IMPORTANT: For some details please see the description of the "qcg-list" command.

List of reservation statuses:

SUBMITTED - reservation request was submitted to the system,
RESERVED - resources ware reserved, but the reservation is not yet active,
ACTIVE - reservation is active,
FINISHED - reservation finished,
FAILED - reservation request failed,
CANCELED - reservation was canceled.

For the user convenience some constants were introduced:

all - all statuses,
terminated - any of terminal statuses (FINISHED,FAILED,CANCELED)
unterminated - any of not terminal statuses (SUBMITTED,ACTIVE,RESERVED).