Changes between Version 111 and Version 112 of DCWoRMS

Timestamp:

07/06/20 16:46:53 (5 years ago)

Author:

wojtekp

Comment:

--

DCWoRMS

@@ -15 +15 @@
 = Introduction =
 
-Data Center workload and resource management simulator (DCworms) is an extended simulation tool developed by PSNC [1]. It has been based on based on the GSSIM simulation framework which enabled modeling and simulation of computing infrastructures to estimate their performance, energy consumption, and energy-efficiency metrics for diverse workloads and management policies as described in the [2].
+Data Center workload and resource management simulator (DCworms) is an extended simulation tool developed by PSNC [1]. It has been based on  the GSSIM simulation framework which enabled modeling and simulation of computing infrastructures to estimate their performance, energy consumption, and energy-efficiency metrics for diverse workloads and management policies as described in the [2].
 
 
@@ -23 +23 @@
 = DCworms architecture =
 
-In general, input data for the DCworms consist of a description of workload and resources. Input data can be read from real traces (for details see sections below) or generated using the generator module. However, the key elements of the presented architecture are plugins. They allow a researcher to configure and adapt the simulation framework to his/her experiment scenario starting from modeling job performance, through energy estimations up to implementation of resource management and scheduling policies. Each plugin can be implemented independently and plugged into a specific experiment. Results of experiments are collected, aggregated, and visualized using the statistics tool. Due to a modular and plug-able architecture DCworms enables adapting it to specific resource management problems and users’ requirements.
+In general, input data for the DCworms consist of a description of workload and resources. Input data can be read from real traces (for details see sections below) or generated using the generator module. However, the key elements of the presented architecture are plugins. They allow a researcher to configure and adapt the simulation framework to his/her experiment scenario starting from modeling job performance, through energy estimations up to the implementation of resource management and scheduling policies. Each plugin can be implemented independently and plugged into a specific experiment. Results of experiments are collected, aggregated, and visualized using the statistics module. Due to a modular and plug-able architecture DCworms enables adapting it to specific resource management problems and users’ requirements.
 
 The following figure presents the overall architecture of the simulation tool.
@@ -36 +36 @@
 = Input data =
 
-In general, input data in DCworms consist of a single configuration file, description of workload and resources. Users may both generate new or read the existing synthetic data. Third party real workloads can also be imported by DCworms. If any parameters are missing after importing a workload, they can be generated by DCworms and added.
+In general, input data in DCworms consist of a single configuration file, a description of workload and resources. Users may both generate new or read the existing synthetic data. Third-party real workloads can also be imported by DCworms. If any parameters are missing after importing a workload, they can be generated by DCworms and added.
 
 
@@ -43 +43 @@
 == Configuration file == #ConfigurationFile
 
-The Experiment configuration file has typical, java resource bundle format. List of all available parameters and their interpretation is available below.
+The Experiment configuration file has a typical, java resource bundle format. A list of all available parameters and their interpretation is available below.
 
 * '''resdesc''' - path to file containing description of resources (required)
 
-* '''readscenario.workloadfilename''' - path to workload file (in SWF or GWF format) (required)
-* '''readscenario.inputfolder''' - path to directory with xml job descriptions
-
-* '''createscenario.tasksdesc''' - path to xml file which describes detail workload generator configuration
-* '''createscenario.outputfolder''' - path to directory where all generated jobs will be placed
-* '''createscenario.workloadfilename''' - name of workload file in swf format which will be generated
-* '''createscenario.overwrite_files''' - determines if previously generated files should be overwritten. Two possible values of this field are: "true" and "false".
-
-* '''creatediagrams.gantt''' - determines if gantt chart should be generated. Two possible values of this field are: "true" and "false", deafualt value: "false".
-* '''creatediagrams.tasks''' - determines if tasks execution times diagram should be generated. Two possible values of this field are: "true" and "false", deafualt value: "false".
-* '''creatediagrams.taskswaitingtime''' - determines if tasks waiting time gantt diagram should be generated. Two possible values of this field are: "true" and "false", deafualt value: "false".
-
-* '''creatediagrams.resutlization''' - determines if resource utlization chart should be generated. This field should contain the types of resources for which the diagram should be created
-* '''creatediagrams.respowerusage''' - determines if resource power usage chart should be generated. This field should contain the types of resources for which the diagram should be created
-* '''creatediagrams.resairflow''' - determines if resource airflow chart should be generated. This field should contain the types of resources for which the diagram should be created
-* '''creatediagrams.restemperature''' - determines if resource power temperature chart should be generated. This field should contain the types of resources for which the diagram should be created
-
-Parameters from readscenario and createscenario groups should be used mutual exclusively.
+* '''resourcedescriptionfile''' - path to file(s) containing description of resources (required)
+* '''workloadfile''' - path to workload file(s) (in SWF or GWF format) (required)
+* '''xmljobfolder''' - input folder with generated tasks descriptions
+* '''applicationprofilesfolder''' - input folder with application profiles
+* '''creatediagrams.generators''' - determines which library should be used for diagram generation. (JFreeChart / Gnuplot)
+* '''creatediagrams.gantt''' - determines if gantt chart should be generated. (true / false)
+* '''creatediagrams.gantt.applicationprofiles''' - todo
+* '''creatediagrams.tasks''' - determines if tasks execution times diagram should be generated. (true / false)
+* '''creatediagrams.taskswaitingtime''' - determines if tasks waiting time gantt diagram should be generated. (true / false)
+* '''creatediagrams.rooflinemodel''' - determines if roofline model diagram should be generated. It is specific for ESCAPE simulation mode. (true / false)
+* '''creatediagrams.resutlization''' - determines if resource utilization chart should be generated. This field should contain the types of resources for which the diagram should be created.
+* '''creatediagrams.cost''' - determines if resource utilization cost chart should be generated. This field should contain the types of resources for which the diagram should be created.
+* '''creatediagrams.respowerusage''' - determines if resource power usage chart should be generated. This field should contain the types of resources for which the diagram should be created.
+* '''creatediagrams.resairflow''' - determines if resource airflow chart should be generated. This field should contain the types of resources for which the diagram should be created.
+* '''creatediagrams.restemperature''' - determines if resourcetemperature chart should be generated. This field should contain the types of resources for which the diagram should be created
+* '''excludetextstats''' - disables creation of text files containing specific statistics. This field should contain the types of statistics for which the files should not be created.
+* '''creatextstatistics.jobs''' - determines if additional job statistics file should be created. (true / false)
+* '''creatextstatistics.simulation''' - determines if additional simulation statistics file should be created. (true / false)
+* '''outputfolder''' - path to directory where all generated statistics will be placed.
+* '''simulationidentifier''' - name of the simulation, will be used in result files names.
+* '''simplifiedschema''' - tells the simulator to use the simplified version of the resource schema and translate it to the regular one before loading resources. (true / false) (experimental)
+* '''usedatabase''' - determines if ….. should be stored in database instead of memory. (true / false) (experimental)
+* '''simulationmode''' - allows usage of certain features and behaviours specific for individual projects. (Default, CoolEmAll, ESCAPE)
+* '''numberofrepetitions''' - allows to automatically repeat an experiment for a given number of times. It can be used when using changing or random data. (Default: 1)
+* '''comparestatistics''' - allows automatic comparison of several performed simulations. It is also possible to select individual statistics to compare. (true / false)
+* '''comparestatistics.resourceload'''
+* '''comparestatistics.energyusage'''
+* '''comparestatistics.makespan'''
+* '''comparestatistics.taskexecutiontime'''
+* '''comparestatistics.taskqueuelength'''
+* '''comparestatistics.taskcompletiontime'''
+* '''comparestatistics.taskwaitingtime'''
+* '''comparestatistics.taskflowtime'''
+* '''comparestatistics.tasklateness'''
+* '''comparestatistics.delayedtasks'''
+* '''comparestatistics.tasktardiness'''
+* '''comparestatistics.failedrequests'''
+
 
 Path to this configuration file is the main argument of the DCworms program. It should be used as follows:
@@ -112 +132 @@
 
 Example:
-Assume, that swf file contains two tasks with id 1 and 2. You can create new job, with two tasks by defining following mapping:
+Assume, that SWF file contains two tasks with id 1 and 2. You can create new job, with two tasks by defining following mapping:
 {{{
  ;IDMapping: swfID:jobID:taskID
@@ -121 +141 @@
 New job with id = 4 consisting of two tasks with id 10 and 20 will be created.
 
-The only constraint of IDMapping section is that swf jobs, which will become tasks in new job, must occur in swf one by one. No other jobs are allowed between these swf jobs which are mapped to tasks of one new job.
-
-The experiment can be executed with usage of single swf file or swf file with xml extension. 
-If single swf is used, then task requirements like cpu count and requested memory are read directly from swf file. Notice, that information included in swf file is insufficient for using advance reservation in scheduling algorithm. To do so, you must provide xml extension of each job description and fill up its executionTime section. In xml files you can use any ids for job and tasks but you must provide correct IDMapping section (in swf file header) between xml job/task ids and swf job id. Otherwise, task start up parameters like submit time or task length in instructions will not be calculated correctly.
-If xml job description is used, then task requirements are read from xml description instead of swf file.
+The only constraint of IDMapping section is that the SWF jobs, which will become tasks in a new job, must occur in SWF one by one. No other jobs are allowed between these SWF jobs which are mapped to tasks of one new job.
+
+The experiment can be executed with usage of single SWF file or SWF file with xml extension. 
+If single SWF is used, then task requirements like cpu count and requested memory are read directly from swf file. Notice, that information included in SWF file is insufficient for using advance reservation in scheduling algorithm. To do so, you must provide XML extension of each job description and fill up its executionTime section. In XML files you can use any ids for job and tasks but you must provide correct IDMapping section (in SWF file header) between XML job/task ids and swf job id. Otherwise, task start up parameters like submit time or task length in instructions will not be calculated correctly.
+If XML job description is used, then task requirements are read from XML description instead of SWF file.
 
 
@@ -131 +151 @@
 === Workload generation ===
 
-The main goal of workload design was to ensure, that all job descriptions which were used in real resource management system like GRMS or obtained from SWF/GWF log can be used to perform experiment in DCworms simulator. However, it my be difficult for all users to reach such workloads, therefore workload generator was created.
-
-Workload generator allows you to create any number of jobs and tasks, with sophisticated resource and time requirements. The result of generation process are: desired number of job descriptions in xml format and swf file with job descriptions and all necessary header parameters (see [https://git.man.poznan.pl/stash/projects/WORMS/repos/dcworms/browse/src/main/resources/simulator/schemas/WorkloadSchema3g.xsd WorkloadSchema.xsd] for details).
+The main goal of workload design was to ensure, that all job descriptions which were used in real resource management system like GRMS or obtained from SWF/GWF log can be used to perform an experiment in DCworms simulator. However, it my be difficult for all users to reach such workloads, therefore workload generator was created.
+
+Workload generator allows you to create any number of jobs and tasks, with sophisticated resource and time requirements. The result of generation process are: desired number of job descriptions in XML format and SWF file with job descriptions and all necessary header parameters (see [https://git.man.poznan.pl/stash/projects/WORMS/repos/dcworms/browse/src/main/resources/simulator/schemas/WorkloadSchema3g.xsd WorkloadSchema.xsd] for details).
 
 Configuration options are provided by two files: