source: papers/SMPaT-2012_DCWoRMS/review/reviewers_remarks @ 957

Reviewer #1: This paper is interesting. It is well written.
It presents a simulator called DCWoRMS which proposes modeling and simulation of computing infrastructures. It can be used to estimate : the performance, the energy consumption or evaluate
different energy-aware management policies.
The authors compare their approach with different ones, maybe they could also include in their state of the art the SimGrid simulator.
DCWoRMS offers many interesting capabilities : different queues, plugins, different profiles, different models, power management interface ... it is a complete tool.

Some comments :
- more equations (proposed in default plugins for the execution time, for the power ...) should be given. For example page 11 for execution time.
-the thermal aware aspects are just mentioned but not detailed. The reader expects explanations about cooling modeling and simulation, default policy...
-they speak about consolidation but there is no mention of VM ?
-page 13 a schema of the link with the CFD solver could be added
-figures 7, 8, 9 and 10 are very difficult to read : it can not be said "as can be observed" !


Some questions :
-page 7 : for workload modeling : is it possible to use SWF file and XML file for the same simulation ?
-page 18 : I think there are mistakes in the sections referenced : the static model is presented in section 4.2 ? the dynamic one in section 4.3 ? and the mapping in section 4.4 ?
-table 8 : the EO+NPM give the same results as R is it normal ?



Some mistakes :
-names authors format in the references have to be the same First name Last name or F.last name+++++





Reviewer #2: This paper proposes a simulation tool for evaluating energy efficiency of data center infrastructures and policies. The presentation of the paper is very good and the simulation results provide an interesting insight into the capabilities of the tool.

Detailed comments
-Section 2:What is the meaning of the phrase: 'imposes and restricts users in terms of modeled resources'? Explain or rephrase to make the contribution of the paper clear
-Section 3.4: No explanation of how 'thermal issues' are taken into account is presented in the experimental section.
-Section 3.2-3.5: It would be useful to provide a short concise summary about which information should be the input to the simulator by the user, how the tool deals with this input and what are the outputs.
-Section 5.4: The question that arises here is how hard/realistic it is to run each application on each type of node for all frequencies  in advance in order to use the mapping approach. It would be useful to show the results from these measurements and show how the equation (7) fits on these data.   
In this context, it would be interesting to run the simulations for the case that the model of equation (7) is used, and show whether the energy optimization techniques result in real savings or not due to the model errors.
-Section 5.4: Please provide some explanation on table 5.
-Section 5.5.1: Figures 6 and especially the Gannt charts in all cases are impossible to read. It would be more useful to present a 'zoom in'  for a specific case to show the information that this chart holds.
-Section 5.5.3: It would be interesting to show or comment on the existence of a 'critical' slow-down frequency above which the slow down is indeed energy efficient despite the longer execution times.
-Section 5.6: In the tables it appears that less load has greater energy usage and makespan values than the respective values of greater load. Please explain.




Reviewer #3: The paper introduces a Data Center Workload and Resource Management Simulator (DCWoRMS), which enables modeling and simulation of computing infrastructures to estimate their performance, energy consumption, and energy-efficiency metrics for diverse workloads and
management policies. Estimation and evaluation of energy consumption in data centers is an important research problem. The impact of several resource management policies on overall energy-efficiency is studied by simulation. The simulation results are compared to measurements of real servers.

The paper is well-written and the topic fits to the aims and scope of the journal.

Some small remarks:
References: Please use the same style for all the references. Currently, some references are ended with a dot, some others not.+++++
Language check recommended. I think there are quite many missing a/an/the.
Figure 5 misses the units (Watts and seconds?)+++++
Figures 6-11 could be bigger/with better resolution.
Authors may find the following recent journal paper interesting and may consider to include it in related work: Majanen, MŠmmelŠ, Giesler, "Energy and carbon aware scheduling in supercomputing", International Journal on Advances in Intelligent Systems. Vol. 5 (2012) Nr: 3 & 4, pp. 451 - 469, http://www.iariajournals.org/intelligent_systems/intsys_v5_n34_2012_paged.pdf





Reviewer #4: The authors of this paper present a simulation tool that can be served in order to predict the energy efficiency of distributed computing infrastructures. Consequently, they presented a generic architecture where the key contribution was the plugins and compared some scheduling policies. The notion of the plugin is interesting, however the paper itself is not self-contained and lacks precision. More precisely, the modern data centres are equipped with networking equipment as well as storage devices (e.g. SAN) where those consume 66% of the ICT resources energy usage (the rest 33% is drawn by servers). However, the authors don't take this important factor into account. Furthermore, the authors claim that the simulation tool is appropriate to distributed computing infrastructures including Cloud, whereas throughout the paper the main focus was dedicated to HPC infrastructure (e.g. Section 5). As a matter of fact, it is very difficult to extract the main
contribution of the authors especially when comparing their work with the one of [13].

As mentioned above, the paper lacks clear description of how cloud computing can be simulated: the overhead of virtualization is not specified as well as specific performance metrics of Virtual Machines are not addressed. Among several questions, for instance, how to predict the lifetime of a Virtual Machine? How to present a Virtual Machine in the form of workload description introduced in Section 3.2.

With respect to Section 3.3 (Resource modeling), the authors need to give a detailed description (e.g UML class diagrams can be useful) of the data center's infrastructure in terms of main ICT resources and their interconnections. More precisely, they need to introduce a generic description of how to present enclosures, racks, networking equipment, storage devices, servers, PDUs, cooling system, etc.

In Section 1, the authors claim that they compare the results of simulations to measurements of real servers. Where such comparison can be found? If in Section 5, there is no such comparison. Furthermore, nothing is said about the confidence intervals of the obtained results.

In page 3, lines 27-31 are inconsistent with the content of the paper: Section 6 covers the conclusion and not how to integrate workload and resource simulations with heat transfer ...

The readability of the paper needs to be improved drastically:
  1. Figure 4 is not understandable on a black and white printout. Furthermore, what are the hardware characteristics of the investigated servers
  3. Figures 7-11 are unreadable
  4. Page 2, lines 13-17: The whole sentence needs to be rephrased
  6. Page 6, line 48: where in the paper the generator module is described?
  7. Page 13 line 55: why it is the core voltage and not the processor's voltage? If the voltage is related to the core, why the frequency is not related to the core?
  8. Section 4.2 needs to be described better. For instance, how the set of states "n" are known in advance? How to correlate the power to the p-state, without taking into account the utilization rate of the CPU? How to specify how long a resource is in a given state (to compute the energy) especially for the case of VMs? How to compute the idle power of the CPU?
  9. In Equation (7), how do we compute the "Papp". Furthermore, the same equation takes into account the utilization rate of the CPU. Then what role does "Papp" play?
10. Table 1: RAM type need to be specified: DDR 1/2/3 -- buffered or unbuffered, etc. Furthermore, what is PSU's efficiency of the tested servers?
11. Page 21 lines 18 - 23: rephrase the whole sentence +++++
12. Page 21 line 50: rephrase the whole sentence +++++
13. Table 6: the measured power of testbed nodes or processors?
14. Page 23 line 56: rephrase the whole sentence
Some Typos:

  Page 10 lines 34 and 38: shouldn't be power instead of energy?
  Equation (1) is missing the variable of integration "dt"