source: papers/SMPaT-2012_DCWoRMS/elsarticle-DCWoRMS.tex @ 683

%% This is file `elsarticle-template-1-num.tex',
%%
%% Copyright 2009 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.2 of this license or (at your option) any
%% later version.  The latest version of this license is in
%%    http://www.latex-project.org/lppl.txt
%% and version 1.2 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%%
%% $Id: elsarticle-template-1-num.tex 149 2009-10-08 05:01:15Z rishi $
%% $URL: http://lenova.river-valley.com/svn/elsbst/trunk/elsarticle-template-1-num.tex $
%%
\documentclass[preprint,12pt]{elsarticle}

%% Use the option review to obtain double line spacing
%% \documentclass[preprint,review,12pt]{elsarticle}

%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}

%% if you use PostScript figures in your article
%% use the graphics package for simple commands
%% \usepackage{graphics}
%% or use the graphicx package for more complicated commands
%% \usepackage{graphicx}
%% or use the epsfig package if you prefer to use the old commands
%% \usepackage{epsfig}

%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}

%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers after \end{frontmatter}.
%% \usepackage{lineno}

%% natbib.sty is loaded by default. However, natbib options can be
%% provided with \biboptions{...} command. Following options are
%% valid:

%%   round  -  round parentheses are used (default)
%%   square -  square brackets are used   [option]
%%   curly  -  curly braces are used      {option}
%%   angle  -  angle brackets are used    <option>
%%   semicolon  -  multiple citations separated by semi-colon
%%   colon  - same as semicolon, an earlier confusion
%%   comma  -  separated by comma
%%   numbers-  selects numerical citations
%%   super  -  numerical citations as superscripts
%%   sort   -  sorts multiple citations according to order in ref. list
%%   sort&compress   -  like sort, but also compresses numerical citations
%%   compress - compresses without sorting
%%
%% \biboptions{comma,round}

% \biboptions{}


\journal{Simulation Modelling Practice and Theory}

\begin{document}

\begin{frontmatter}

%% Title, authors and addresses

%% use the tnoteref command within \title for footnotes;
%% use the tnotetext command for the associated footnote;
%% use the fnref command within \author or \address for footnotes;
%% use the fntext command for the associated footnote;
%% use the corref command within \author for corresponding author footnotes;
%% use the cortext command for the associated footnote;
%% use the ead command for the email address,
%% and the form \ead[url] for the home page:
%%
%% \title{Title\tnoteref{label1}}
%% \tnotetext[label1]{}
%% \author{Name\corref{cor1}\fnref{label2}}
%% \ead{email address}
%% \ead[url]{home page}
%% \fntext[label2]{}
%% \cortext[cor1]{}
%% \address{Address\fnref{label3}}
%% \fntext[label3]{}

\title{DCWoRMS - a tool for simulation of energy efficiency in Grids and Clouds }

%% use optional labels to link authors explicitly to addresses:
%% \author[label1,label2]{<author name>}
%% \address[label1]{<address>}
%% \address[label2]{<address>}

\author{Krzysztof Kurowski, Ariel Oleksiak, Wojciech Piatek, Tomasz Piontek}


\begin{abstract}
%% Text of abstract

\end{abstract}

\begin{keyword}
%% keywords here, in the form: keyword \sep keyword

%% MSC codes here, in the form: \MSC code \sep code
%% or \MSC[2008] code \sep code (2000 is the default)

\end{keyword}

\end{frontmatter}

%%
%% Start line numbering here if you want
%%
% \linenumbers

%% main text
\section{Introduction}

TODO - Introduction

...

The remaining part of this paper is organized as follows. In Section~2 we give a brief overview of the current state of the art concerning modeling and simulation of distributed systems, like Grids and Clouds, in terms of energy efficiency. Section~3 discusses the main features of DCWoRMS. In particular, it introduces our approach to workload and resource management, presents the concept of energy efficiency modeling and explains how to incorporate a specific application performance model into simulations. Section~4 discusses energy models adopted within the DCWoRMS. In Section~5 we present some experiments that were performed using DCWoRMS utilizing real testbed nodes models to show varius types of popular resource and scheduling technics allowing to decrease the total power consumption of the execution of a set of tasks. Section~6 focuses on the role of DCWoRMS within the CoolEmAll project. Final conclusions and directions for future work are given in Section~7.

\section{Related Work}

The growing importance of energy efficiency in information technologies led to significant interest in energy saving methods for computing systems. Therefore, intelligent resource management policies are gaining popularity when considering the energy efficiency of IT infrastructures. Nevertheless, studies of impact of scheduling strategies on energy consumption require a large effort and are difficult to perform in real distributed environments. To overcome these issues extensive research has been conducted in the area of modeling and simulation tools. As a result, a wide variety of simulation tools emerged. The following section contains a short summary of existing simulators that address the green computing issues in distributed infrastructures. 

\subsection{GreenCloud}

GreenCloud \cite{GreenCloud} is a C++ based simulation environment for energy-aware cloud computing data centers. It was developed as an extension of the NS2 network simulator. GreenCloud allows researchers to observe and evaluate data centers performance and study their energy-efficiency, focusing mainly on the communications within a data center. Along with the workload distribution, it offers users a detailed, fine-grained modeling of the energy consumed by the elements of the data center. 

To deliver information about the energy usage, GreenCloud distinguishes three energy consumption components: computing energy, communicational energy, and the energy component related to the physical infrastructure of a data center. This approach enables modeling energy usage associated with computations, network operations and cooling systems. In GreenCloud, the energy models are implemented for every simulated data center entity (computing servers, core and rack switches). Moreover, due to the advantage in the simulation resolution, energy models can operate at the network packet level as well. This allows updating the levels of energy consumption whenever a new packet leaves or arrives from the link, or whenever a new task execution is started or completed at the server.
Servers are modeled as single core nodes that are responsible for task execution and may contain different scheduling strategies.
The server power consumption model implemented in GreenCloud depends on the server state as well as its utilization and allows capturing the effects of both of the Dynamic Voltage and Frequency Scaling (DVFS) and Dynamic Power Management (DPM) schemes.
At the links and switches level, GreenCloud supports Dynamic Voltage Scaling (DVS) and Dynamic Network Shutdown (DNS) techniques. The DVS method introduces a control element at each port of the switch that - depending on the traffic pattern and current levels of link utilization - could downgrade the transmission rate. The DNS approach allows putting some network equipment into a sleep mode. 

To cover the vast majority of cloud computing applications, GreenCloud defines three types of workloads: computationally intensive workloads that load computing servers considerably, data-intensive workloads that require heavy data transfers, and finally balanced workloads which aim to model the applications having both computing and data transfer requirements.
GreenCloud describes application with a number of computational requirements. Moreover, it specifies communication requirements of the applications in terms of the amount of data to be transferred before and after a task completion. The execution of each application requires a successful completion of its two main components: computing and communicational. 
In addition time constraints can be taken into account during the simulation by adding a predefined execution deadline, which aims at introducing Quality of Service constraints specified in a Service Level Agreement. Nevertheless, GreenCloud does not support application performance modeling. Aforementioned capabilities allow only incorporating simple requirements that need to be satisfied before and during the task execution.

Contrary to what the GreenCloud name may suggest, it does not allow testing the impact of a virtualization-based approach on the resource management. 
GreenCloud simulator is released under the General Public License Agreement.


\subsection{CloudSim}

CloudSim \cite{CloudSim} is an event-based simulation tool written in Java. Initially CloudSim was based on the well-known GridSim framework, however since the last few releases it is an independent simulator and does not benefit from most of the GridSim functionality. 

CloudSim allows creating a simple resources hierarchy containing computing resources that consist of machines and processors. Additionally, it may simulate the behavior of other components including storage and network resources. However, it focuses on computational resources and provides an extra virtualization layer that acts as an execution, management, and hosting environment for application services. It is responsible for the VM provisioning process as well as managing the VM life cycle such as: VM creation, VM destruction, and VM migration. It also enables evaluation of different economic policies by modeling the cost metrics related to the SaaS and IaaS models.

The CloudSim framework provides basic models and entities to validate and evaluate energy-conscious provisioning of techniques and algorithms. Each computing node can be extended with a power model that simulates the power consumption. CloudSim offers example implementations of this component that characterize some popular server models. Needless to say, it can be easily extended for simulating user-defined power consumption models. That allows estimating the current power usage according to the utilization level or the host model. This capability enables the creation of energy-conscious provisioning policies that require real-time knowledge of power consumption by Cloud system components.
Furthermore, it allows an accounting of the total energy consumed by the system during the simulation period. CloudSim comes with a set of predefined and extendable policies that manage the process of VM migrations in order to optimize the power consumption. However, the proposed solution is not appropriate for more sophisticated power management policies. In particular, CloudSim is not sufficient for modeling frequency scaling techniques and managing resource power states.

Similar to GreenCloud, CloudSim defines a simple application model that includes computational and data requirements. Although all these constraints are taken into account during scheduling, they do not affect the application execution. Thereby, a researcher is required to put a lot of effort to incorporate an application performance model into his experiments.
On the other hand CloudSim offers modeling of utilization models that are used to estimate the current load of processor, bandwidth and memory and can be taken into account during the task allocation process.
Concerning workloads, simulator is able to partially support SWF \cite{SWF} files and read data in a user-defined file format. Moreover, it can handle a wide variety of workload types, including parallel, and pre-emptive jobs.

CloudSim is available as Open Source under GPL license.


\subsection{DCSG Simulator}

DCSG Simulator \cite{DCSG} is a Data Centre Cost and Energy Simulator that has been developed under the Carbon Trust Low Carbon Collaborations program in conjunction with the BCS and Romonet Ltd. The simulator works at a data center infrastructure level where analysis of the achieved efficiency of the data center mechanical and electrical plant can be performed but also at the IT level. The simulator implements a set of basic rules that have been developed, based on a detailed understanding of the data center as a system, to allow cost and energy use to be usefully allocated to IT devices within the data center. 

As far as data center infrastructure level is concerned, DCSG Simulator calculates the power and cooling schema of data center equipment with respect to their performance. User is able to take into account a wide variety of mechanical and electrical devices like: transformers, power distribution units, power supply, cabling, computer room air conditioning units and chiller plant. For each of them numerous factors can be defined, including device capacity and efficiency, load operating points. These data can be derived from a generic list as well as from the information given by particular manufacturers. There is a wide range of pre-defined models, but user can easily extend them or create new ones.

To perform the IT simulation, it is possible to extend the data center infrastructure by putting IT devices into that data center. That enables detailed simulation of the energy efficiency of devices across a specified time period. 
In this case performance of each piece of equipment (facility and IT) within a data center is determined by a combination of factors, including workload, data center conditions, the manufacturer's specifications of the machine's components and the way in which the machine is utilized based on its provisioned IT load. 
Users are possible to bind the operational characteristics, proper to the particular geographic locations, with the simulation process. These characteristics may include temperature profile as well as the power cost that vary depending on the time and place. The output of this simulation is a set of energy and cost data representing the IT devices (including PUE and DCiE) and data center energy consumption, capital and operational costs.


According to the tool evaluation presented in \cite{DCD_Romonet} an accuracy of models delivered by Romonet is at the level of 95\% when compared with metered data. The simulator is available under an OSL V3.0 open-source license, however it can be only accessed by the DCSG Members.


\subsection{Summary}

TODO - short summary of current SoTA



\section{DCWoRMS}

The following picture (Figure~\ref{fig:arch}) presents the overall architecture of the simulation tool.

Data Center workload and resource management simulator (DCWoRMS) is a simulation tool based on the GSSIM framework \cite{GSSIM} developed by Poznan Supercomputing and Networking Center (PSNC).
GSSIM has been proposed to provide an automated tool for experimental studies of various resource management and scheduling strategies in distributed computing systems. DCWoRMS extends its basic functionality and adds some additional features related to the energy efficiency issues in data centers. In this section we will introduce the functionality of the simulator, in terms of modeling and simulation of large scale distributed systems like Grids and Clouds.


\subsection{Architecture}


\begin{figure}[tbp]
\centering
\includegraphics[width = 12cm]{fig/arch.png}
\caption{\label{fig:arch} DCWoRMS architecture}
\end{figure}

DCWoRMS is an event-driven simulation tool written in Java. In general, input data for the DCWoRMS consist of workload and resources descriptions. They can be provided by the user, read from real traces or generated using the generator module. However, the key elements of the presented architecture are plugins. They allow the researchers to configure and adapt the simulation environment to the peculiarities of their studies, starting from modeling job performance, through energy estimations up to implementation of resource management and scheduling policies. Each plugin can be implemented independently and plugged into a specific experiment. Results of experiments are collected, aggregated, and visualized using the statistics module. Due to a modular and plug-able architecture DCWoRMS can be applied to specific resource management problems and address different users’ requirements.


\subsection{Workload modeling}

As it was said, experiments performed in DCWoRMS require a description of applications that will be scheduled during the simulation. As a primary definition, DCWoRMS uses files in the Standard Workload Format (SWF) or its extension the Grid Workload Format (GWF) \cite{GWF}. In addition to the SWF file, some more detailed specification of a job and tasks can be included in an auxiliary XML file. This form of description provides the scheduler with more detailed information about application profile, task requirements, user preferences and execution time constraints, which are unavailable in SWF/GWF files. To facilitate the process of adapting the traces from real resource management systems, DCWoRMS supports reading those delivered from the most common ones like SLURM \cite{SLURM} and Torque \cite{TORQUE}. 
Since the applications may vary depending on their nature in terms of their requirements and structure, DCWoRMS provides user flexibility in defining the application model. Thus, considered workloads may have various shapes and levels of complexity that range from multiple independent jobs, through large-scale parallel applications, up to whole workflows containing time dependencies and preceding constraints between jobs and tasks. Each job may consist of one or more tasks and these can be seen as groups of processes. Moreover, DCWoRMS is able to handle rigid and moldable jobs, as well as pre-emptive ones. To model the application profile in more detail, 
DCWoRMS follows the DNA approach proposed in \cite{Ghislain}. Accordingly, each task can be presented as a sequence of phases, which shows the impact of this task on the resources that run it. Phases are then periods of time where the system is stable (load, network, memory) given a certain threshold. Each phase is linked to values of the system that represent a resource consumption profile. Such a stage could be for example described as follows: “60\% CPU, 30\% net, 10\% mem.”

Levels of information about incoming jobs are presented in Figure~\ref{fig:jobsStructure}.


\begin{figure}[tbp]
\centering
\includegraphics[width = 8cm]{fig/jobsStructure.png}
\caption{\label{fig:jobsStructure} Levels of information about jobs}
\end{figure}


This form of representation allows users to define a wide range of workloads:
HPC (long jobs, computational-intensive, hard to migrate) or virtualization (short requests) typical for cloud computing environments.
Further, the DCWoRMS benefits from the GSSIM workload generator tool and extends it with that allows creating synthetic workloads.


\subsection{Resource modeling}

The main goal of DCWoRMS is to enable researchers evaluation of various resource management policies in diverse computing environments. To this end, it supports flexible definition of simulated resources both on physical (computing resources) as well as on logical (scheduling entities) level. This flexible approach allows modeling of various computing entities consisting of compute nodes, processors and cores. In addition, detailed location of the given resources can be provided in order to group them and arrange into physical structures such as racks and containers. Each of the components may be described by different parameters specifying available memory, storage capabilities, processor speed etc. In this way, it is possible to describe power distribution system and cooling devices. Due to an extensible description, users are able to define a number of experiment-specific and visionary characteristics. Moreover, with every component, dedicated profiles can be associated that determines, among others, power, thermal and air throughput properties. The energy estimation plugin can be bundled with each resource. This allows defining various power models that can be then followed by different computing system components. Details concerning the approach to energy-efficiency modeling in DCWoRMS can be found in the next sections.

Scheduling entities allow providing data related to the brokering or queuing system characteristics. Thus, information about available queues, resources associated with them and their parameters like priority, availability of AR mechanism etc. can be defined. Moreover, allocation policy and task scheduling strategy for each scheduling entity can be introduced in form of the reference to an appropriate plugin. DCWoRMS allows building a hierarchy of schedulers corresponding to the hierarchy of resource components over which the task may be distributed.

In this way, the DCWoRMS supports simulation of a wide scope of physical and logical architectural patterns that may span from a single computing resource up to whole data centers or geographically distributed grids and clouds. In particular, it supports simulating complex distributed architectures containing models of the whole data centers, containers, racks, nodes, etc. In addition, new resources and distributed computing entities can easily be added to the DCWoRMS environment in order to enhance the functionality of the tool and address more sophisticated requirements. Granularity of such topologies may also differ from coarse-grained to very fine-grained modeling single cores, memory hierarchies and other hardware details.


\subsection{Energy management concept in DCWoRMS}

The DCWoRMS allows researchers to take into account energy efficiency and thermal issues in distributed computing experiments. That can be achieved by the means of appropriate models and profiles. In general, the main goal of the models is to emulate the behavior of the real computing resources, while profiles support models by providing data essential for the power consumption calculations. Introducing particular models into the simulation environment is possible through choosing or implementation of dedicated energy plugins that contain methods to calculate power usage of resources, their temperature and system air throughput values. Presence of detailed resource usage information, current resource energy and thermal state description and a functional energy management interface enables an implementation of energy-aware scheduling algorithms. Resource energy consumption and thermal metrics become in this context an additional criterion in the resource management process. Scheduling plugins are provided with dedicated interfaces, which allow them to collect detailed information about computing resource components and to affect their behavior.
The following subsections present the general idea behind the energy-efficiency simulations.


\subsubsection{Power management}

The motivation behind introducing a power management concept in DCWoRMS is providing researchers with the means to define the energy efficiency of resources, dependency of energy consumption on resource load and specific applications, and to manage power modes of resources. Proposed solution extends the power management concept presented in GSSIM \cite{GSSIM_Energy} by offering a much more granular approach with the possibility of plugging energy consumption models and power profiles into each resource level.

\paragraph{\textbf{Power profile}}
In general, power profiles allow specifying the power usage of resources. Depending on the accuracy of the model, users may provide additional information about power states which are supported by the resources, amounts of energy consumed in these states, and other information essential to calculate the total energy consumed by the resource during runtime. In such a way each component of IT infrastructure may be described, including computing resources, system components and data center facilities. Moreover, it is possible to define any number of new, resource specific, states, for example so called P-states, in which processor can operate.

\paragraph{\textbf{Power consumption model}}
The main aim of these models is to emulate the behavior of the real computing resource and the way it consumes energy. Due to a rich functionality and flexible environment description, DCWoRMS can be used to verify a number of theoretical assumptions and to develop new energy consumption models. Modeling of energy consumption is realized by the energy estimation plugin that calculates energy usage based on information about the resource power profile, resource utilization, and the application profile including energy consumption and heat production metrics. Relation between model and power profile is illustrated in Figure~\ref{fig:powerModel}.

\begin{figure}[tbp]
\centering
\includegraphics[width = 8cm]{fig/powerModel.png}
\caption{\label{fig:powerModel} Power consumption modeling}
\end{figure}

\paragraph{\textbf{Power management interface}}
DCWoRMS is complemented with an interface that allows scheduling plugins to collect detailed power information about computing resource components and to change their power states. It enables performing various operations on the given resources, including dynamically changing the frequency level of a single processor, turning off/on computing resources etc. The activities performed with this interface find a reflection in total amount of energy consumed by the resource during simulation. 

Presence of detailed resource usage information, current resource energy state description and functional energy management interface enables an implementation of energy-aware scheduling algorithms. Resource energy consumption becomes in this context an additional criterion in the scheduling process, which use various techniques to decrease energy consumption, e.g. workload consolidation, moving tasks between resources to reduce a number of running resources, dynamic power management, cutting down CPU frequency, and others.

\subsubsection{Air throughput management concept}

The presence of an air throughput concept addresses the issue of resource air-cooling facilities provisioning. Using the air throughput profiles and models allows anticipating the air flow level on output of the computing system component, resulting from air-cooling equipment management.

\paragraph{\textbf{Air throughput profile}}
The air throughput profile, analogously to the power profile, allows specifying supported air flow states. Each air throughput state definition consists of an air flow value and a corresponding power draw. It can represent, for instance, a fan working state. In this way, associating the air throughput profile with the given computing resource, it is possible to describe mounted air-cooling devices.
Possibility of introducing additional parameters makes the air throughput description extensible for new specific characteristics.

\paragraph{\textbf{Air throughput model}}
Similar to energy consumption models, the user is provided with a dedicated interface that allows him to describe the resulting air throughput of the computing system components like cabinets or server fans. The general idea of the air throughput modeling is shown in Figure~\ref{fig:airModel}. Accordingly, air flow estimations are based on detailed information about the involved resources, including their air throughput states. 

\begin{figure}[tbp]
\centering
\includegraphics[width = 8cm]{fig/airModel.png}
\caption{\label{fig:airModel} Air throughput modeling}
\end{figure}

\paragraph{\textbf{Air throughput management interface}}
The DCWoRMS delivers interfaces that provide access to the air throughput profile data, allows acquiring detailed information concerning current air flow conditions and changes in air flow states. The availability of these interfaces support evaluation of different cooling strategies.



\subsubsection{Thermal management concept}

The primary motivation behind the incorporation of thermal aspects in the DCWoRMS is to exceed the commonly adopted energy use-cases and apply more sophisticated scenarios. By the means of dedicated profiles and interfaces, it is possible to perform experimental studies involving temperature-aware workload placement.

\paragraph{\textbf{Thermal profile}}
Thermal profile expresses the thermal specification of resources. It consists of the definition of the thermal design power (TDP), thermal resistance and thermal states that describe how the temperature depends on dissipated heat. For the purposes of more complex experiments, introducing of new, user-defined characteristics is supported. The aforementioned values may be provided for all computing system components distinguishing them, for instance, according to their material parameters and/or models.

\paragraph{\textbf{Temperature estimation model}}
Thermal profile, complemented with the temperature measurement model implementation may introduce temperature sensors simulation. In this way, users have means to approximately predict the temperature of the simulated objects by taking into account basic thermal characteristics as well as the estimated impact of cooling devices. However, the proposed approach assumes some simplifications that ignore heating and cooling dynamics understood as a heat flow process.

Figure~\ref{fig:tempModel} summarizes relation between model and profile and input data.

\begin{figure}[tbp]
\centering
\includegraphics[width = 8cm]{fig/tempModel.png}
\caption{\label{fig:tempModel} Temperature estimation modeling}
\end{figure}

\paragraph{\textbf{Thermal resource management interface}}
As the temperature is highly dependent on the dissipated heat and cooling capacity, thermal resource management is performed via a power and air throughput interface. Nevertheless, the interface provides access to the thermal resource characteristics and the current temperature values


\subsection{Application performance modeling}\label{sec:apps}

In general, DCWoRMS implements user application models as objects describing computational, communicational as well as energy requirements and profiles of the task to be scheduled. Additionally, simulator provides means to include complex and specific application performance models during simulations. They allow researchers to introduce specific ways of calculating task execution time. These models can be plugged into the simulation environment through a dedicated API and implementation of an appropriate plugin. To specify the execution time of a task user can apply a number of parameters, including:
\begin{itemize}
  \item  task length (number of CPU instructions)
  \item task requirements
  \item detailed description of allocated resources (processor type and
parameters, available memory)
  \item input data size
  \item network parameters
\end{itemize}
Using these parameters developers can for instance take into account the architectures of the underlying systems, such as multi-core processors, or virtualization overheads, and their impact on the final performance of applications.



\section{Modeling of energy efficiency in DCWoRMS}

DCWoRMS is an open framework in which various models and algorithms can be investigated as presented in Section \ref{sec:apps}. We discuss possible approaches to modeling that can be applied to simulation of energy-efficiency of distributed computing systems in this section. Additionally, to facilitate the simulation process, DCWoRMS provides some basic implementation of power consumption, air throughput and thermal models. We described them as examples and validate part of them by experiments in real computing system (in Section \ref{sec:experiments}).

The most common questions explored by researchers who study energy-efficiency of distributed computing systems is how much energy $E$ do these systems require to execute workloads. In order to obtain this value the simulator must calculate values of power $P_i(t)$ and load $L_i(t)$ in time for all $m$ computing nodes, $i=1..m$. Load function may depend on specific load models applied. In more complex cases it can even be defined as vectors of different resource usage in time. In a simple case load can be either idle or busy but even in this case estimation of job processing times $p_j$ is needed to calculate total energy consumption. The total energy consumption of computing nodes is given by (\ref{eq:E}):

\begin{equation}
E=\sum_i^m{\int_t{P_i(t)}} \label{eq:E}
\end{equation}


Power function may depend on load and states of resources or even specific applications as explained in \ref{sec:power}. Total energy can be also completed by adding constant power usage of components that does not depend on load or state of resources. 

In large computing systems which are often characterized by high computational density, total energy consumption of computing nodes is not the only result interesting for researchers. Temperature distribution is getting more and more important as it affects energy consumption of cooling devices, which can reach even half of a total data center energy use. In order to obtain accurate values of temperatures heat transfer simulations based on the Computational Fluid Dynamics (CFD) methods  have to be performed. These methods require as an input (i.e. boundary conditions) a heat dissipated by IT hardware and air throughput generated by fans at servers' outlets. Another approach is based on simplified thermal models that without costly CFD calculations provide rough estimations of temperatures. DCWoRMS enables the use of either approaches. In the the former, the output of simulations including power usage of computing nodes in time and air throughput at node outlets can be passed to CFD solver. This option is further elaborated in Section \ref{sec:coolemall}. Simplified thermal models required by the latter approach are proposed in \ref{sec:thermal}.


\subsection{Power consumption models}\label{sec:power}

As stated above power usage of computing nodes depend on a number of factors. 

Generally, the power consumption of a modern CPU is given by the Ohm's law: 
%\cite{intel_speedstep}:

\begin{equation}
P=C\cdot V_{core}^{2}\cdot f\label{eq:ohm-law}
\end{equation}

with $C$ being the processor switching capacitance, $V_{core}$ the
current P-State's core voltage and $f$ the frequency. Based on the
above equation it is suggested that although the reduction of frequency
causes an increase in the time of execution, the reduction of frequency
also leads to the reduction of $V_{core}$ and thus the power savings
from the $P\sim V_{core}^{2}$ relation outweigh the increased computation
time. However, experiments performed on several HPC servers shown that this dependency does not reflect theoretical shape and is often close to linear as presented in Figure \ref{fig:power}. This phenomenon can be explained by impact of other component than CPU and narrow range of available voltages. A good example of impact by other components is power usage of servers with visible influence of fans as illustrated in Figure \ref{fig:fans_P}.

%
\begin{figure}\label{fig:power}
\centering
\includegraphics[width=6cm]{fig/power_default}
\caption{Average power usage with regard to CPU frequency\protect \\
\textbf{Tests}: Linpack (\emph{green}), Abinit (\emph{purple}),
Namd (\emph{blue}) and Cpuburn (\emph{red}). \protect \\
}
%
\end{figure}

\begin{figure}[tbp]
\centering
\includegraphics[width = 8cm]{fig/power-fans.png}
\caption{\label{fig:fans_P} Power in time for highest frequency}
\end{figure}


For these reasons, DCWoRMS allows users to define dependencies between power usage and resource states (such as CPU frequency) in the form of tables or arbitrary functions using energy estimation plugins.

The energy consumption models provided by default can be classified into the following groups, starting from the simplest model up to the more complex ones. Users can easily switch between the given models and incorporate new, visionary scenarios.

\textbf{Static approach} is based on a static definition of resource power usage. This model calculates the total amount of energy consumed by the computing resource system as a sum of energy, consumed by all its components (processors, disks, power adapters, etc.). More advanced versions of this approach assume definition of resource states along with corresponding power usage. This model follows changes of resource power states and sums up the amounts of energy defined for each state.
In this case, specific values of power usage are defined for all discrete $n$ states as shown in (\ref{eq:static}): 

\begin{equation}
S_1 \to P_1, S_2 \to P_2, ..., S_n \to P_n  \label{eq:static}
\end{equation}

\textbf{Resource load} model extends the static power state description and enhances it with real-time resource usage, most often simply the processor load. In this way it enables a dynamic estimation of power usage based on resource basic power usage and state (defined by the static resource description) as well as resource load. For instance, it allows distinguishing between the amount of energy used by idle processors and processors at full load. In this manner, energy consumption is directly connected with power state and describes average power usage by the resource working in a current state.
In this case, specific values of power usage are defined for all pairs state and load values (discretized to $l$ values) as shown in (\ref{eq:load}): 

\begin{equation}
(S_1, L_1) \to P_{11}, (S_1, L_2) \to P_{12}, ...,  (S_2, L_1) \to P_{21}, ..., (S_n, L_l) \to P_{nl},  \label{eq:load}
\end{equation}

\textbf{Application specific} model allows expressing differences in the amount of energy required for executing various types of applications at diverse computing resources. It considers all defined system elements (processors, memory, disk, etc.), which are significant in total energy consumption. Moreover, it also assumes that each of these components can be utilized in a different way during the experiment and thus have different impact on total energy consumption. To this end, specific characteristics of resources and applications are taken into consideration. Various approaches are possible including making the estimated power usage dependent on defined classes of applications, ratio between CPU-bound and IO-bound operations, etc.
In this case, power usage is an arbitrary function of state, load, and application characteristics as shown in (\ref{eq:app}): 

\begin{equation}
f(S, L, A) \to P \label{eq:app}
\end{equation}

\subsection{Air throughput models}\label{sec:air}

The DCWoRMS comes with the following air throughput models.
By default, air throughput estimations are performed according to the first one.

\textbf{Static} model refers to a static definition of air throughput states. According to this approach, output air flow depends only on the present air cooling working state and the corresponding air throughput value. Each state change triggers the calculations and updates the current air throughput value. This strategy requires only a basic air throughput profile definition.

\textbf{Space} model allows taking into account a duct associated with the investigated air flow. On the basis of the given fan rotation speed and the obstacles before/behind the fans, the output air throughput can be roughly estimated. To this end, additional manufacturer's specifications will be required, including resulting air velocity values and fan duct dimensions. Thus, it is possible to estimate the air flow level not only referring to the current fan operating state but also with respect to the resource and its subcomponent placement. More advanced scenario may consider mutual impact of several air flows.

\subsection{Thermal models}\label{sec:thermal}

\begin{figure}[tbp]
\centering
\includegraphics[width = 8cm]{fig/temp-fans.png}
\caption{\label{fig:tempModel} Temperature in time for highest frequency}
\end{figure}


The following models are supported natively. By default, the static strategy is applied.

\textbf{Static} approach follows the changes in heat, generated by the computing system components and matches the corresponding temperature according to the specified profile. Since it tracks the power consumption variations, corresponding values must be delivered, either from power consumption model or on the basis of user data. Replacing the appropriate temperature values with function based on the defined material properties and/o experimentally measured values can easily extend this model.

\textbf{Ambient} model allows taking into account the surrounding cooling infrastructure. It calculates the device temperature as a function adopted from the static approach and extends it with the influence of cooling method. The efficiency of cooling system may be derived from the current air throughput value.

\section{Experiments and evaluation}\label{sec:experiments}

Results + RECS and MOP description

....

In this section, we present computational analysis that were conducted to emphasize the role of modelling and simulation in studying computing systems performance. We carried out two types of experiments. The former one aimed at demonstrating the capabilities of the simulator in termis of verifying the research hypotheses. The latter set of experiments was performed on the CoolEmAll testbed and then repeated using DCWoRMS tool. The comparative analysis of obtained results shows the reproducibility of experiments and prove the correctness of the adopted models and assumptions.

\subsection{Testbed description}

The RECS Cluster System is an 18 node computer system that has an monitoring and controlling mechanism integrated. Through the integrated novel monitoring approach of the RECS Cluster System the network load can be reduced, the dependency of polling every single
compute node at operation system layer can be avoided. Furthermore this concept build up a basis on which new monitoring- and controlling-concepts can be developed. Therefore, each compute node of the RECS Cluster Server is connected to an Operation System independent microcontroller that collects the most important sensor data like temperature, power consumption and the status (on/off) from every single node.


\begin {table}[ tp]

\begin{tabular}{llr}
\hline
\multicolumn{3}{c}{Nodes} \\
Type & Memory (RAM) & Count  \\
\hline
 Intel i7 & 16 GB &     4 \\
AMD Fusion T40N 64 Bit & 4 GB   & 6 \\
Atom D510 64 Bit & 2 GB &       4 \\
Atom Z510 VT &2 GB &    4 \\
\hline
\multicolumn{3}{c}{Storage} \\
Type & Size & Connection  \\
\hline
Storage Head 520 & 16 x 300 GB SSD & 2 x 10 Gbit/s CX4 \\
\hline
\end{tabular}
\caption {CoolEmAll testbed}
\end {table}

%Node i7, 16 GB RAM     4
%Node AMD Fusion T40N Dualcore, 1,0 Ghz, 4 GB (64 Bit)  6
%Node Atom D510 64 Bit, 2 GB    4
%Node Atom Z510 VT, 2 GB        4
%RECS | Storage Head 520, 16 x 300 GB SSD, 2 x 10 Gbit/s CX4

\subsection{Computational analysis}

TODO - experiments

\section{DCWoRMS application/use cases}\label{sec:coolemall}

DCWoRMS in CoolEmAll, integration with CFD

...

Being based on the GSSIM framework, that has been successfully applied in a substantial number of research projects and academic studies, DCWoRMS with its sophisticated energy extension has become an essential tool for studies of energy efficiency in distributed environments. For this reason, it has been adopted within the CoolEmAll project as a component of Simulation, Visualisation and Decission Support (SVD) Toolkit. In general the main goal of CoolEmAll is to provide advanced simulation, visualisation and decision support tools along with blueprints of computing building blocks for modular data centre environments. Once developed, these tools and blueprints should help to minimise the energy consumption, and consequently the CO2 emissions of the whole IT infrastructure with related facilities. The SVD Toolkit is designed to support the analysis and optimization of IT modern infrastructures. For the recent years the special attention has been paid for energy utilized by the data centers which considerable contributes to the data center operational costs.  Actual power usage and effectiveness of energy saving methods heavily depends on available resources, types of applications and workload properties. Therefore, intelligent resource management policies are gaining popularity when considering the energy efficiency of IT infrastructures.
Hence, SVD Toolkit integrates also workload management and scheduling policies to support complex modeling and optimization of modern data centres.

The main aim of DCWoRMS within CoolEmAll project is to enable studies of dynamic states of IT infrastructures, like power consumption and air throughput distribution, on the basis of changing workloads, resource model and energy-aware resource management policies.
In this context, DCWoRMS takes into account the specific workload and application characteristics as well as detailed resource parameters. It will benefit from the CoolEmAll benchmarks and classification of applications and workloads. In particular various types of workload, including data centre workloads using virtualization and HPC applications, may be considered. The knowledge concerning their performance and properties as well as information about their energy consumption and heat production will be used in simulations to study their impact on thermal issues and energy efficiency. Detailed resource characteristics, will be also provided according to the CoolEmAll blueprints. Based on this data, workload simulation will support evaluation process of various resource management approaches. These policies may include a wide spectrum of energy-aware strategies such as workload consolidation/migration, dynamic switching off nodes, Dynamic Voltage and Frequency Scaling (DVFS), and thermal-aware methods. In addition to typical approaches minimizing energy consumption, policies that prevent too high temperatures in the presence of limited cooling (or no cooling) may also be analyzed. Moreover, apart from the set of predefined strategies, new approaches can easily be applied and examined.
The outcome of the workload and resource management simulation phase is a distribution of power usage and air throughput for the computing models specified within the SVD Toolkit. These statistics may be analyzed directly by data centre designers and administrators and/or provided as an input to the CFD simulation phase. The former case allows studying how the above metrics change over time, while the latter harness CFD simulations to identify temperature differences between the computing modules, called hot spots. The goal of this scenario is to visualise the behavior of the temperature distribution within a server room with a number of racks for different types of executed workloads and for various policies used to manage these workloads.


\section{Conclusions and future work}

TODO - Conclusions and future research


\label{}

%% The Appendices part is started with the command \appendix;
%% appendix sections are then done as normal sections
%% \appendix

%% \section{}
%% \label{}

%% References
%%
%% Following citation commands can be used in the body text:
%% Usage of \cite is as follows:
%%   \cite{key}          ==>>  [#]
%%   \cite[chap. 2]{key} ==>>  [#, chap. 2]
%%   \citet{key}         ==>>  Author [#]

%% References with bibTeX database:

%%\bibliographystyle{model1-num-names}
%%\bibliography{<your-bib-database>}

%% Authors are advised to submit their bibtex database files. They are
%% requested to list a bibtex style file in the manuscript if they do
%% not want to use model1-num-names.bst.

%% References without bibTeX database:

 \begin{thebibliography}{00}

%% \bibitem must have the following form:
%%   \bibitem{key}...
%%

% \bibitem{}

\bibitem{CloudSim} Rodrigo N. Calheiros, Rajiv Ranjan, Anton Beloglazov, Cesar A. F. De Rose, and Rajkumar Buyya, CloudSim: A Toolkit for Modeling and Simulation of Cloud Computing Environments and Evaluation of Resource Provisioning Algorithms, Software: Practice and Experience (SPE), Volume 41, Number 1, Pages: 23-50, ISSN: 0038-0644, Wiley Press, New York, USA, January, 2011.

\bibitem{DCSG} http://dcsg.bcs.org/welcome-dcsg-simulator

\bibitem{DCD_Romonet} http://www.datacenterdynamics.com/blogs/ian-bitterlin/it-does-more-it-says-tin\%E2\%80\%A6

\bibitem{Ghislain} Ghislain Landry Tsafack Chetsa, Laurent LefÚvre, Jean-Marc Pierson, Patricia Stolf, Georges Da Costa. “DNA-inspired Scheme for Building the Energy Profile of HPC Systems”. In: International Workshop on Energy-Efficient Data Centres, Madrid, Springer, 2012

\bibitem{GreenCloud} D. Kliazovich, P. Bouvry, and S. U. Khan, A Packet-level Simulator of Energy- aware Cloud Computing Data Centers, Journal of Supercomputing, vol. 62, no. 3, pp. 1263-1283, 2012

\bibitem{GSSIM} S. Bak, M. Krystek, K. Kurowski, A. Oleksiak, W. Piatek and J. Weglarz, GSSIM - a Tool for Distributed Computing Experiments, Scientific Programming Journal, vol. 19, no. 4, pp. 231-251, 2011.

\bibitem{GSSIM_Energy} M. Krystek, K. Kurowski, A. Oleksiak, W. Piatek, Energy-aware simulations with GSSIM. Proceedings of the COST Action IC0804 on Energy Efficiency in Large Scale Distributed Systems, 2010, pp. 55-58.

\bibitem{GWF} http://gwa.ewi.tudelft.nl/

\bibitem{SLURM} https://computing.llnl.gov/linux/slurm/

\bibitem{SWF} Parallel Workload Archive, http://www.cs.huji.ac.il/labs/parallel/workload/ 

\bibitem{TORQUE} http://www.adaptivecomputing.com/products/open-source/torque/


 \end{thebibliography}


\end{document}

%%
%% End of file `elsarticle-template-1-num.tex'.