Towards Optimal Collaboration of Policies in the Two-Phase Scheduling of Cloud Tasks

. The use of virtualization technology makes software applications more scalable and cost effective when they are deployed over cloud computing platforms, but virtualization technology also brings challenges to task scheduling over cloud. The commonly used list scheduling schemes split the scheduling process into two phases: ordering and dispatching. However, majorities of recent researches about scheduling of cloud tasks concentrate on optimizing the schedulers’ performance in one phase, but seldom consider the collaborations of scheduling policies used in different phases. This paper summarizes some representative ordering and dispatching policies used in list schedulers, models the execution processes of these ordering and dispatching policies using Stochastic Petri Nets (SPN), and simulates the list scheduling process of cloud tasks. Based on the modeling and experimental results, we further evaluate which composition of ordering and dispatching policies provides optimal performance in the two-phase scheduling process of cloud tasks.


Introduction
Cloud computing service providers are interested in improving the task scheduling mechanisms to provide better Quality of Service (QoS) to their users. However, with the increasing scale of cloud computing platform, a single cloud may host and provide more and more diverse services, which brings new challenges to the scheduling of cloud tasks. The dynamicity of virtual environment [1], diversity of software applications [2] and elasticity of cloud services [3] exacerbate the dynamicity and complexity of the scheduling mechanisms: a cloud task's processing rate tends to have significant variations since the virtual resources allocated to the same task may be different. Hence, how to optimally dispatch the task requests to the proper virtual resources for processing is an important issue. Moreover, there will also be times when a task's arrival rate is greater than its maximum possible processing rate, then how to determine an optimal task execution order to reduce the peak load of a cloud computing platform is another significant issue.
List schedulers [4] are widely used in the dynamic scheduling of cloud tasks. As shown in Fig. 1, a list scheduler splits the scheduling process into two phases: one is the ordering phase, where tasks are sorted according to a specific ordering policy; the other is the dispatching phase, where tasks are dispatched to the allocated virtual resources (VMs) for processing according to a specific dispatching policy. The existing studies on task scheduling over cloud mostly concentrate on improving the schedulers' performance in one phase, such as optimizing the task processing sequence in the first phase to minimize the overall response time [4][5][6][7][8][9][10][11]; or optimizing the assignment of the task requests to the VMs to improve the utilization of cloud resources [12][13][14][15][16][17][18] and to ensure the fairness of each task [19][20][21]. However, only few researches have further studied the optimal collaboration of ordering and dispatching policies in different phases [22]. This paper summarizes some representative ordering and dispatching policies which are widely used in both industry and academic communities. Using Stochastic Petri Nets (SPN), we theoretically and consistently model the execution processes of these ordering and dispatching policies for the first time, and analyze the performance of a list scheduler under different compositions of policies. Based on the modeling and experimental results, we further evaluate which composition of ordering and dispatching policies provide optimal performance in the two-phase scheduling process of cloud tasks.
The rest of this paper is organized as follows. Section 2 summarizes representative ordering and dispatching policies. Section 3 proposes a Stochastic Petri Net model to describe the scheduling process, studies some features of different ordering and dispatching policies and analyzes the performance of a list scheduler under different ordering and dispatching policies. Section 4 evaluates the performance of the list scheduler under different collaborations of ordering and dispatching policies. Section 5 concludes the paper.

Scheduling Policies in the Two Phases
This section summarizes some representative ordering and dispatching policies used in the list scheduler of cloud computing platforms. These policies are commonly used in both industry and academic communities.

Ordering policies in the first phase
When the average task arrival rate exceeds the platform's processing rate, the unhandled tasks will be stalled in the scheduling queue, and the platform tends to be overloaded. At this time, an appropriate ordering policy is needed to determine an optimal task processing sequence to slow down the growth rate of the scheduling queue. The commonly used ordering policies are listed below: Random ordering policy: This policy randomly chooses one type of the cloud tasks for processing. All the tasks have equal chances to be scheduled in this policy, which achieves better fairness of tasks.
First in first out (FIFO) policy: This is a simple and widely used ordering policy, which makes the tasks to be executed in the order they arrived. This policy ensures all the tasks suffer almost the same waiting delay in the ordering phase.
Shortest remaining time first (SRTF) policy: This policy ensures the smallest task in the scheduling queue to be scheduled first [5], aiming to improve the responsiveness of the scheduler when the load is heavy.
Longest remaining time first (LRTF) policy: This policy ensures the largest task in the scheduling queue to be scheduled first, aiming to improve the throughput of the scheduler.
Myopic MaxWeight policy: This policy can be viewed as an improvement of SRTF or LRTF policy. Neither SRTF nor LRTF policy is a starvation-free ordering since the large sized tasks under SRTF or the small sized tasks under LRTF are likely to wait forever when the load is heavy [22]. To solve this problem, a weight is assigned to each task in the scheduling queue in this policy, and the value of a task's weight is the task's size (for LRTF) or task's execution rate (for SRTF) multiplies its waiting time. This policy schedules the task with the maximum weight first, which ensures the tasks suffering long waiting time have chances to be processed.

Dispatching policies in the second phase
The dispatching policies assign tasks to virtual resources for processing. An appropriate dispatching policy optimizes the resource utilization of a cloud platform and further improves the performance of the scheduler. The commonly used dispatching policies are listed below: Random routing policy (RR): This policy randomly dispatches a task to one of the VMs for processing, which ensures each allocated VM has an equal chance to process a task.
Shortest queue routing policy (SQR): This policy dispatches a task to the VM with the shortest processing queue, which achieves better fairness by balancing the load on each VM [19,20].
Shortest expected delay routing policy (SEDR): This policy dispatches a task to the VM with the shortest processing delay, which can be viewed as a generalization of the SQR policy for homogeneous servers [12].
Overall shortest expected delay routing policy (OSEDR): This policy dispatches a task to the appropriate VM to keep the overall processing delay of a server minimum [21], it optimizes the sum of the execution delays of all the tasks on each server.

Modeling and Analysis of Scheduling Policies
In this section, we describe the list scheduling process using the SPN model. By modeling different ordering and dispatching policies, we formulate some performance metrics (e.g. average resource utilization of a service). Furthermore, we evaluate the performance of a list scheduler according to our modeling results.

Arriving of tasks
Ordering of tasks Dispatching of tasks   Table 1. The scenario considered here is dynamic or online scheduling of small sized tasks onto a fixed nonpreemptive cloud system.
Probability that a request of task i passes Di at time t

M i
Transportation of task i from global scheduling queue to a specific VM for processing For analytic tractability, we assume the arrival of task i satisfies Poisson distribution with mean rate λ i . After arriving, a token representing task i in the SPN will come to place AQ i and decide if it can pass through transition L i to be buffered in the global scheduling queue. For analytic tractability, we assume that the size of the global scheduling queue is properly set to avoid task losses.
An ordering policy determines the value of P i [D(t)], which is the probability that a request of task i in the scheduling queue can be handled at time t. To accurately describe the ordering policies, we assume that there exists a sub logical scheduling queue WQ i which buffers the arrived requests for task i. Suppose the arrival rate of task i changes to λ i after the token passes through transition L i , then A dispatching policy determines the enabling function of each transition F ij , which further determines the value of P ij [F(t)]. P ij [F(t)] is the probability that a request of task i is dispatched to a VM located on server j for processing. Finally, the arrival rate of task i to its processing queue TQ ij on server j is: Then the tasks in the processing queue will be serially processed by the program running on the corresponding VM, which is similar to the scenario described in M/G/1 model. For analytic tractability, we assume the processing rate of each task i on server j is exponentially distributed with mean value μ ij , and the tasks' sizes are sorted in ascending order from task 1 to task n. Suppose the VMs on the same server have the same configuration, then the execution rates of the tasks dispatched to the same server satisfy: Since the utilization metrics and responsiveness metrics are the most frequently used metrics to evaluate the performance of a scheduling policy, we derive the expressions of average resource utilization and average completion time of task i based on existing modeling results introduced in queue theory [24]: The value of ρ avg (i) and T avg are determined by the value of P i [D(t)] and P ij [F(t)]. Hence, we need to derive the expressions of P i [D(t)] and P ij [F(t)] under different ordering and dispatching policies.

Modeling of ordering policies
First, we derive the expressions of P i [D(t)] in the steady-state under different ordering policies.
Random ordering policy. Since this policy randomly chooses one type of the cloud tasks for processing, the probability is time-independent, so the expression of P i [D(t)] in the steady-state is: Using FIFO policy, the value of P i [D(t)] in the steady-state is positively correlated to the arrival rate of task i [24]. Thus: This policy ensures the smallest task in the queue to be scheduled first. Using this policy, if task i is to be processed, then the logical scheduling queue of task 1 to task i-1 should be empty, hence the expression of P i [D(t)] is: Consider the steady-state, if the logical scheduling queue of task j is not empty, then at least one of this task's processing queue is full. Therefore, we get: This policy ensures the largest task in the scheduling queue to be scheduled first. Similar to the derivation in SRTF policy, we get the expression of Myopic MaxWeight policy. This policy determines the weight of a task based on a task's size and waiting time. To approximately model a task's waiting time, we assume that the waiting time of task i is positively correlated to WQ i (t), (If the first task in WQ i cannot be handled, all the followed tasks will be blocked. So the queue length reflects the waiting time of the first task). Based on SRTF policy, the expression of P i [D(t)] under Myopic MaxWeight policy should be modified as: Based on the above derivations, we get the specific expressions of P i [D(t)] under different ordering policies, which is the foundation of further evaluations.

Modeling of ordering policies
In the second phase, we need to formulate P ij [F(t)] in the steady-state under different dispatching policies.
Random routing policy (RR). This policy randomly dispatches a task to one of the m allocated VMs for processing, so the probability is time-independent: Shortest queue routing policy (SQR). This policy dispatches a task to the VM with the shortest processing queue. The probability P ij [F(t)] can be formulated as: Consider the steady-state, the probability that task i is dispatched to VM located on server j is positively correlated to its expected processing rate on server j. Therefore, the expression of P ij [F(t)] in the steady-state is: Shortest expected delay routing policy (SEDR). This policy dispatches a task to the VM with the shortest processing delay. P ij [F(t)] can be formulated as: Consider the steady-state, the probability that task i is dispatched to VM located on server j is positively correlated to the square of its processing rate on server j. Therefore, the expression of P ij [F(t)] in the steady-state is: Overall shortest expected delay routing policy (OSEDR). This policy dispatches a task to the appropriate VM to keep the overall processing delay of a server minimum. The probability P ij [F(t)] can be formulated as: This policy minimizes the sum of all the tasks' processing delays on a server, aiming to improve the overall execution time of a server. However, in the scenario described in Fig. 1, the VMs located on the same server will process different tasks simultaneously; hence the optimization goal should be the processing delay of the slowest task, but not the sum of processing delays. This policy may be not applicable in the scheduling of cloud tasks, hence will not be modeled in the following context.

Performance analysis of scheduling policies
Substitute the expressions of P i [D(t)] and P ij [F(t)] into formula (1), we can calculate the value of ρ avg (i), which can be further used to evaluate the performance of different collaborations of ordering and dispatching policies. Some useful conclusions are illustrated below: Based on the modeling results, we prove that SEDR and SQR optimize the utilization of each task in average and worst cases respectively, no matter what ordering policy they are collaborated with.
Optimizing the average resource utilization of task i is to minimize the value of ρ avg (i). Substitute the expression of P ij [F(t)] into (1), we get the expressions of ρ avg (i) under different dispatching policies: SQR: Calculate the value of (4)-(3): Thus, SEDR policy provides better resource utilization of a task than SQR policy in average case. Similarly, we can prove that SEDR policy provides better resource utilization of a task than the other dispatching policies shown in this paper. Due to the space limitation, we omit the proof details here.
On the other hand, in the worst case, a request of task i is likely to be blocked in the scheduling queue when task i's processing queue with the heaviest load is full. Hence the worst case resource utilization of task i is determined by the utilization of task i's VM on the heaviest load: An optimal scheduling policy should minimize the value of ρ worst (i). We know ρ worst (i) gets the minimum value when: To satisfy (5), the expression of P ij [F(t)] should be: We find that (6) is the same as the expression of P ij [F(t)] under SQR policy. Hence SQR dispatching policy provides optimal utilization of tasks in the worst case.
Next, we prove that, when the requests of different tasks are arriving uniformly, SRTF is the optimal ordering policy to collaborate with SEDR in the average case, while LRTF is the worst ordering policy to collaborate with SEDR.
Average resource utilization of all the tasks can be expressed as follows: Since the dispatching policy SEDR is chosen, the value of P ij [F(t)] is determined. The requests of different tasks are uniformly arrived, so the value of λ i is basically identical. And we have assumed μ 1j ≥ μ 2j ≥ …≥ μ nj , hence we get: And the value range of ρ total should be: At this time, the value of ρ total is minimum when: And the value of ρ total is maximum when: In the average case, the processing queue is not full, hence: By calculating the value of P i [D(t)] under different ordering policies, we get the following conclusions:  The value of P 1 [D(t)] in the steady-state is closest to 1 under the SRTF ordering policy, which indicates that SRTF is the optimal ordering policy to cooperate with SEDR  The value of P n [D(t)] in the steady-state is closest to 1 under the LRTF ordering policy, which indicates that LRTF is not an appropriate ordering policy to achieve optimal resource utilization.  The value of P i [D(t)] is fixed under the FIFO ordering policy, which indicates that FIFO is a moderate ordering policy.  P 1 [D(t)] also gets the maximum value under the Myopic MaxWeight ordering policy, but the value is not so close to 1 compared with the value of P 1 [D(t)] under the SRTF policy. Thus, using Myopic MaxWeight policy, we can also achieve relatively good resource utilization in the average case, but it is not the best choice. If the requests of different tasks are not uniformly arrived, similarly we can prove that the optimal ordering policy to collaborate with SEDR should satisfy: However, in the worst case, the processing queue is nearly full, and when a request of task i is blocked in the scheduling queue, it will not be processed. So, there is a probability that: At this time, SRTF is not the optimal ordering policy since: And Myopic MaxWeigtht is an optimal ordering policy to cooperate with SQR, since it reduces the value of P 1 [D(t)], but also keep it maximum. The collaboration of Myopic MaxWeight ordering and SQR dispatching has been proved to be throughput optimal elsewhere [22,25].
Using our model, we can also evaluate the average responsiveness of the tasks under different collaborations of ordering and dispatching policies, by substituting the expressions of P i [D(t)] and P ij [F(t)] into (2) and comparing the value of T avg . Due to the space limitation, we omit the evaluation of other performance metrics here.

Experiments and Evaluations
In this section, we realize different collaborations of ordering and dispatching policies in the scheduling process of cloud tasks. Based on the experimental results, we evaluate the performance of the list schedulers and validate our modeling results.
We simulate the scenario where a cloud computing platform comprises N servers, and processes tasks with 5 different sizes (i.e. 5 types of tasks). The tasks simulated here are the simple character counting tasks, thus we can approximately predict the processing time of a task according to the task's size. The task scheduling process simulated in our experiment is the same as shown in Fig. 1. The arrival rate of task i satisfies Poisson distribution with mean rate λ i , and the requests of different tasks are uniformly arrived, thus λ 1 = λ 2 =…= λ 5 . The tasks' sizes are sequenced in ascending order, hence the processing rate of each task on the platform satisfies: μ 1j ≥ μ 2j ≥…≥ μ 5j (1≤ j≤ N). The simulation time is set long enough (1000 time units) to make the scheduling process reach its steady-state.
Since the utilization metrics reflect the throughput of a system, we record the percentage of processed tasks at each time point to evaluate the throughput of the platform. To make our simulation more convincible, we simulate dynamic scheduling of tasks onto a large scale cloud system with 500 servers (N=500). Twenty-five list schedulers work together to order the arrived tasks and dispatch them to 2500 VMs located on the 500 cloud servers. Fig. 3 shows the throughput of the cloud computing platform at each time point under different collaborations of ordering and dispatching policies. We simulated the collaborations of all the dispatching policies introduced in this paper and ordering policies FIFO (shown in Fig. 3A), SRTF (shown in Fig. 3B), LRTF (shown in Fig.  3C) and Myopic MaxWeight (shown in Fig. 3D).
We can see from Fig. 3A-3D that, no matter what ordering policy is used, SQR and SEDR dispatching policies achieve better throughput to the cloud platform among the four dispatching policies. This phenomenon validates our modeling results in Section 3 that SEDR and SQR achieve better resource utilization of a task in the scheduling process.
We also notice that the scheduling results using OSEDR is not effective: sometimes, the scheduling results of OSEDR are even no better than Random policy. The reason of this phenomenon is that optimizing the sum of execution delays on a server is not reasonable in cloud computing platforms just as previously discussed in Section 3, since the VMs co-located on the same cloud server can process different tasks simultaneously, the applicable dispatching policy is to optimize the execution delay of the slowest task on a server.
Although the platform achieves better throughput using SEDR and SQR policies, there are still some differences in the scheduling results. Using SEDR policy, the throughput is the best in the initial and middle periods of the scheduling (typically from time unit 1-600), but the throughput will then drop down below SQR policy; on the other hand, using SQR policy, the throughput of the platform rises to the best in the final period of the scheduling. This fact can be explained using the conclusions presented in Section 3: SEDR is the optimal dispatching policy to collaborate with in the average case, hence it provides best throughput in the initial and middle periods when the task loads are not heavy enough. However, when it comes to the final periods of the scheduling, the accumulated unhandled tasks will affect the scheduling results, and the scenario will be changed from the average case scheduling to the worst case scheduling. According to our conclusions in Section 3, SQR is the utilization optimal dispatching policy in the worst case; it explains the phenomenon that using SQR, the platform gets the best throughput in the final period of the scheduling. Comparing the results shown in Fig. 3A-3C, we find that the collaboration of SRTF ordering and SEDR dispatching achieves best throughput in the initial and middle scheduling periods; the collaboration of FIFO and SEDR performs moderately; and the collaboration of LRTF and SEDR performs worst. This fact confirms to our modeling results that SRTF is the optimal ordering policy to cooperate with SEDR in the average case when the task requests are uniformly distributed, while LRTF is the worst policy to collaborate with.
We notice from Fig. 3B and 3C that, using SRTF and LRTF policy, the throughput will suffer attenuation at a specific time point (around time unit 400-500 for LRTF, 700-800 for SRTF). This is because SRTF and LRTF are not starvation-free orderings: some tasks have already been stalled in the scheduling queue even if their processing queues are not full; to make things worse, if the shortest (for SRTF) or longest (for LRTF) remaining time tasks are blocked, all the arrived tasks at this time point will not be processed, which will dramatically attenuate the system's throughput close to zero at this time point. However, we can see in Fig. 3D that Myopic MaxWeight policy solves the throughput attenuation problem by improving the SRTF policy. The collaboration of Myopic MaxWeight and SQR provides best throughput in the final period of scheduling (after time unit 700), which theoretically and experimentally validate the conclusion that the collaboration of Myopic MaxWeight ordering and SQR dispatching is the utilization optimal list scheduling policy in the worst case.

Conclusion
We briefly summarize some representative ordering and dispatching policies which are commonly used in both industry and academic communities, model the execution processes of these ordering and dispatching policies using the SPN model, and apply different collaborated scheduling policies to schedule cloud tasks in simulation environment. Based on the modeling and experimental results, we have evaluated that SEDR and SQR are utilization optimal dispatching policies to collaborate with in the average case and worst case scheduling respectively; the collaboration of SRTF ordering and SEDR dispatching achieves optimal throughput in the average case, while the collaboration of Myopic MaxWeight ordering and SQR dispatching achieves optimal throughput in the worst case.