Publication: An elastic and traffic-aware scheduler ...
Master data
Title: | An elastic and traffic-aware scheduler for distributed data stream processing in heterogeneous clusters |
Subtitle: | |
Abstract: | Existing Data Stream Processing (DSP) systems perform poorly while encountering heavy workloads, particularly on clustered set of (heterogeneous) computers. Elasticity and changing application parallelism degree can limit the performance degradation in the face of varying workloads that negatively impact the overall application response time. Elasticity can be achieved by operator scaling, i.e., by replication and relocation in operators at runtime. However, scaling decisions at runtime is challenging, since it first increases the overall communication overhead between operators and secondly changes any initial scheduling that could lead to a non-optimal scheduling plan. In this paper, we investigate the problem of elasticity and scaling decisions and propose a DSP system called ER-Storm. To curb communication overhead, we propose a new 3-step mechanism for replication and relocation of operators upon detecting a bottleneck operator that overutilizes a worker node. The other challenge is to select the proper worker nodes to host relocated operators. By discretizing the input workload, we model the relocation of operators between worker nodes at runtime through a scalable Markov Decision Process (MDP) and use a model-free notion of reinforcement learning (Q-Learning) to find optimal solutions. We have implemented our propositions on the Apache Storm version 2.1.0. Our experimental results show that ER-Storm reduces the average topology response time by 20–60 percent based on the rate of input workload (low or high) compared to the R-Storm scheduler and the Online-Scheduler of Storm. |
Keywords: | Distributed data stream processing, Elasticity, Scheduling, Resource-awareness, Heterogeneous clusters, Reinforcement learning, Apache storm |
Publication type: | Article in journal (Authorship) |
Publication date: | 11.07.2022 (Online) |
Published by: |
Journal of Supercomputing
Journal of Supercomputing
(
Springer;
)
to publication |
Title of the series: | - |
Volume number: | - |
Issue: | - |
First publication: | Yes |
Version: | - |
Page: | pp. 1 - 38 |
Versionen
Keine Version vorhanden |
Publication date: | 07.2022 |
ISBN: | - |
ISSN: | 0920-8542 |
Homepage: | https://link.springer.com/article/10.1007/s11227-022-04669-z |
Publication date: | 11.07.2022 |
ISBN (e-book): | - |
eISSN: | 1573-0484 |
DOI: | http://dx.doi.org/10.1007/s11227-022-04669-z |
Homepage: | https://link.springer.com/article/10.1007/s11227-022-04669-z |
Open access |
|
Authors
Hamid Hadian (internal) |
Mohammadreza Farrokh (external) |
Mohsen Sharifi (external) |
Ali Jafari (external) |
Assignment
Organisation | Address | ||||
---|---|---|---|---|---|
Fakultät für Technische Wissenschaften
Institut für Informationstechnologie
|
AT - 9020 Klagenfurt am Wörthersee |
Categorisation
Subject areas | |
Research Cluster | No research Research Cluster selected |
Citation index |
Information about the citation index: Master Journal List
|
Peer reviewed |
|
Publication focus |
Classification raster of the assigned organisational units:
|
working groups |
|
Research activities
Projects: |
|
Publications: | No related publications |
Events: | No related events |
Lectures: | No related lectures |