Dynamic resource management in a massively parallel stream processing engine

Kasper Grud Skat Madsen, Yongluan Zhou

Publikation: Kapitel i bog/rapport/konference-proceedingKonferencebidrag i proceedingsForskningpeer review

Abstrakt

The emerging interest in Massively Parallel Stream Processing Engines (MPSPEs), which are able to process long-standing computations over data streams with ever-growing velocity at a large-scale cluster, calls for efficient dynamic resource management techniques to avoid any waste of resources and/or excessive processing latency. In this paper, we propose an approach to integrate dynamic resource management with passive fault-tolerance mechanisms in a MPSPE so that we can harvest the checkpoints prepared for failure recovery to enhance the efficiency of dynamic load migrations. To maximize the opportunity of reusing checkpoints for fast load migration, we formally define a checkpoint allocation problem and provide a pragmatic algorithm to solve it. We implement all the proposed techniques on top of Apache Storm, an open-source MPSPE, and conduct extensive experiments using a real dataset to examine various aspects of our techniques. The results show that our techniques can greatly improve the efficiency of dynamic resource reconfiguration without imposing significant overhead or latency to the normal job execution.
OriginalsprogEngelsk
TitelProceedings of the 24th ACM International on Conference on Information and Knowledge Management
ForlagAssociation for Computing Machinery
Publikationsdato2015
Sider13-22
ISBN (Elektronisk)978-1-4503-3794-6
DOI
StatusUdgivet - 2015
Begivenhed24th ACM International on Conference on Information and Knowledge Management - Melbourne, Australien
Varighed: 19. okt. 201523. okt. 2015
Konferencens nummer: CIKM

Konference

Konference24th ACM International on Conference on Information and Knowledge Management
NummerCIKM
LandAustralien
ByMelbourne
Periode19/10/201523/10/2015

Fingeraftryk

Dyk ned i forskningsemnerne om 'Dynamic resource management in a massively parallel stream processing engine'. Sammen danner de et unikt fingeraftryk.

Citationsformater