Integrating Fault-tolerance and Elasticity in a Distributed Data Stream Processing System

Kasper Grud Skat Madsen, Philip Pontoppidan Thyssen, Yongluan Zhou

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Abstract

Recently there has been an increasing interest in building distributed platforms for processing of fast data streams. In this demonstration, we highlight the need for elasticity in distributed data stream processing systems and present Enorm, a data stream processing platform with focus on elasticity, i.e. the ability to dynamically scale resource usage according to the runtime workload fluctuations. In order to achieve dynamic scaling with minimal overhead and latency, we use an integrated approach for both fault-tolerance and elasticity. The idea is that both fault-tolerance and elasticity essentially require replicating or migrating computation states among different nodes. Integrating and sharing the state management operations between the two modules can not only provide abundant opportunities to reduce the system's runtime overhead but also simplify the system's architecture.
Original languageEnglish
Title of host publicationProceedings of the 26th International Conference on Scientific and Statistical Database Management
Number of pages4
PublisherAssociation for Computing Machinery
Publication date2014
Article number48
ISBN (Print)978-1-4503-2722-0
DOIs
Publication statusPublished - 2014
EventInternational Conference on Scientific and Statistical Database Management - Aalborg, Denmark
Duration: 30. Jun 20142. Jul 2014

Conference

ConferenceInternational Conference on Scientific and Statistical Database Management
Country/TerritoryDenmark
CityAalborg
Period30/06/201402/07/2014
SponsorDanish Otto Monsted Foundation, TARGIT

Keywords

  • Elasticity
  • Fault-tolerance
  • Mapreduce
  • Resource management

Fingerprint

Dive into the research topics of 'Integrating Fault-tolerance and Elasticity in a Distributed Data Stream Processing System'. Together they form a unique fingerprint.

Cite this