Resilient stream processing in edge computing

Abstract

The proliferation of Internet-of-Things (IoT) devicesis rapidly increasing the demands for efficient processing oflow latency stream data generated close to the edge of thenetwork. A large number of IoT applications require continuousprocessing of data streams in real-time. Examples include virtualreality applications, connected autonomous vehicles and smartcity applications. Although current distributed stream processingsystems offer various forms of fault tolerance, existing schemes donot understand the dynamic characteristics of edge computing in-frastructures and the unique requirements of edge computing ap-plications. Optimizing fault tolerance techniques to meet latencyrequirements while minimizing resource usage becomes a criticaldimension of resource allocation and scheduling when dealingwith latency-sensitive IoT applications in edge computing. In thispaper, we present a novel resilient stream processing frameworkthat achieves system-wide fault tolerance while meeting thelatency requirements for edge-based applications. The proposedapproach employs a novel resilient physical plan generation forstream queries and optimizes the placement of operators tominimize the processing latency during recovery and reducesthe overhead of checkpointing. We implement a prototype of theproposed techniques in Apache Storm and evaluate it in a realtestbed. Our results demonstrate that the proposed approach ishighly effective and scalable while ensuring low latency and low-cost recovery for edge-based stream processing applications.

Publication
In 2021 IEEE/ACM International Symposium on Cluster, Cloud andInternet Computing (CCGrid 2021)