HotCSE Seminar
Computational Science & Engineering
Wednesday Nov 08, 12pm-1pm, 1116-E Klaus

Smoother Stream Processing Through Smarter Backpressure

Xin Chen
Advisor: Prof. Richard Vuduc


Distributed micro-batch streaming systems, such as Spark Streaming, employ backpressure mechanisms to maintain a stable, high throughput stream of results that is robust to runtime dynamics. Checkpointing in stream processing systems is a process that creates periodic snapshots of the data flow for fault tolerance. These checkpoints can be expensive to produce and add significant delay to the data processing. The checkpointing latencies are also variable at runtime, which in turn compounds the challenges for the backpressure mechanism to maintain stable performance. Consequently, the interferences caused by the checkpointing may degrade system performance significantly, even leading to exhaustion of resources or system crash. We present GOVERNOR, a controller that factors the checkpointing costs into the backpressure mechanism. It not only guarantees a smooth execution of the stream processing but also reduces the throughput loss caused by interferences of the checkpointing.


Xin Chen is a Ph.D. student supervised by Prof. Rich Vuduc. He earned his master degree in software engineering from Tsinghua University. His current research interests are in the field of high performance computing, GPU and computer graphics.