Apache Celeborn(Incubating) is a high-performance, highly available, scalable universal Shuffle service that supports both of the major Spark/Flink engines (and more engines such as Tez/MR Will be supported in the future). Celeborn supports the production of Shuffle at dozens of P per day in Alibaba and many well-known companies, improving stability and performance while reducing costs. This presentation will cover Celeborn’s high-performance, high-availability core design, unified architecture that supports multiple engines, user stories, and how to better engage the community.

Speakers:

Zhou Keyong: Aliyun, EMR Spark Engine manager, Head of Alibaba Cloud EMR Spark engine, initial author of Apache Celeborn (Incubating), has some experience in Remote Shuffle Service, vectorization engine, optimizer, etc.