Spark on Kubernetes has gained more and more attention and usage. Due to the lack of Kubernetes' batch scheduling support, resource deadlocks often occur in big data scheduling scenarios. In addition, It lacks advanced capabilities such as queue, priority, resource reservation, and multiple computing force scheduling. This topic describes the latest developments and best practices of the Apache Spark community Support Customized Kubernetes Schedulers.

• Current status and challenges of Spark on Kubernetes
• Latest developments in the Spark community: Support Customized Kubernetes Schedulers
• Display Spark on Kubernetes scheduling capability through Volcano
• Demo: Demonstrates the overall functions of Spark + Volcano, including the queue, priority, resource reservation, and multiple computing force scheduling capabilities.

Speakers:

Yikun Jiang: Senior Software Engineer, Huawei Computing Open Source team senior software engineer, Apache Spark Contributor, openEuler Infra Maintainer, OpenStack storage project Committer, currently committed to support and improve multi-architecture projects in big data and cloud computing.

Leibo Wang: Huawei cloud container service architect, Huawei Cloud Container Service architect, Volcano community Tech-leader. He worked for Platform Computing, IBM, etc. More than 10 years of experience in large-scale distributed computing and high-performance computing. Familiar with cloud native, big data and AI acceleration. Focus on large-scale cluster resource management, scheduling, job scheduling engine design and development. Currently, I am mainly responsible for the research and development of Volcano cloud native batch computing platform.