Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler for container orchestrator systems. It was created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.

As a custom K8s scheduler, YuniKorn can be easily deployable in K8s on-prem or on cloud. For this talk, we will talk about gaps in resource scheduling in Cloud-Native environment, and how YuniKorn can support running big data applications (like Spark/Flink/Tensorflow, etc.) on K8s.

We will share the advantages for onboarding Spark workloads in to K8s by using Yunikorn’s gang scheduling support. This is one of the biggest paintpoints for Big Data workload migration to Cloud-Native ecosystem. We will also share how YuniKorn being used in community partners such as Apple, Aplibaba, Cloudera, etc.

Speakers:

Sunil Govindan: Sunil is contributing majorly towards resource scheduling and is looking into Kubernetes, YARN big data workload scheduling. He is contributing to Apache Hadoop projects since 2013 in various roles as Hadoop Contributor, Hadoop Committer, and a member of the Project Management Committee (PMC). Majorly working on YARN Scheduling improvements / Multiple Resource types support in YARN etc. He is also serving as Apache Submarine PMC member, Apache YuniKorn (incubating) PMC member.

Julia Kinga Marton: Kinga is a Software Engineer at Cloudera, where initially she joined the Apache Oozie team. She is an Apache Oozie committer and PMC member. In early 2020 she joined the YuniKorn team and in the summer of 2020 she became an Apache YuniKorn (Incubating) committer. Before Cloudera she worked at Lufthansa Systems where she was contributing into a weight & balance solution for airlines.