Powered by Artificial Intelligence technology, BIGO’s video-based products and services have gained immense popularity, with users in more than 150 countries. These include Bigo Live (live streaming) and Likee (short-form video). Bigo Live is available in more than 150 countries and Likee has more than 100 million users and is popular among the Generation Z.

In the past few years, we have deployed many Kafka clusters to support real-time ETL and short-form video recommendation. The Apache Pulsar’s layered architecture and new features, such as Low latency with durability, Horizontally scalable, Multi-tenancy etc, help us solve a lot of problems in production. We have adopted Apache Pulsar to build our Message Processing System, especially in Real-Time ETL, short-form video recommendation and Real-Time Data report.

In this presentation, I will share our journals using KOP (Kafka on Pulsar) to provide seamless migration from Kafka to Pulsar, especially in terms of improving performance and stability. I will also discuss major cases of using Apache Pulsar in BIGO, such as millions of topics support, real-time machine learning, and integration with Flink and Flink SQL.

Speakers:

Hang Chen: Hang Chen, Apache Pulsar Committer and BIGO Staff Engineer, is the Leader of the Messaging Platform team from BIGO, and responsible for creating a centralized pub-sub messaging Platform which provides a vast number of service/application traffics. He introduced Apache Pulsar into their Messaging Platform and integrate with upstream and downstream systems, such as Flink, ClickHouse and other inner systems for Real-Time recommendation and analysis. He focus on pulsar performance tuning, new features development and pulsar ecosystem integration.