Introduction: In the field of big data, data integration technology has a wide range of application scenarios, providing convenient solutions for data synchronization, sharing, and integration of heterogeneous data systems. This sharing session will delve into Xiaomi’s application and practical scenarios in data integration technology, including internal evolution, self-developed middleware engine capabilities, and platformization practices. The session will be divided into the following three aspects:

• Introduction to Data Integration Services Sharing the evolution route, overall service architecture, and service scale of data integration technology, explaining issues and solutions encountered in offline/real-time synchronization scenarios, multi-source heterogeneous synchronization implementation, link exception fault tolerance, performance, and stability.
• Analysis of Self-developed Engine Capabilities Explaining the practice of stream-batch integration, engine task orchestration model, and multiple optimization rules in practice. Explaining the implementation principle, overall technical capabilities, and design intention of the self-developed distributed engine.
• Platformization Capabilities Analyzing the capabilities of data quality and link monitoring, integration of heterogeneous data platforms, link data correction, and resource estimation. From the perspective of platformization, explaining the practice of data integration in the engine platformization process in actual applications.

Speakers:

Zheng Junyuan is a senior software development engineer at Xiaomi. He has participated in the construction of Xiaomi’s data platform and has many years of experience in big data development. He is committed to the construction of distributed computing and data integration technology fields.