Implementing Dynamically Controlled Stream Pattern

Boris Lublinsky from Lightbend

May 14th, 2019, 2:10 PM (duration: 40 minute)

Location: The Native

A common problem in building streaming application is integration of stream processing systems with other data existing in an enterprise in the form of enterprise services. Traditional approaches to this problem use RPC-based service invocations from within stream processing. The advantages of such an approach are: - Simple integration with existing technologies and organizational processes - Easier to understand if you come from a non-streaming world On another hand, such an approach can suffer from the following drawbacks: - Worse latency, due to remote calls instead of local function calls - Coupling the availability, scalability, and latency/throughput of your streaming application with the SLAs of the RPC interface An alternative that I will cover is to use stateful stream processing, with the ability to update the state dynamically as the state of the services changes – i.e., dynamically controlled streams. I will explain a pattern and show how to implement it using different technologies, including Apache Spark, Apache Flink, Akka Streams and Apache Kafka