Developing fast scalable Big Data applications has been made significantly easier over the last decade with horizontally scalable open-source databases and streaming technologies such as Apache Cassandra and Apache Kafka. Cloud-native trends have also accelerated the uptake and ease of use of these technologies, and they are available as managed services on multiple cloud platforms.But maybe it has become too easy to embark on building complex distributed applications using multiple massively scalable open-source technologies, as there are still many performance and scalability issues to be aware of.In this talk, I will give a high-level overview of some of the performance and scalability challenges I’ve overcome over the last six years building realistic demonstration applications using Apache Cassandra and Apache Kafka (and more), supplemented with performance insights from our operation of thousands of production clusters.

Speakers:

Paul Brebner: NetApp, Open Source Technology Evangelist, Paul is the Open Source Technology Evangelist at Instaclustr (now part of Spot by NetApp). For the past six years, he has been learning new scalable Big Data technologies, building realistic demonstration applications, and blogging and talking about a growing list of open-source technologies including Apache Cassandra, Apache Spark, Apache Kafka, Apache ZooKeeper, Redis, OpenSearch, PostgreSQL, Cadence, and many more.

Since learning to program on a VAX 11/780, Paul has extensive R&D, teaching, and consulting experience in distributed systems, technology innovation, software architecture and engineering, performance engineering, grid and cloud computing, and data analytics and machine learning.

Paul has also worked at Waikato University (New Zealand), University of New South Wales (UNSW, Sydney), Commonwealth Scientific and Industrial Research Organisation (CSIRO, Australia), University College London (UCL, UK), National ICT Australia (NICTA), Australian National University (ANU), and several tech start-ups (including as a Founder/CTO). Paul has an MSc (1st Class Hons, Waikato) in Machine Learning and a BSc (Computer Science and Philosophy, Waikato).