Lambda architecture, Kappa architecture, it’s all Greek to me!

Sometimes, it’s ok to wait for correct results of a long Extract-Transform-Load data process. Sometimes, it’s not: better have slightly correct results sooner. But if one still wants to have both, there’s an architecture for that. It’s called Lambda architecture, and it has been documented. One of the issues of Lambda architecture is that one needs to maintain two processes: the standard correct batching model, and the new real-time but incomplete streaming model. This translates to higher development costs, and can lead very higher maintenance costs. What if one could get the benefits of both worlds?

In this talk, I’ll present the Lambda architecture, its foundations, and how it’s usually implemented. I’ll also present the Kappa architecture that solves issues from the Lambda architecture. This will be done via some use-cases, banking and/or e-commerce. Finally, I’ll demo a sample of the Kappa architecture in action.