Kafka’s Solution : Event Driven Architecture: OTKafkaDiaries

Heroism often results as a response to extreme events.

James Geary

Event Driven Architecture:

Modern digital businesses work in real-time based events. Event-driven architecture is based on the design principle which follows loose-coupling and message-driven architecture. This Architecture helps to publish events/messages that applications and services can consume, and then perform an action based upon those events.

Where are we Today?

Back in the days when we started implementing microservices, were focused more on service decoupling, communication, and security which we were going to handle in such a system. 

Although we were able to achieve most of the things, the challenge today has shifted to delivering responses in real-time on a mass scale. So, even after decoupling the service into microservices, we can get service timeout errors when our depending service fails to return the response when it’s needed.

It sounds very interesting from an architectural perspective to solve the problem of real-time as it is such an extreme challenge. Like, today, we are seeing rising new use cases with microservices, IoT, event hubs, cloud, machine learning, and many more software solutions.

An increase in demand for such apps has led today’s era to EVENT Driven instead of data-driven.

Why Use events?

It took the era of big data for us to realize that while having lots of data is useful, things get slower as you have more of it. And yet, we are collecting more of it. Over the last few years, there has been a change in organizations that they must become real-time; to become real-time, they must be event-driven. 

For Instance,

Any Application that has to send/Notify its user; should not wait for the response all the time. They can just do it in real-time by publishing it on some topics and at the same time consumer applications in the next end can get the request/message from the producing application. (read pub-sub intro)

Social platform LinkedIn found this issue and developed a way of solving it. They created an in-house application named “Kafka”

One Example because that’s what the reader demands.

Brush Vocab: 

Events  – something which is going to happen

Trigger – an event that is responsible to provoke a behavior 

Behavior – a reaction to an event.

The Event example is simple: Check your calendar 🤓.  No? Okay then let’s see a real one where we are using event-driven architecture.

In most of the gaming applications, there are events set for most of the user action,

while playing GTA(a recent game I played) I checked that whenever I was fired at by some goons/police my health gets decreased (obviously), but as soon as I pick a vehicle my health gets increased, or another time when police are chasing me, and I was running away from them, there was siren walkie-talkie voice all over the game sound, (people will be gone from the game I cannot see any normal people, NPC, in the game other than me and the police ) but once I hide or get far from police radar(Voiceover: All party stop, we lost him) everything works normal (NPC’s will reappear). 

All these are events that get triggered on some action and change the behavior causing that action.

In AWS we set Autoscaling group and we tune it to some CPU condition which triggers SNS topic sending an email, or maybe scaling up down your instances

Conclusion: Where does it fit today?

Event-driven applications are common for use cases including IoT, fraud detection, payment processing, website monitoring, and real-time marketing.

Event-driven applications often treat data as immutable, or unchangeable, making it easy to look up the values of data at previous points in time. 

So whenever information “changes,” what actually happens is that a new data point is created with a new time period, which means you have old and new values.

Software Solutions :

• Kafka 
• Pulsar

Note: This is Series Blog A complete Kafka setup will follow this blog.