Introduction To KAFKA: OTKafkaDiaries

Starting Our Journey:

This year’s pandemic has forced businesses all around the world to adopt a “remote-first” approach for executing daily operations. Although our lives have been greatly disrupted due to lockdown measures and economic impact, we have managed a balance in our social lives through online modes such as shopping, learning, messaging, gaming, and other activities.

Modern software design is also taking a remote-first mindset to ensure that users can collaborate and share information within each app, including the ability to interact with real-time data.

We are starting our journey to explore Kafka in this pandemic. Starting from “zero to hero“, this Kafka series will help you in understanding the Kafka concepts.

Why do we choose to Learn Kafka?

In our legacy architecture, we were following the usage of the database or relational databases with the same key-value storage. The only problem I see with this setup is that when you have an ever-growing data stream, there are lots of databases and other systems built to store data, but what was missing in our architecture was something that would help us to handle the “continuous flow of data”.

Though Kafka started from a social network, powering real-time applications and data flow behind the scenes, now you can see it at the heart of next-generation architectures in every industry imaginable. Big eCommerce web portals, finance companies, car companies, all these are collecting and processing real-time data streams from the internet.

So what is this Kafka thing all about? How does it compare to the systems you already know and use?

Think of Kafka as a streaming platform: a system that lets you publish and subscribe to streams of data, store them,` and process them, and that is exactly what Apache Kafka is built to be. {shown in img-OTkD1.0.jpg}

Concepts Before Deep-Diving Into Apache Kafka :

Event-driven Model :

Event-driven architectures are getting a lot of attention nowadays.

What is an Event: An Event can be anything it can be your bank balance or when you go outside to buy a jacket. Getting a covid vaccine is too an event Everything that carries a piece of data is an Event.

Event-Driven model:

Events are both a fact and a notification. Unlike the request/response model, they (events) travel in only one direction and expect no response (sometimes called “fire and forget”).

• Example: OrderCreated{Widget} , CustomerDetailsUpdated{Customer}

When to use: Events can be used when the loose coupling is important (e.g., in multiteam systems), where the event stream is useful to more than one service, or where data must be replicated from one application to another. Events also lend themselves to concurrent execution.

We can think of Git as the best example of event-driven and an eCommerce Website, so the main objective is whenever you click on buy, it packs certain information in the form of an event and drops it into the cloud. Then, other pieces of software even run that track your product which you have bought. So all these events run at different rates. So the rate of events is huge sometimes.

Pub/Sub Model:

Publish/Subscribe messaging model is a way that is characterized by the sender (publisher) of a data (often called a message) not sending directly it to a receiver. Instead, the publisher classifies the message in a topic (Kafka Topic), and that receiver (subscriber) subscribes to receive certain classes of messages within a topic. Pub/sub systems often have a broker, a central point where messages are published {shown in img-OTkD2.0.jpg}.

We can remove the dependencies of services to be online all the time. We will have one central message broker which makes sure that your services should sending (publishing) messages and at the same time, other services are consuming by subscribing to these message topics.

Even if any point of time service is not running, your broker(Kafka server) will be persisting those messages, later which can be consumed by consumers from the point they left(termed as -commit logs). So, even if you think of the worst-case which is a failure, you will have a single point of failure.

That’s all for now! This blog is a series blog. We will soon be publishing the next blog.

Till then Stay Healthy! Stay Safe! BACK TO WORK…