A commonly used monitoring and logging stack is the EFK stack<\/strong>, which includes Elasticsearch, Fluentd, and Kibana. In many production environments, Kafka is also introduced into this stack to handle log ingestion more reliably.<\/p>\n Kafka is an open-source event streaming platform and is widely used across organizations for handling high-throughput data streams.<\/p>\n This naturally raises an important question.<\/p>\n Why should Kafka be used along with the EFK stack?<\/p>\n In this blog, we will explore why Kafka is introduced, what benefits it brings, and how it integrates with the EFK stack.<\/p>\n Let\u2019s get started.<\/p>\n While traveling, we often see crossroads controlled by traffic lights or traffic police. At a junction where traffic flows from multiple directions, these controls ensure smooth movement by allowing traffic from one direction while holding others temporarily.<\/p>\n In technical terms, traffic is regulated by buffering and controlled flow<\/strong>.<\/p>\n Kafka plays a very similar role in log management.<\/p>\n Imagine hundreds of applications sending logs directly to Elasticsearch. During peak traffic, Elasticsearch may become overwhelmed. Scaling Elasticsearch during heavy ingestion is not always a good solution because frequent scaling and re-sharding can cause instability.<\/p>\n Kafka solves this problem by acting as a buffer layer<\/strong>. Instead of pushing logs directly to Elasticsearch, logs are first sent to Kafka. Kafka then delivers them in controlled, manageable batches to Elasticsearch. The complete flow consists of the following blocks.<\/p>\n Application containers or instances<\/p>\n<\/li>\n Kafka<\/p>\n<\/li>\n Fluentd forwarder<\/p>\n<\/li>\n Elasticsearch<\/p>\n<\/li>\n Kibana<\/p>\n<\/li>\n<\/ul>\n Each block is explained below with configurations.<\/p>\n This block represents application containers or EC2 instances where logs are generated. The td-agent<\/strong> service runs alongside the application to collect logs and forward them to Kafka.<\/p>\n td-agent is a stable distribution of Fluentd maintained by Treasure Data and the Cloud Native Computing Foundation. It is a data collection daemon that gathers logs from various sources and forwards them to destinations such as Kafka or Elasticsearch.<\/p>\n Use the following configuration inside the td-agent configuration file.<\/p>\nWhy Kafka Is Needed in the EFK Stack<\/h2>\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/10\/screenshot-2022-10-14-at-5.58.12-pm.png?w=1024\")
<\/p>\nHigh-Level Architecture Overview<\/h2>\n
\n
Block 1 Application Logs and td-agent Configuration<\/h2>\n
td-agent Configuration<\/h3>\n
Source Configuration<\/h3>\n
<source><\/span>
\n