{"id":12726,"date":"2023-01-24T11:53:29","date_gmt":"2023-01-24T06:23:29","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=12726"},"modified":"2023-08-21T11:25:58","modified_gmt":"2023-08-21T05:55:58","slug":"protected-efk-stack-setup-for-kubernetes","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2023\/01\/24\/protected-efk-stack-setup-for-kubernetes\/","title":{"rendered":"Protected EFK Stack Setup for Kubernetes"},"content":{"rendered":"\n
\n\n\n\n

In this blog, we will see how we can deploy the Elasticsearch, Fluent-bit, and Kibana (EFK) stack on Kubernetes. EFK stack\u2019s prime objective is to reliably and securely retrieve data from the K8s cluster in any format, as well as to facilitate anytime searching, analyzing, and visualizing of the data.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

What is EFK Stack?<\/h3>\n\n\n\n
\n

EFK stands for Elasticsearch, Fluent bit, and Kibana<\/strong>.<\/p>\n<\/blockquote>\n\n\n\n

Elasticsearch<\/strong> is a scalable and distributed search engine that is commonly used to store large amounts of log data. It is a NoSQL database. Its primary function is to store and retrieve logs from fluent bit.<\/p>\n\n\n\n

Fluent Bit<\/strong> is a logging and metrics processor and forwarder that is extremely fast, lightweight, and highly scalable. Because of its performance-oriented design, it is simple to collect events from various sources and ship them to various destinations without complexity.<\/p>\n\n\n\n\n\n\n\n

Kibana <\/strong>is a graphical user interface (GUI) tool for data visualization, querying, and dashboards. It is a query engine that lets you explore your log data through a web interface, create visualizations for event logs, and filter data to detect problems. Kibana is being used to query elasticsearch indexed data.<\/p>\n\n\n\n

Why do we need EFK Stack?<\/h3>\n\n\n\n
    \n
  • Using the EFK stack in your Kubernetes cluster can make it much easier to collect, store, and analyze log data from all the pods and nodes in your cluster, making it more manageable and more accessible for different users.<\/li>\n\n\n\n
  • The kubectl logs command is useful for looking at logs from individual pods, but it can quickly become unwieldy when you have a large number of pods running in your cluster. <\/li>\n\n\n\n
  • With the EFK stack, you can collect logs from all the nodes and pods in your cluster and store them in a central location. It allows you to quickly troubleshoot issues and identify patterns in your log data. <\/li>\n\n\n\n
  • It also enables people who are not familiar with using the command line to check logs and keep track of the Kubernetes cluster and the applications that are deployed on it. <\/li>\n\n\n\n
  • It also allows you to easily create alerts, dashboards, and create monitoring and reporting capabilities that can give you an overview of your system\u2019s health and performance, and It will notify you in real-time if something goes wrong. <\/li>\n<\/ul>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    In this tutorial, we will be deploying EFK components as follows:<\/h6>\n\n\n\n
      \n
    1. Elasticsearch<\/strong> is deployed as statefulset as it stores the log data.<\/li>\n\n\n\n
    2. Kibana<\/strong> is deployed as deployment and connects to elasticsearch service endpoint.<\/li>\n\n\n\n
    3. Fluent-bit<\/strong> is deployed as a daemonset to gather the container logs from every node. It connects to the Elasticsearch service endpoint to forward the logs.<\/li>\n<\/ol>\n\n\n\n
      So let\u2019s get started with EFK stack deployment.<\/strong><\/h6>\n\n\n\n

      Creating a Namespace<\/h3>\n\n\n\n

      It\u2019s a good practice to create a separate namespace for every functional unit in Kubernetes. By creating a namespace for each functional unit, you can easily see which resources belong to which unit and manage them accordingly.<\/p>\n\n\n\n

      let’s create a namespace<\/p>\n\n\n\n

      \n