Tag: Kubernetes

Opstree’s Logging (EFK) Operator

Logging is a critical part of monitoring and there are a lot of tools for logs monitoring like Splunk, Sumologic, and Elasticsearch, etc. Since Kubernetes is becoming so much popular now, and running multiple applications and services on a Kubernetes cluster requires a centralized, cluster-level stack to analyze the logs created by pods.
One of the well-liked centralized logging solutions is the combination of multiple opensource tools i.e. Elasticsearch, Fluentd, and Kibana. In this blog, we will talk about setting up the logging stack on the Kubernetes cluster with our newly developed operator named “Logging Operator”.

Continue reading “Opstree’s Logging (EFK) Operator”

Redis Setup on Kubernetes

Redis is a popular and opensource in-memory database that supports multiple data structures like strings, hashes, lists, and sets. But similar to other tools, we can scale standalone redis to a particular extent and not beyond that. That’s why we have a cluster mode setup in which we can scale Redis nodes horizontally and then distribute data among those nodes.

Since Kubernetes is becoming buzz technology and people are using it to manage their applications, databases, and middlewares at a single place. So in this blog, we will see how we can deploy the Redis cluster in production mode in the Kubernetes cluster and test failover.

Continue reading “Redis Setup on Kubernetes”

Kafka Manager On Kubernetes

 

                                kafka-manager-on-kubernetes

 

We likely know Kafka as a durable, scalable and fault-tolerant publish-subscribe messaging system. Recently I got a requirement to efficiently monitor and manage our Kafka cluster, and I started looking for different solutions. Kafka-manager is an open source tool introduced by Yahoo to manage and monitor the Apache Kafka cluster via UI.

Before I share my experience of configuring Kafka manager on Kubernetes, let’s go through its considerable features
 

As per their documentation on github below are the major features: 

Clusters:
 
  • Manage multiple clusters.
  • Easy inspection of the cluster state.

Brokers:

  • Run preferred replica election.
  • Generate partition assignments with the option to select brokers to use
  • Run reassignment of a partition (based on generated assignments)

Topics:

  • Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)
  • Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)
  • The topic list now indicates topics marked for deletion (only supported on 0.8.2+)
  • Batch generate partition assignments for multiple topics with the option to select brokers to use
  • Batch run reassignment of partition for multiple topics
  • Add partitions to an existing topic
  • Update config for an existing topic

Metrics:

  • Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.
  • Optionally enable JMX polling for broker level and topic level metrics.

Prerequisites of Kafka Manager:

We should have a running Apache Kafka with Apache Zookeeper.

 
  • Apache Zookeeper
  • Apache Kafka

Deployment on Kubernetes: 

To deploy Kafka Manager on Kubernetes, we need to create deployment and service file as given below.
 
You can find these sample file at https://github.com/vishant07/kafka-manager

After deployment, we should able to access Kafka manager service via http://:8080

We have two files to Kafka-manager-service.yaml and kafka-manager.yaml to achieve above-mentioned setup. Let’s have a brief description of the different attributes used in these files. 

Deployment configuration file: 

namespace: provide a namespace to isolate application within Kubernetes.
replicas: number of containers to spun up.
image: provide the path of docker image to be used.
containerPorts: on which port you want to run your application.
environment: “ZK_HOSTS” provide the address of already running zookeeper.

Service configuration file:
This file contains the details to create Kafka manager service ok Kubernetes. For demo purpose, I have used the node port method to expose my service. 

As we are using Kubernetes for our underlying platform of deployment it is recommended not to use external IP to access any service. Either we should go with LoadBalancer or use ingress (recommended method) rather than exposing all microservices.  

To configure ingress, please take a note from Kubernetes Ingress.

Once we are able to access Kafka manager we can see similar screens. 
 

Cluster Management

Topic List

 

 
 

Major Issues

 
To get broker level and topic level metrics we have to enable JMX polling.
 
So what we will generally do is to set the environment variable in the kubernetes manifest but somehow it is not working most of the times.

 

To resolve this you need to update JMX settings while creating your docker image as given as below.

vim /opt/kafka/bin/kafka-run-class.sh


if [ -z "$KAFKA_JMX_OPTS" ]; then

#KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false "

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=$HOSTNAME -Djava.net.preferIPv4Stack=true"

fi
 

Conclusion

 
Deploying Kafka manager on Kubernetes encourages the easy setup, provides efficient manageability and all-time availability. Managing Kafka cluster over CLI becomes a tedious task and here Kafka manager helps to focus more on the use of Kafka rather than investing our time to configure and manage it.  It becomes useful at Enterprise Level, where system engineers can manage multiple Kafka clusters easily via UI.
Reference links:
Image: google image search
Documentation: https://github.com/yahoo/kafka-manager