We likely know Kafka as a durable, scalable and fault-tolerant publish-subscribe messaging system. Recently I got a requirement to efficiently monitor and manage our Kafka cluster, and I started looking for different solutions. Kafka-manager is an open source tool introduced by Yahoo to manage and monitor the Apache Kafka cluster via UI.
As per their documentation on github below are the major features:
- Manage multiple clusters.
- Easy inspection of the cluster state.
Brokers:
- Run preferred replica election.
- Generate partition assignments with the option to select brokers to use
- Run reassignment of a partition (based on generated assignments)
Topics:
- Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)
- Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)
- The topic list now indicates topics marked for deletion (only supported on 0.8.2+)
- Batch generate partition assignments for multiple topics with the option to select brokers to use
- Batch run reassignment of partition for multiple topics
- Add partitions to an existing topic
- Update config for an existing topic
Metrics:
- Optionally filter out consumers that do not have ids/ owners/ & offsets/ directories in zookeeper.
- Optionally enable JMX polling for broker level and topic level metrics.
Prerequisites of Kafka Manager:
We should have a running Apache Kafka with Apache Zookeeper.
- Apache Zookeeper
- Apache Kafka
Deployment on Kubernetes:
After deployment, we should able to access Kafka manager service via http://:8080
We have two files to Kafka-manager-service.yaml and kafka-manager.yaml to achieve above-mentioned setup. Let’s have a brief description of the different attributes used in these files.
Deployment configuration file:
namespace: provide a namespace to isolate application within Kubernetes.
replicas: number of containers to spun up.
image: provide the path of docker image to be used.
containerPorts: on which port you want to run your application.
environment: “ZK_HOSTS” provide the address of already running zookeeper.
Service configuration file:
This file contains the details to create Kafka manager service ok Kubernetes. For demo purpose, I have used the node port method to expose my service.
As we are using Kubernetes for our underlying platform of deployment it is recommended not to use external IP to access any service. Either we should go with LoadBalancer or use ingress (recommended method) rather than exposing all microservices.
To configure ingress, please take a note from Kubernetes Ingress.
Once we are able to access Kafka manager we can see similar screens.
Cluster Management
Topic List
Major Issues
To resolve this you need to update JMX settings while creating your docker image as given as below.
vim /opt/kafka/bin/kafka-run-class.sh
if [ -z "$KAFKA_JMX_OPTS" ]; then
#KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=$HOSTNAME -Djava.net.preferIPv4Stack=true"
fi