{"id":78,"date":"2019-04-23T10:51:00","date_gmt":"2019-04-23T10:51:00","guid":{"rendered":""},"modified":"2020-02-17T15:54:57","modified_gmt":"2020-02-17T10:24:57","slug":"kafka-manager-on-kubernetes","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2019\/04\/23\/kafka-manager-on-kubernetes\/","title":{"rendered":"Kafka Manager On  Kubernetes"},"content":{"rendered":"\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-flow wp-block-group-is-layout-flow\">\n<div>\u00a0<\/div>\n<p>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <img loading=\"lazy\" decoding=\"async\" class=\"alignnone  wp-image-2293\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/kafka-manager-on-kubernetes.png\" alt=\"kafka-manager-on-kubernetes\" width=\"358\" height=\"249\" \/><\/p>\n<div dir=\"ltr\" style=\"text-align:left;\">\n<div style=\"text-align:left;\">\n<p>\u00a0<\/p>\n<p>We likely know Kafka as a durable, scalable and fault-tolerant publish-subscribe messaging system. Recently I got a requirement to efficiently monitor and manage our Kafka cluster, and I started looking for different solutions. Kafka-manager is an open source tool introduced by Yahoo to manage and monitor the Apache Kafka cluster via UI.<\/p>\n<div dir=\"ltr\" style=\"line-height:1.38;margin-bottom:0;margin-top:0;\">\n<div style=\"text-align:left;\"><span style=\"font-size:14.6667px;white-space:pre-wrap;\">Before I share my experience of configuring Kafka manager on Kubernetes, let&#8217;s go through its considerable features<\/span><\/div>\n<div style=\"text-align:left;\">\u00a0<\/div>\n<p>As per their documentation on\u00a0<a href=\"https:\/\/github.com\/yahoo\/kafka-manager\" target=\"_blank\" rel=\"noopener\">github<\/a>\u00a0below are the major features:\u00a0<\/p>\n<div style=\"text-align:left;\"><u>Clusters<\/u>:<\/div>\n<div style=\"text-align:left;\">\u00a0<\/div>\n<ul style=\"text-align:left;\">\n<li>Manage multiple clusters.<\/li>\n<\/ul>\n<ul style=\"text-align:left;\">\n<li>Easy inspection of the cluster state.<\/li>\n<\/ul>\n<p><span style=\"font-size:14.6667px;white-space:pre-wrap;\"><u>Brokers<\/u>: <\/span><\/p>\n<ul style=\"text-align:left;\">\n<li>Run preferred replica election.<\/li>\n<li>Generate partition assignments with the option to select brokers to use<\/li>\n<li>Run reassignment of a partition (based on generated assignments)<\/li>\n<\/ul>\n<p><span style=\"font-size:14.6667px;white-space:pre-wrap;\"><u>Topics<\/u>: <\/span><\/p>\n<ul style=\"text-align:left;\">\n<li>Create a topic with optional topic configs (0.8.1.1 has different configs than 0.8.2+)<\/li>\n<li>Delete topic (only supported on 0.8.2+ and remember set delete.topic.enable=true in broker config)<\/li>\n<li>The topic list now indicates topics marked for deletion (only supported on 0.8.2+)<\/li>\n<li>Batch generate partition assignments for multiple topics with the option to select brokers to use<\/li>\n<li>Batch run reassignment of partition for multiple topics<\/li>\n<li>Add partitions to an existing topic<\/li>\n<li>Update config for an existing topic<\/li>\n<\/ul>\n<p><span style=\"font-size:14.6667px;white-space:pre-wrap;\"><u>Metrics<\/u>:<\/span><\/p>\n<ul style=\"text-align:left;\">\n<li>Optionally filter out consumers that do not have ids\/ owners\/ &amp; offsets\/ directories in zookeeper.<\/li>\n<li>Optionally enable JMX polling for broker level and topic level metrics.<\/li>\n<\/ul>\n<\/div>\n<div dir=\"ltr\" style=\"background-color:white;line-height:1.38;margin-bottom:0;margin-top:8pt;\">\n<div>\n<h4 style=\"text-align:left;\"><strong style=\"border:0;font-family:inherit;font-size:inherit;font-stretch:inherit;font-style:inherit;font-variant:inherit;line-height:inherit;margin:0;padding:0;vertical-align:baseline;\">Prerequisites of Kafka Manager:<\/strong><\/h4>\n<p>We should have a running Apache Kafka with Apache Zookeeper.<\/p>\n<\/div>\n<div>\u00a0<\/div>\n<ul>\n<li>Apache Zookeeper<\/li>\n<li>Apache Kafka<\/li>\n<\/ul>\n<div dir=\"ltr\" style=\"line-height:1.38;margin-bottom:0;margin-top:8pt;\">\n<h3 style=\"text-align:left;\"><b>Deployment on Kubernetes:\u00a0<\/b><\/h3>\n<div style=\"text-align:left;\">To deploy Kafka Manager on Kubernetes, we need to create deployment and service file as given below.<\/div>\n<div style=\"text-align:left;\">\u00a0<\/div>\n<div style=\"text-align:left;\">You can find these sample file at\u00a0<i style=\"font-family:arial, helvetica, sans-serif;\"><a href=\"https:\/\/github.com\/vishant07\/kafka-manager\" target=\"_blank\" rel=\"noopener\">https:\/\/github.com\/vishant07\/kafka-manager<\/a><\/i><\/div>\n<\/div>\n<\/div>\n<div class=\"separator\" style=\"clear:both;text-align:center;\"><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><a style=\"margin-left:1em;margin-right:1em;\" href=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/bf40f-screen2bshot2b2019-04-122bat2b12.04.192bpm.png\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/bf40f-screen2bshot2b2019-04-122bat2b12.04.192bpm.png?w=300\" width=\"400\" height=\"206\" border=\"0\" \/><\/a><\/span><\/div>\n<div><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/><\/span><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\">After deployment, we should able to access Kafka manager service via http:\/\/:8080<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/><\/span><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\">We have two files to\u00a0<a id=\"e40e77996248b04a4b058f62638a9a5c-ef9ac0cb22ecd1d6a0abde41deec554a9bd649eb\" class=\"js-navigation-open\" style=\"background-color:white;box-sizing:border-box;color:#0366d6;font-size:14px;text-decoration-line:none;white-space:nowrap;\" title=\"kafka-manager-service.yaml\" href=\"https:\/\/github.com\/vishant07\/kafka-manager\/blob\/master\/kafka-manager-service.yaml\" target=\"_blank\" rel=\"noopener\">Kafka-manager-service.yaml<\/a>\u00a0and\u00a0<a id=\"a7c8673b1caae32ab6e4a50502e94baa-9ab274fbaa801922225d62600aa4bb5fecb49c2a\" class=\"js-navigation-open\" style=\"background-color:white;box-sizing:border-box;color:#0366d6;font-size:14px;text-decoration-line:none;white-space:nowrap;\" title=\"kafka-manager.yaml\" href=\"https:\/\/github.com\/vishant07\/kafka-manager\/blob\/master\/kafka-manager.yaml\" target=\"_blank\" rel=\"noopener\">kafka-manager.yaml<\/a>\u00a0to achieve above-mentioned setup. Let&#8217;s have a brief description of the different attributes used in these files.\u00a0<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/>Deployment configuration file:\u00a0<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/>namespace: provide a namespace to isolate application within Kubernetes.<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\">replicas: number of containers to spun up.<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\">image: provide the path of docker image to be used.<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\">containerPorts: on which port you want to run your application.<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\">environment: &#8220;ZK_HOSTS&#8221; provide the address of already running zookeeper.<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/>Service configuration file:<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\">This file contains the details to create Kafka manager service ok Kubernetes. For demo purpose, I have used the node port method to expose my service.\u00a0<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/>As we are using Kubernetes for our underlying platform of deployment it is recommended not to use external IP to access any service. Either we should go with LoadBalancer or use ingress (recommended method) rather than exposing all microservices.\u00a0\u00a0<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/>To configure ingress, please take a note from\u00a0<a style=\"font-family:arial, helvetica, sans-serif;\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/services-networking\/ingress\/\" target=\"_blank\" rel=\"noopener\">Kubernetes Ingress<\/a>.<\/span><br \/><span id=\"docs-internal-guid-99d83212-7fff-e5d8-fb1b-5287f88be1c8\"><br \/>Once we are able to access Kafka manager we can see similar screens.\u00a0<\/span><\/div>\n<div>\n<div>\u00a0<\/div>\n<h2 id=\"KafkaMonitoring-ClusterManagement\"><strong><span style=\"color:#24292e;\">Cluster Management<\/span><\/strong><\/h2>\n<\/div>\n<div class=\"separator\" style=\"clear:both;text-align:center;\"><a style=\"margin-left:1em;margin-right:1em;\" href=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/4110d-screen2bshot2b2019-04-122bat2b12.17.372bpm.png\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/4110d-screen2bshot2b2019-04-122bat2b12.17.372bpm.png?w=300\" width=\"400\" height=\"271\" border=\"0\" \/><\/a><\/div>\n<div>\n<h2 id=\"KafkaMonitoring-TopicList\"><strong><span style=\"color:#24292e;\">Topic List<\/span><\/strong><\/h2>\n<\/div>\n<div class=\"separator\" style=\"clear:both;text-align:center;\"><a style=\"margin-left:1em;margin-right:1em;\" href=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/25e5b-screen2bshot2b2019-04-122bat2b12.18.142bpm.png\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/25e5b-screen2bshot2b2019-04-122bat2b12.18.142bpm.png?w=300\" width=\"400\" height=\"191\" border=\"0\" \/><\/a><\/div>\n<div>\n<h2 id=\"KafkaMonitoring-TopicView\">\u00a0<\/h2>\n<\/div>\n<div class=\"separator\" style=\"clear:both;text-align:center;\">\u00a0<\/div>\n<div>\u00a0<\/div>\n<div>\n<h3 style=\"background-color:white;border-bottom-color:#cccccc;color:#172b4d;font-size:1.714em;font-weight:normal;line-height:1.166;margin:30px 0 0;padding:0;text-align:left;\"><u><strong>Major Issues<\/strong><\/u><\/h3>\n<\/div>\n<div>\u00a0<\/div>\n<div>To get broker level and topic level metrics we have to enable JMX polling.<\/div>\n<div>\u00a0<\/div>\n<div class=\"separator\" style=\"clear:both;text-align:center;\"><a style=\"margin-left:1em;margin-right:1em;\" href=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/bf005-screen2bshot2b2019-04-122bat2b3.22.372bpm.png\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/04\/bf005-screen2bshot2b2019-04-122bat2b3.22.372bpm.png?w=300\" width=\"640\" height=\"94\" border=\"0\" \/><\/a><\/div>\n<div>So what we will generally do is to set the environment variable in the kubernetes manifest but somehow it is not working most of the times.\n<p>\u00a0<\/p>\n<p>To resolve this you need to update\u00a0JMX settings while creating your docker image as given as below.<\/p>\n<p>vim \/opt\/kafka\/bin\/kafka-run-class.sh<\/p>\n<\/div>\n<div>\n<pre><span class=\"pl-k\" style=\"box-sizing:border-box;color:#d73a49;\">\nif [ -z \"$KAFKA_JMX_OPTS\" ]; then\n<\/span><\/pre>\n<pre><span class=\"pl-k\" style=\"box-sizing:border-box;color:#d73a49;\">#KAFKA_JMX_OPTS=\"-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false \"\n\nKAFKA_JMX_OPTS=\"-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=$HOSTNAME -Djava.net.preferIPv4Stack=true\"\n\nfi\n<\/span><\/pre>\n<\/div>\n<div>\u00a0<\/div>\n<div>\n<h1 id=\"KafkaMonitoring-MajorIssues\" style=\"background-color:white;border-bottom-color:#cccccc;color:#172b4d;font-size:1.714em;font-weight:normal;line-height:1.166;margin:30px 0 0;padding:0;\"><strong><u>Conclusion<\/u><\/strong><\/h1>\n<div>\u00a0<\/div>\n<\/div>\n<div style=\"text-align:left;\">\n<div style=\"text-align:left;\">Deploying Kafka manager on Kubernetes encourages the easy setup, provides efficient manageability and all-time availability. Managing Kafka cluster over CLI becomes a tedious task and here Kafka manager helps to focus more on the use of Kafka rather than investing our time to configure and manage it.\u00a0\u00a0It becomes useful at Enterprise Level, where system engineers can manage multiple Kafka clusters easily via UI.<\/div>\n<div style=\"text-align:left;\"><b>Reference links:<\/b><\/div>\n<div style=\"text-align:left;\">Image: google image search<\/div>\n<div style=\"text-align:left;\">Documentation: https:\/\/github.com\/yahoo\/kafka-manager<\/div>\n<\/div>\n<div>\u00a0<\/div>\n<div>\u00a0<\/div>\n<div>\u00a0<\/div>\n<div>\u00a0<\/div>\n<div>\u00a0<\/div>\n<\/div>\n<\/div>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 We likely know Kafka as a durable, scalable and fault-tolerant publish-subscribe messaging system. Recently I got a requirement to efficiently monitor and manage our Kafka cluster, and I started looking for different solutions. Kafka-manager is an open &hellip; <a href=\"https:\/\/opstree.com\/blog\/2019\/04\/23\/kafka-manager-on-kubernetes\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Kafka Manager On  Kubernetes&#8221;<\/span><\/a><\/p>\n","protected":false},"author":159458069,"featured_media":29900,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[1],"tags":[768739308,207392,528067816,768739309,768739293],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/11\/DevSecOps-1.jpg","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-1g","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/78"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/159458069"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=78"}],"version-history":[{"count":8,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/78\/revisions"}],"predecessor-version":[{"id":2297,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/78\/revisions\/2297"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/29900"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=78"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=78"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=78"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}