{"id":1748,"date":"2019-10-29T13:41:03","date_gmt":"2019-10-29T08:11:03","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=1748"},"modified":"2020-02-24T17:19:12","modified_gmt":"2020-02-24T11:49:12","slug":"redis-cluster-setup-sharding-and-failover-testing","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2019\/10\/29\/redis-cluster-setup-sharding-and-failover-testing\/","title":{"rendered":"Redis Cluster: Setup, Sharding and Failover Testing"},"content":{"rendered":"\r\n<figure class=\"wp-block-embed aligncenter is-type-rich\">\r\n<div class=\"wp-block-embed__wrapper\">https:\/\/blog.advids.co\/wp-content\/uploads\/\/2017\/09\/Industrial-robotics1.gif<\/div>\r\n<\/figure>\r\n\r\n\r\n\r\n<p>Watching cluster sharding and failover management is as gripping as visualizing a robotic machinery work.<\/p>\r\n\r\n\r\n\r\n<p>My last blog on <a href=\"https:\/\/opstree.com\/blog\/\/2019\/10\/22\/redis-cluster-architecture-replication-sharding-and-failover\/\">Redis Cluster<\/a> was primarily focussed on its related concepts and requirements. I would highly recommend to go through the concepts first to have better understanding.<\/p>\r\n\r\n\r\n\r\n<p>Here, I will straight forward move to its setup along with the behaviour of cluster when I intentionally turned down one Redis service on one of the node.<br \/>Let&#8217;s start from the scratch.<\/p>\r\n<p><!--more--><\/p>\r\n\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Redis Setup<\/h2>\r\n\r\n\r\n\r\n<p>Here, I will follow the approach of a 3-node Redis Cluster with Redis v5.0 on all the three CentOS 7.x nodes.<\/p>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\"><strong>Setup Epel Repo<\/strong><\/h3>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">wget <a href=\"http:\/\/dl.fedoraproject.org\/pub\/epel\/epel-release-latest-7.noarch.rpm\" target=\"_blank\" rel=\"noopener\">http:\/\/dl.fedoraproject.org\/pub\/epel\/epel-release-latest-7.noarch.rpm<\/a><br \/>rpm -ivh epel-release-latest-7.noarch.rpm<\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\"><strong>Setup Remi Repo<\/strong><\/h3>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">yum install <a href=\"http:\/\/rpms.remirepo.net\/enterprise\/remi-release-7.rpm\" target=\"_blank\" rel=\"noopener\">http:\/\/rpms.remirepo.net\/enterprise\/remi-release-7.rpm<\/a><br \/>yum --enablerepo=remi install redis<br \/><br \/>redis-server --version<br \/>Redis server v=5.0.5 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=619d60bfb0a92c36<\/pre>\r\n\r\n\r\n\r\n<h2 class=\"wp-block-heading\">3-Node Cluster Prerequisites<\/h2>\r\n\r\n\r\n\r\n<p>While setting up Redis cluster on 3 nodes, I will be following the strategy of having 3 master nodes and 3 slave nodes with one master and one slave running on each node serving redis at different ports. As shown in the diagram Redis service is running on Port 7000 and Port 7001<\/p>\r\n\r\n\r\n\r\n<div class=\"wp-block-image\">\r\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1764\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/10\/screenshot-2019-10-28-at-11.29.50-pm.png?w=1024\" alt=\"\" width=\"445\" height=\"276\" \/><\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n<ul>\r\n<li>7000 port will serve Redis Master<\/li>\r\n<li>7001 port will serve Redis Slave<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Directory Structure<\/h3>\r\n\r\n\r\n\r\n<p>We need to design the directory structure to server both redis configurations.<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">tree \/etc\/redis<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">\/etc\/redis\r\n`-- cluster\r\n\u00a0\u00a0\u00a0\u00a0|-- 7000\r\n\u00a0\u00a0\u00a0\u00a0| \u00a0 `-- redis_7000.conf\r\n\u00a0\u00a0\u00a0\u00a0`-- 7001\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0`-- redis_7001.conf<\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Redis Configuration<\/h3>\r\n\r\n\r\n\r\n<p>Configuration file for Redis service 1<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">cat \/etc\/redis\/cluster\/7000\/redis_7000.conf<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">port 7000\r\ndir \/var\/lib\/redis\/7000\/\r\nappendonly yes\r\nprotected-mode no\r\ncluster-enabled yes\r\ncluster-node-timeout 5000\r\ncluster-config-file \/etc\/redis\/cluster\/7000\/nodes_7000.conf\r\npidfile \/var\/run\/redis_7000.pid<\/pre>\r\n\r\n\r\n\r\n<p>Configuration file for Redis service 2<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">cat \/etc\/redis\/cluster\/7001\/redis_7001.conf<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">port 7001\r\ndir \/var\/lib\/redis\/7001\r\nappendonly yes\r\nprotected-mode no\r\ncluster-enabled yes\r\ncluster-node-timeout 5000\r\ncluster-config-file \/etc\/redis\/cluster\/7001\/nodes_7001.conf\r\npidfile \/var\/run\/redis_7001.pid<\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Redis Service File<\/h3>\r\n\r\n\r\n\r\n<p>As we are managing multiple service on a single instance, we need to update service file for easier management of redis services.<\/p>\r\n\r\n\r\n\r\n<p>Service management file for Redis service 1<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">cat \/etc\/systemd\/system\/redis_7000.service<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">[Unit]\r\nDescription=Redis persistent key-value database\r\nAfter=network.target\r\n\r\n[Service]\r\nExecStart=\/usr\/bin\/redis-server \/etc\/redis\/cluster\/7000\/redis_7000.conf --supervised systemd\r\nExecStop=\/bin\/redis-cli -h 127.0.0.1 -p 7000 shutdown\r\nType=notify\r\nUser=redis\r\nGroup=redis\r\nRuntimeDirectory=redis\r\nRuntimeDirectoryMode=0755\r\nLimitNOFILE=65535\r\n\r\n[Install]\r\nWantedBy=multi-user.target<\/pre>\r\n\r\n\r\n\r\n<p>Service management file for Redis service 2<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">cat \/etc\/systemd\/system\/redis_7001.service<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">[Unit]\r\nDescription=Redis persistent key-value database\r\nAfter=network.target\r\n\r\n[Service]\r\nExecStart=\/usr\/bin\/redis-server \/etc\/redis\/cluster\/7001\/redis_7001.conf --supervised systemd\r\nExecStop=\/bin\/redis-cli -h 127.0.0.1 -p 7001 shutdown\r\nType=notify\r\nUser=redis\r\nGroup=root\r\nRuntimeDirectory=\/etc\/redis\/cluster\/7001\r\nRuntimeDirectoryMode=0755\r\nLimitNOFILE=65535\r\n\r\n[Install]\r\nWantedBy=multi-user.target<\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Redis Service Status<\/h3>\r\n\r\n\r\n\r\n<h5 class=\"wp-block-heading\"><strong>Master Service <\/strong><\/h5>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">systemctl status redis_7000.service <\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">\u25cf redis_7000.service - Redis persistent key-value database\r\n\u00a0\u00a0\u00a0Loaded: loaded (\/etc\/systemd\/system\/redis_7000.service; enabled; vendor preset: disabled)\r\n\u00a0\u00a0\u00a0Active: active (running) since Wed 2019-09-25 08:14:15 UTC; 30min ago\r\n\u00a0\u00a0Process: 2902 ExecStop=\/bin\/redis-cli -h 127.0.0.1 -p 7000 shutdown (code=exited, status=0\/SUCCESS)\r\n<strong>\u00a0Main PID: 2917 (redis-server)<\/strong>\r\n<strong>\u00a0\u00a0\u00a0CGroup: \/system.slice\/redis_7000.service<\/strong>\r\n<strong>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2514\u25002917 \/usr\/bin\/redis-server *:7000 [cluster]<\/strong>\r\nsystemd[1]: Starting Redis persistent key-value database...\r\nredis-server[2917]: 2917:C 25 Sep 2019 08:14:15.752 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\r\nredis-server[2917]: 2917:C 25 Sep 2019 08:14:15.752 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=2917, just started\r\nredis-server[2917]: 2917:C 25 Sep 2019 08:14:15.752 # Configuration loaded\r\nredis-server[2917]: 2917:C 25 Sep 2019 08:14:15.752 * supervised by systemd, will signal readiness\r\nsystemd[1]: Started Redis persistent key-value database.\r\nredis-server[2917]: 2917:M 25 Sep 2019 08:14:15.754 * No cluster configuration found, I'm ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2\r\nredis-server[2917]: 2917:M 25 Sep 2019 08:14:15.756 * <strong>Running mode=cluster, port=7000<\/strong>.\r\nredis-server[2917]: 2917:M 25 Sep 2019 08:14:15.756 # Server initialized\r\nredis-server[2917]: 2917:M 25 Sep 2019 08:14:15.756 *<strong> Ready to accept connections<\/strong><\/pre>\r\n\r\n\r\n\r\n<h5 class=\"wp-block-heading\">Slave Service<\/h5>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">systemctl status redis_7001.service<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">\u25cf redis_7001.service - Redis persistent key-value database\r\n\u00a0\u00a0\u00a0Loaded: loaded (\/etc\/systemd\/system\/redis_7001.service; enabled; vendor preset: disabled)\r\n\u00a0\u00a0\u00a0Active: active (running) since Wed 2019-09-25 08:14:15 UTC; 30min ago\r\n\u00a0\u00a0Process: 2902 ExecStop=\/bin\/redis-cli -h 127.0.0.1 -p 7001 shutdown (code=exited, status=0\/SUCCESS)\r\n<strong>\u00a0Main PID: 2919 (redis-server)<\/strong>\r\n<strong>\u00a0\u00a0\u00a0CGroup: \/system.slice\/redis_7001.service<\/strong>\r\n<strong>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2514\u25002919 \/usr\/bin\/redis-server *:7001 [cluster]<\/strong>\r\nsystemd[1]: Starting Redis persistent key-value database...\r\nredis-server[2919]: 2917:C 25 Sep 2019 08:14:15.752 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo\r\nredis-server[2919]: 2917:C 25 Sep 2019 08:14:15.752 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=2917, just started\r\nredis-server[2919]: 2917:C 25 Sep 2019 08:14:15.752 # Configuration loaded\r\nredis-server[2919]: 2917:C 25 Sep 2019 08:14:15.752 * supervised by systemd, will signal readiness\r\nsystemd[1]: Started Redis persistent key-value database.\r\nredis-server[2919]: 2917:M 25 Sep 2019 08:14:15.754 * No cluster configuration found, I'm ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2\r\nredis-server[2919]: 2917:M 25 Sep 2019 08:14:15.756 * <strong>Running mode=cluster, port=700<\/strong>1.\r\nredis-server[2919]: 2917:M 25 Sep 2019 08:14:15.756 # Server initialized\r\nredis-server[2919]: 2917:M 25 Sep 2019 08:14:15.756 *<strong> Ready to accept connections<\/strong><\/pre>\r\n\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Redis Cluster Setup<\/h2>\r\n\r\n\r\n\r\n<p>Redis itself provides cli tool to setup cluster. <br \/>In the current 3 node scenario, I opt 7000 port on all node to serve Redis master and 7001 port to serve Redis slave.<\/p>\r\n\r\n\r\n\r\n<div class=\"wp-block-image\">\r\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1765\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/10\/screenshot-2019-10-28-at-11.29.21-pm.png?w=1024\" alt=\"\" width=\"435\" height=\"270\" \/><\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>redis-cli --cluster create 172.19.33.7:7000 172.19.42.44:7000 172.19.45.201:7000 172.19.33.7:7001 172.19.42.44:7001 172.19.45.201:7001 --cluster-replicas 1<\/strong><\/pre>\r\n\r\n\r\n\r\n<p>The first 3 address will be the master and the next 3 address will be the slaves. It will be a cross node replication, say, Slave of any Mater will reside on a different node and the <em>cluster-replicas<\/em> define the replication factor, i.e each master will have 1 slave.<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">&gt;&gt;&gt; Performing hash slots allocation on 6 nodes...\r\nMaster[0] -&gt; Slots 0 - 5460\r\nMaster[1] -&gt; Slots 5461 - 10922\r\nMaster[2] -&gt; Slots 10923 - 16383\r\nAdding replica 172.19.42.44:7001 to 172.19.33.7:7000\r\nAdding replica 172.19.45.201:7001 to 172.19.42.44:7000\r\nAdding replica 172.19.33.7:7001 to 172.19.45.201:7000\r\n<strong>M: ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 172.19.33.7:7000<\/strong>\r\n<strong>\u00a0\u00a0\u00a0slots:[0-5460] (5461 slots) master<\/strong>\r\n<strong>M: 314038a48bda3224bad21c3357dbff8305735d72 172.19.42.44:7000<\/strong>\r\n<strong>\u00a0\u00a0\u00a0slots:[5461-10922] (5462 slots) master<\/strong>\r\n<strong>M: 19a2c81b7f489bec35eed474ae8e1ad787327db6 172.19.45.201:7000<\/strong>\r\n<strong>\u00a0\u00a0\u00a0slots:[10923-16383] (5461 slots) master<\/strong>\r\n<strong>S: 896b2a7195455787b5d8a50966f1034c269c0259 172.19.33.7:7001<\/strong>\r\n<strong>\u00a0\u00a0\u00a0replicates 19a2c81b7f489bec35eed474ae8e1ad787327db6<\/strong>\r\n<strong>S: 89206df4f41465bce81f44e25e5fdfa8566424b8 172.19.42.44:7001<\/strong>\r\n<strong>\u00a0\u00a0\u00a0replicates ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2<\/strong>\r\n<strong>S: 20ab4b30f3d6d25045909c6c33ab70feb635061c 172.19.45.201:7001<\/strong>\r\n<strong>\u00a0\u00a0\u00a0replicates 314038a48bda3224bad21c3357dbff8305735d72<\/strong>\r\n<strong>Can I set the above configuration? (type 'yes' to accept):<\/strong><\/pre>\r\n\r\n\r\n\r\n<p>A dry run will showcase the cluster setup and ask for confirmation.<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>Can I set the above configuration? (type 'yes' to accept): yes\r\n<\/strong>&gt;&gt;&gt; Nodes configuration updated\r\n&gt;&gt;&gt; Assign a different config epoch to each node\r\n&gt;&gt;&gt; Sending CLUSTER MEET messages to join the cluster\r\nWaiting for the cluster to join\r\n..\r\n&gt;&gt;&gt; Performing Cluster Check (using node 172.19.33.7:7000)\r\nM: ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 172.19.33.7:7000\r\n\u00a0\u00a0\u00a0slots:[0-5460] (5461 slots) master\r\n\u00a0\u00a0\u00a01 additional replica(s)\r\nS: 20ab4b30f3d6d25045909c6c33ab70feb635061c 172.19.45.201:7001\r\n\u00a0\u00a0\u00a0slots: (0 slots) slave\r\n\u00a0\u00a0\u00a0replicates 314038a48bda3224bad21c3357dbff8305735d72\r\nM: 314038a48bda3224bad21c3357dbff8305735d72 172.19.42.44:7000\r\n\u00a0\u00a0\u00a0slots:[5461-10922] (5462 slots) master\r\n\u00a0\u00a0\u00a01 additional replica(s)\r\nM: 19a2c81b7f489bec35eed474ae8e1ad787327db6 172.19.45.201:7000\r\n\u00a0\u00a0\u00a0slots:[10923-16383] (5461 slots) master\r\n\u00a0\u00a0\u00a01 additional replica(s)\r\nS: 89206df4f41465bce81f44e25e5fdfa8566424b8 172.19.42.44:7001\r\n\u00a0\u00a0\u00a0slots: (0 slots) slave\r\n\u00a0\u00a0\u00a0replicates ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2\r\nS: 896b2a7195455787b5d8a50966f1034c269c0259 172.19.33.7:7001\r\n\u00a0\u00a0\u00a0slots: (0 slots) slave\r\n\u00a0\u00a0\u00a0replicates 19a2c81b7f489bec35eed474ae8e1ad787327db6\r\n<strong>[OK] All nodes agree about slots configuration.\r\n&gt;&gt;&gt; Check for open slots...\r\n&gt;&gt;&gt; Check slots coverage...\r\n[OK] All 16384 slots covered.<\/strong><\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Check Cluster Status<\/h3>\r\n\r\n\r\n\r\n<p>Connect to any of the cluster node to check the status of cluster.<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">redis-cli -c -h 172.19.33.7 -p 7000<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>172.19.33.7:7000&gt; cluster nodes\r\n<span style=\"text-decoration:underline;\">20ab4b30f3d6d25045909c6c33ab70feb635061c 172.19.45.201:7001@17001 slave 314038a48bda3224bad21c3357dbff8305735d72 0 1569402961000 6 connected\r\n314038a48bda3224bad21c3357dbff8305735d72 172.19.42.44:7000@17000 master - 0 1569402961543 2 connected 5461-10922<\/span><\/strong>\r\n19a2c81b7f489bec35eed474ae8e1ad787327db6 <strong>172.19.45.201:7000<\/strong>@17000 <strong>master<\/strong> - 0 1569402960538 3 <strong>connected<\/strong> 10923-16383\r\nff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 <strong>172.19.33.7:7000<\/strong>@17000 <strong>myself,master<\/strong> - 0 1569402959000 1 <strong>connected<\/strong> 0-5460\r\n89206df4f41465bce81f44e25e5fdfa8566424b8 <strong>172.19.42.44:7001<\/strong>@17001 <strong>slave<\/strong> ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 0 1569402960000 5 <strong>connected<\/strong>\r\n896b2a7195455787b5d8a50966f1034c269c0259 <strong>172.19.33.7:7001<\/strong>@17001 <strong>slave<\/strong> 19a2c81b7f489bec35eed474ae8e1ad787327db6 0 1569402959936 4 <strong>connected<\/strong><\/pre>\r\n\r\n\r\n\r\n<p>Redis cluster itself manages the cross node replication, as seen in the above screen, <strong>172.19.42.44:7000<\/strong> master is associated with <strong>172.19.45.201:7001<\/strong> slave.<\/p>\r\n\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Data Sharding<\/h2>\r\n\r\n\r\n\r\n<p>There are 16384 slots. These slots are divided by the number of servers.<br \/>If there are 3 servers; 1, 2 and 3 then<\/p>\r\n\r\n\r\n\r\n<ul>\r\n<li>Server 1 contains hash slots from 0 to 5500.<\/li>\r\n<li>Server 2 contains hash slots from 5501 to 11000.<\/li>\r\n<li>Server 3 contains hash slots from 11001 to 16383.<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">redis-cli -c -h 172.19.33.7 -p 7000<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">172.19.33.7:7000&gt; set a 1\r\n-&gt; Redirected to slot [15495] located at 172.19.45.201:7000\r\nOK\r\n172.19.45.201:7000&gt; set b 2\r\n-&gt; Redirected to slot [3300] located at 172.19.33.7:7000\r\nOK\r\n172.19.33.7:7000&gt; set c 3\r\n-&gt; Redirected to slot [7365] located at 172.19.42.44:7000\r\nOK\r\n172.19.42.44:7000&gt; set d 4\r\n-&gt; Redirected to slot [11298] located at 172.19.45.201:7000\r\nOK\r\n172.19.45.201:7000&gt; get b\r\n-&gt; Redirected to slot [3300] located at 172.19.33.7:7000\r\n\"2\"\r\n172.19.33.7:7000&gt; get a\r\n-&gt; Redirected to slot [15495] located at 172.19.45.201:7000\r\n\"1\"\r\n172.19.45.201:7000&gt; get c\r\n-&gt; Redirected to slot [7365] located at 172.19.42.44:7000\r\n\"3\"\r\n172.19.42.44:7000&gt; get d\r\n-&gt; Redirected to slot [11298] located at 172.19.45.201:7000\r\n\"4\"\r\n172.19.45.201:7000&gt;<\/pre>\r\n\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Redis Cluster Failover<\/h2>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Stop Master Service<\/h3>\r\n\r\n\r\n\r\n<p>Let&#8217;s stop the Redis master service on Server 3.<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">systemctl stop redis_7000.service\r\nsystemctl status redis_7000.service<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">\u25cf redis_7000.service - Redis persistent key-value database\r\n\u00a0\u00a0\u00a0Loaded: loaded (\/etc\/systemd\/system\/redis_7000.service; enabled; vendor preset: disabled)\r\n\u00a0\u00a0\u00a0<strong>Active: inactive (dead)<\/strong> since Wed 2019-09-25 09:32:37 UTC; 23s ago\r\n\u00a0\u00a0Process: 3232 ExecStop=\/bin\/redis-cli -h 127.0.0.1 -p 7000 shutdown (code=exited, status=0\/SUCCESS)\r\n\u00a0\u00a0Process: 2892 ExecStart=\/usr\/bin\/redis-server \/etc\/redis\/cluster\/7000\/redis_7000.conf --supervised systemd (code=exited, status=0\/SUCCESS)\r\n\u00a0Main PID: 2892 (code=exited, status=0\/SUCCESS)<\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Cluster State (Failover)<\/h3>\r\n\r\n\r\n\r\n<p>While checking the cluster status, Redis master service running on server 3 at port 7000 is shown fail and disconnected.<\/p>\r\n\r\n\r\n\r\n<div class=\"wp-block-image\">\r\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1767\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/10\/screenshot-2019-10-28-at-11.35.03-pm.png?w=1024\" alt=\"\" width=\"422\" height=\"262\" \/><\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n<p>At the same moment its respective slave gets promoted to master which is running on port 7001 on server 1.<\/p>\r\n\r\n\r\n\r\n<div class=\"wp-block-image\">\r\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1776\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/10\/screenshot-2019-10-28-at-11.34.19-pm-1.png?w=1024\" alt=\"\" width=\"475\" height=\"253\" \/><\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">redis-cli -c -h 172.19.33.7 -p 7000<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>172.19.45.201:7000&gt; CLUSTER NODES<\/strong>\r\n<strong><span style=\"text-decoration:underline;\"><em>314038a48bda3224bad21c3357dbff8305735d72 172.19.42.44:7000@17000 master,fail - 1569403957138 1569403956000 2 disconnected<\/em><\/span><\/strong>\r\nff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 172.19.33.7:7000@17000 <strong>master<\/strong> - 0 1569404037252 1 <strong>connected<\/strong> 0-5460\r\n896b2a7195455787b5d8a50966f1034c269c0259 172.19.33.7:7001@17001 <strong>slave<\/strong> 19a2c81b7f489bec35eed474ae8e1ad787327db6 0 1569404036248 4 <strong>connected<\/strong>\r\n89206df4f41465bce81f44e25e5fdfa8566424b8 172.19.42.44:7001@17001 <strong>slave<\/strong> ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 0 1569404036752 5 <strong>connected<\/strong>\r\n<strong><em><span style=\"text-decoration:underline;\">20ab4b30f3d6d25045909c6c33ab70feb635061c 172.19.45.201:7001@17001 master - 0 1569404036000 7 connected 5461-10922<\/span><\/em><\/strong>\r\n19a2c81b7f489bec35eed474ae8e1ad787327db6 172.19.45.201:7000@17000 <strong>myself,master<\/strong> - 0 1569404035000 3 <strong>connected<\/strong> 10923-16383<\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Restarting Stopped Redis<\/h3>\r\n\r\n\r\n\r\n<p>Now we will check the behaviour of cluster once we fix or restart the redis service that we intentionally turned down earlier.<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">systemctl start redis_7000.service\r\nsystemctl status redis_7000.service<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">\u25cf redis_7000.service - Redis persistent key-value database\r\n\u00a0\u00a0\u00a0Loaded: loaded (\/etc\/systemd\/system\/redis_7000.service; enabled; vendor preset: disabled)\r\n\u00a0\u00a0\u00a0<strong>Active: active (running)<\/strong> since Wed 2019-09-25 09:35:12 UTC; 8s ago\r\n\u00a0\u00a0Process: 3232 ExecStop=\/bin\/redis-cli -h 127.0.0.1 -p 7000 shutdown (code=exited, status=0\/SUCCESS)\r\n\u00a0Main PID: 3241 (redis-server)\r\n\u00a0\u00a0\u00a0CGroup: \/system.slice\/redis_7000.service\r\n<strong>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u2514\u25003241 \/usr\/bin\/redis-server *:7000 [cluster]<\/strong><\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Cluster State (Recovery)<\/h3>\r\n\r\n\r\n\r\n<p>Finally, all redis service are back in running state. The master service that we turned down and restarted has now become slave to its promoted master.<\/p>\r\n\r\n\r\n\r\n<div class=\"wp-block-image\">\r\n<figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1778\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2019\/10\/screenshot-2019-10-29-at-11.55.44-am.png?w=1024\" alt=\"\" width=\"481\" height=\"255\" \/><\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\">redis-cli -c -h 172.19.33.7 -p 7000<\/pre>\r\n\r\n\r\n\r\n<pre class=\"wp-block-preformatted\"><strong>172.19.45.201:7000&gt; CLUSTER NODES<\/strong> <strong><em><span style=\"text-decoration:underline;\">314038a48bda3224bad21c3357dbff8305735d72 172.19.42.44:7000@17000 slave 20ab4b30f3d6d25045909c6c33ab70feb635061c 0 1569404162565 7 connected<\/span><\/em><\/strong> ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 172.19.33.7:7000@17000 <strong>master<\/strong> - 0 1569404162000 1 <strong>connected<\/strong> 0-5460 896b2a7195455787b5d8a50966f1034c269c0259 172.19.33.7:7001@17001 <strong>slave<\/strong> 19a2c81b7f489bec35eed474ae8e1ad787327db6 0 1569404163567 4 <strong>connected<\/strong> 89206df4f41465bce81f44e25e5fdfa8566424b8 172.19.42.44:7001@17001 <strong>slave<\/strong> ff3e4300bec02ed4bd1be9af5d83a5b44249c2b2 0 1569404163000 5 <strong>connected<\/strong> <strong><em><span style=\"text-decoration:underline;\">20ab4b30f3d6d25045909c6c33ab70feb635061c 172.19.45.201:7001@17001 master - 0 1569404162000 7 connected 5461-10922 <\/span><\/em><\/strong>19a2c81b7f489bec35eed474ae8e1ad787327db6 172.19.45.201:7000@17000 <strong>myself,master<\/strong> - 0 1569404161000 3 <strong>connected<\/strong> 10923-16383<\/pre>\r\n\r\n\r\n\r\n<p>It&#8217;s not done yet, further we can explore around having a single endpoint to point from the application. I will am currently working on that and soon will come up with the solution. <br \/>Apart from this monitoring the Redis Cluster will also be a major aspect to look forward. <br \/>Till then get your hands dirty playing around the Redis Cluster setup and failover.<\/p>\r\n\r\n\r\n\r\n<p><strong>Reference links:<\/strong> <br \/>Image: Google image search (blog.advids.co)<\/p>\r\n\r\n\r\n\r\n<p>&nbsp;<\/p>\r\n","protected":false},"excerpt":{"rendered":"<p>Watching cluster sharding and failover management is as gripping as visualizing a robotic machinery work. My last blog on Redis Cluster was primarily focussed on its related concepts and requirements. I would highly recommend to go through the concepts first to have better understanding. Here, I will straight forward move to its setup along with &hellip; <a href=\"https:\/\/opstree.com\/blog\/2019\/10\/29\/redis-cluster-setup-sharding-and-failover-testing\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Redis Cluster: Setup, Sharding and Failover Testing&#8221;<\/span><\/a><\/p>\n","protected":false},"author":159458173,"featured_media":29900,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[28070474,5307557],"tags":[768739308,768739306,688976814,688729979,688976819,688976817],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/11\/DevSecOps-1.jpg","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-sc","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/1748"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/159458173"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=1748"}],"version-history":[{"count":24,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/1748\/revisions"}],"predecessor-version":[{"id":2375,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/1748\/revisions\/2375"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/29900"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=1748"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=1748"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=1748"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}