Cross Cluster Replication (CCR) is a powerful feature in Elasticsearch that facilitates seamless data migration and disaster recovery by allowing real-time replication of data from a remote source cluster to a target cluster. In this guide, we’ll cover setting up CCR to migrate data from a Remote Cluster in Singapore to a Local Cluster in Mumbai. This step-by-step process involves configuring both clusters, creating follower indices, and monitoring the replication status. By the end, you’ll have a reliable setup for cross-regional data replication, enhancing both data redundancy and accessibility across geographical locations.

Pre-requisites

1. 1. Should have “manage” cluster privilege on the local cluster 
   2. A license on both the cluster that includes cross-cluster replication. 
   3. In local cluster, all master nodes should also have a remote-cluster-client role. It must also have one data role and remote-cluster-client role.

Steps:

1. 1. Establish trust between your clusters. This can be done in two ways: API key or TLS certificate authentication.
   2. To setup API key authentication you may want to Enable the remote cluster server on every node of the remote cluster. In elasticsearch.yml specify the domain of your remote cluster (singapore)
   3. Generate a certificate authority (CA) and a wildcard server certificate/key pair that would work on all nodes of your cluster. Mention the .p12 certification in your elasticsearch.yml.
   4. Make sure to add the SSL keystore password to the Elasticsearch keystore to keep your password secure. 
   5. Restart the cluster. 
   6. For additional instructions, please refer to this document: API key authentication
   7. You can check your cluster connection from Stack Management > Data > Remote Clusters
   8. Configure privileges by creating separate roles on the local and remote clusters, and then create a user with the required roles.
   9. Finally create a follower index in your local cluster to replicate data from a selected index in the remote cluster.
   10. Elasticsearch initializes the follower by transferring Lucene segment files from the leader index to follower, setting the index status to Paused, then switches to Active once replication is done.
   11. Even if you add more data to the leader index, the follower will continue to replicate these updates as well.

Additional Considerations for CCR

• • Operational Continuity:It is not required to bring down connections or stop reads and writes in the Singapore cluster to set up Cross-Cluster Replication (CCR). You can perform CCR while the Singapore cluster continues to handle multiple reads and writes.
  • Replication Time: Replicating 1 GB of data typically takes around 1 minute from remote to local cluster.
  • Ongoing Replication: CCR continuously replicates data from the remote to the local cluster. As long as both clusters are operational, there should be no downtime. However, if there’s a network disruption or if either cluster experiences an outage, replication might temporarily pause until connectivity is restored.
  • Downtime during failover: If you have pre-configured your Mumbai cluster to handle writes as soon as a failover is triggered, then the downtime can be minimized or even eliminated entirely. 

Versions and Instance Details:

1. 1. Elasticsearch version 8.15.08
   2. Ubuntu 22.04 (4 CPU cores, 8GB RAM, 160gb storage for each node) 

Conclusion

By setting up CCR from the Singapore cluster to the Mumbai cluster, we ensure that data remains consistent and accessible, regardless of geographical constraints. This approach not only enhances disaster recovery capabilities but also reduces latency for region-specific data access. With Elasticsearch’s flexible configuration options and secure authentication methods, CCR is an ideal choice for businesses looking to scale and maintain high data resilience. Reference Links:

• CCR setup guide 
• Managing Cross Cluster replication
• API key authentication
• TLS certificate authentication

Blog Pundit: Sandeep Rawat

Connect With Us