Recently while trying to set up Apache Nifi in cluster mode manually, I faced the challenge of performing same tasks on all nodes manually. In addition configuring the right cluster configurations was not easy. In my last blog here, I have covered the advantages of having nifi cluster over standalone and the manual steps to configure a Nifi cluster with external zookeeper. In this article, I will show you how you can set up a three-node Zookeeper and Nifi cluster with terraform which will minimize the steps we have to perform with manual setup.
Apache NiFi is an open-source data integration and automation tool that enables the automation of data flow between different systems. NiFi provides a user-friendly interface to design, control, and manage the flow of data between various sources and destinations. The tool is particularly useful in handling data from different sources, applying transformations, and routing it to different systems in real-time.
Advantage of Using Terraform for Nifi
Terraform allows you to define your NiFi cluster infrastructure as code, making it easily versioned, shareable, and easy to understand. This ensures that your infrastructure is consistent across different environments. This helps in maintaining consistency and reducing the chances of configuration drift. As your NiFi cluster requirements evolve, Terraform makes it simple to scale your infrastructure up or down by adjusting the configuration.
Setting Up Apache Nifi Cluster with External Zookeeper
While working on Azure, you might come across a requirement that says the resources being deployed should be in accordance with the organization’s policies. Suppose you might want to grant a particular or a set of permissions on the resource group or on the management group so that the owner of it should be restricted like denying deploying of resources by enforcing resource tagging, region enforcement, allowing approved Virtual machines (VM) images, and many more.
In this blog, we will try to resolve these issues by applying Azure policies.
First, let’s get familiar with the azure policy.
The azure policy is a service that has been designed to help you enforce different rules and to act based on the rule’s effect on your Azure resources. You can use it to create, assign and manage policies. Azure policy evaluates your resources for non-compliance with assigned policies and performs the assigned effect.
If we can dynamically change the values of terraform.tfvars.
If we can restrict the regions of deployments.
If we can limit our VM types to maintain better cost optimization.
In this article, we will touch upon these problems and try to resolve them in a way that the same concepts can also be applied to similar requirements.
Soo… Let’s Get Started !!!
First of all, we need to know what is Terraform & Azure DevOps.
Talking About Terraform: HashiCorp Terraform is an infrastructure as a code tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share. You can then use a consistent workflow to provision and manage all of your infrastructures throughout its cycle. Terraform can manage low-level components like compute, storage, and networking resources, as well as high-level components like DNS entries and SaaS features.
Talking about Azure DevOps: Azure DevOps provides developer services for allowing teams to plan work, collaborate on code development, and build and deploy applications. Azure DevOps supports a collaborative culture and set of processes that bring together developers, project managers, and contributors to develop software. It allows organizations to create and improve products at a faster pace than they can with traditional software development approaches.
DevOps lifecycle in Azure DevOps
If you want to learn more about Azure DevOps click here.
Pre-requisites:
No matter whether we are deploying our infrastructure into Azure Cloud Services or Amazon Web Services (AWS). All we need are the following checklist:
Active Cloud Service (Azure/AWS)
Azure DevOps Account
Terraform Code to deploy.
A Linux machine (VM or EC2) for agent pool or Azure Microsoft-hosted agent.
Storage Account (Azure Blob Container or AWS S3)
Terraform code to deploy using terraform.tfvars.
Azure DevOps Pipeline
Let’s take a scenario in which we will deploy a simple terraform code of Azure Virtual Machine using Azure DevOps pipelines.
In these pipeline parameters, we’re also restricting/limiting the range of values by providing a list of values to our parameters. In this way, the user cannot go beyond these pre-defined values while executing the pipeline.
If you want to Learn More about Pipeline click here.
Pipeline Steps:
In our pipeline, ‘we will use the below-mentioned steps
1. Replacing Values
- bash: |
sed -i "s/{vm}/${{ parameters.name }}/g" terraform.tfvars
sed -i "s/{West Europe}/${{ parameters.region }}/g" terraform.tfvars
sed -i "s/{StandardF2}/${{ parameters.vmSize }}/g" terraform.tfvars
sed -i "s/{StandardLRS}/${{ parameters.vmStorageAccountType }}/g" terraform.tfvars
sed -i "s/{16.04-LTS}/${{ parameters.imageSKU }}/g" terraform.tfvars
cat terraform.tfvars
displayName: 'Replace Values'
This is the heart of our pipeline. In this step, we are using the terraform azure pipeline parameters.
This step will execute the configuration file and launch a VM instance. When you run apply command, it will ask you, “Do you want to perform these actions?”, you need to type yes and hit enter. To skip that we have provided our configuration with an “-auto-approve” argument.
Upon saving and running our pipeline we can choose our desired parameters in this way.
We will get a drop-down for each parameter whose value we restricted.
Conclusion
So far we’ve learned how to make the pipeline for our terraform code using Azure DevOps Pipelines. Along with that, we’ve found out how to pass the runtime parameters to dynamically give values to our terraform.tfvars file and also restrict or limit the values as per our requirements.
In this blog, we will create an active-active infrastructure on Microsoft Azure using Terraform and Jenkins.
Prime Reasons to have an active-active set-up of your infrastructure
Disaster Recovery:
Disaster recovery (DR) is an organization’s method of regaining access and functionality to its IT infrastructure after events like a natural disaster, cyber attack, or even business disruptions just like during the COVID-19 pandemic.
Ensure business resilience No matter what happens, a good DR plan can ensure that the business can return to full operations rapidly, without losing data or transactions.
Maintain competitiveness Loyalty is rare and when a business goes offline, customers turn to competitors to get the goods or services they require. A DR plan prevents this.
Avoid data loss The longer a business’s systems are down, the greater the risk that data will be lost. A robust DR plan minimizes this risk.
Maintain reputation A business that has trouble resuming operations after an outage can suffer brand damage. For that reason, a solid DR plan is critical.
Starting the blog with the question – What is Terraform?
It can be called a magic wand that creates Infrastructure on the basis of the code that you write.
In Hashicorp’s words, “Terraform is an open-source Infrastructure as A Code software tool that enables you to safely and predictably create, change, and improve infrastructure.“