![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-2.png?w=1024\")
If you want to learn more about terraform you can click here<\/a>.<\/p>\n\n\n\n Talking about Azure DevOps<\/strong>: Azure DevOps provides developer services for allowing teams to plan work, collaborate on code development, and build and deploy applications. Azure DevOps supports a collaborative culture and set of processes that bring together developers, project managers, and contributors to develop software. It allows organizations to create and improve products at a faster pace than they can with traditional software development approaches.<\/p>\n\n\n\n If you want to learn more about Azure DevOps click here<\/a>.<\/p>\n\n\n\n Okay !!! I know that’s a lot of information. Now let’s get back to the point… the question we all started with, “How the hell are we going to achieve all this and that too in a single pipeline?” Well, the answer is very simple, by using different tools dedicated to a particular task.<\/p>\n\n\n\n This is the broad architecture of the pipeline that we are going to create.<\/p>\n\n\n\n No matter whether we are deploying our infrastructure into Azure Cloud Services or Amazon Web Services (AWS). All we need are the following checklist:<\/p>\n\n\n\n TFsec <\/em><\/strong>is a static analysis security scanner for your Terraform code.<\/p>\n\n\n\n TFsec <\/em><\/strong>takes a developer-first approach to scan your Terraform templates; using static analysis and deep integration with the official HCL parser ensures that security issues can be detected before your infrastructure changes take effect.<\/p>\n\n\n\n TFlint <\/em><\/strong>is a framework and each feature is provided by plugins, the key features are as follows:<\/p>\n\n\n\n Infracost shows cloud cost estimates for Terraform. It lets DevOps, SRE, and engineers see a cost breakdown and understand costs before making changes, either in the terminal or in pull requests. It can also show us the difference between our present state and desired state.<\/p>\n\n\n\n Rest all the requirements we can fulfill within our pipeline itself<\/p>\n\n\n\n Assuming that we have already configured an agent in the agent pool which is going to assist us in executing all the commands to achieve our pipeline goals. Also, try importing your terraform code into your Azure Repos as it’ll benefit you in a very unique way that we’ll find out further in this article in the bonus section.<\/p>\n\n\n\n If you what to know how to configure an agent you can click here<\/a> <\/p>\n\n\n\n You can click here<\/a> to follow the steps to import a repo.<\/p>\n\n\n\n All we need to do in this part is to download and install all the dependencies required in your pipeline.<\/p>\n\n\n\n Here we will use the docker images for different tasks, eg:<\/p>\n\n\n\n Alongside you can try this YAML format of the pipeline.<\/p>\n\n\n\n For more detailed info click here<\/a>.<\/p>\n\n\n\n This is one of the most simple and well-known step of the whole pipeline<\/p>\n\n\n\n In this part, we’ll initialize, validate and plan our terraform code and store the output into a file using For more detailed steps click here<\/a>.<\/p>\n\n\n\n Now using the above-mentioned tools we can achieve these tasks Terraform Security Compliance, Linting, Cost Estimation & Cost Difference<\/strong> in a bash task. <\/p>\n\n\n\n Though you don’t need custom settings while linting or cost calculating but you can definitely use a custom checks file for Terraform Compliance step.<\/p>\n\n\n\n A custom check file of Tfsec will look like this:<\/p>\n\n\n\n YAML Pipeline for the task:<\/p>\n\n\n\n Here we also need to generate an API Key for the Infracost app, to learn how to do it you can click here<\/a>.<\/p>\n\n\n\n The need for this step is to store our logs generated in previous steps into a storage account and make sure that we are not going to lose them.<\/p>\n\n\n\n For a detailed description of this part, you can click here<\/a>.<\/p>\n\n\n\n In this step, we’ll generate two artifacts, one named Release which will trigger the Release pipeline, and the other one named Repot<\/em><\/strong> which will publish the reports for our output files of Compliance, Cost Estimation & Cost Difference.<\/p>\n\n\n\n For a detailed description of this part, you can click here<\/a>.<\/p>\n\n\n\n Our Build Step will generate an artifact named Release<\/em><\/strong> which will contain all the terraform files required to apply our desired configuration. Also after all the checks and validations, we’ve found out that there is no error in our code and it is exactly what we have desired, so we will allow the Continuous Deployment for our Release Pipeline.<\/p>\n\n\n\n In this step, we will simply apply our terraform code and keep this stage as Auto-Approved.<\/p>\n\n\n\n To know more about this stage click here<\/a>.<\/p>\n\n\n\n Here in our Terraform Destroy pipeline, we will configure it for manual approval as it is going to be very sensitive & secured. An unnecessarily or unwanted destroyed infrastructure can cause a huge loss of time, money, resources, backup & data. So we’ll keep it highly secured and limit the access to reliable users only. <\/p>\n\n\n\n To know more about this stage click here<\/a>.<\/p>\n\n\n\n We need branching policies in order to save our main\/master branch from unwanted commits. To make any changes to our main\/master branch we need to Merge a branch, after committing changes, into the main\/master branch using Pull Request. Also, remember that our Terraform-CD or Release branch will only be triggered if Terraform-CI or Build branch was triggered from the main branch.<\/p>\n\n\n\n To learn how to configure the branching policy in Azure DevOps click here<\/a>.<\/p>\n\n\n\n So in this article, we get to learn about how to automate and streamline our terraform code deployment using Azure DevOps and use tools like Tfsec for security and compliance, Tflint for linting terraform code and Infracost for Cost Estimation & Cost Difference. Along with that we also learned how to upload our logs into Blob Containers, publish artifacts and make a release pipeline using it which will have different stages for Terraform apply and terraform destroy.<\/p>\n\n\n\n Content References <\/strong>– Reference 1<\/a>, Reference 2<\/a> Opstree<\/a><\/strong> <\/a>is an End to End DevOps solution provider<\/p>\n\n\n\n![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-3.png?w=1024\")
Azure Pipeline For Terraform<\/h2>\n\n\n\n
![](\"https:\/\/lh6.googleusercontent.com\/1llIZz_ka-zdnmK9gA82mczGrsMpMclXPcUFKjHupnXgdZPJ6gn3XilaOuEA68z-OmiciuAiVyA6UZ6vZWSaBXX8eLIHHYBdvA1RIvlkhu7pqQc6MmL9xTrAvFHM_gtWfBYUlI388P8pqiNUGg4v\")
Pre-requisites:<\/h3>\n\n\n\n
Tools used:<\/h3>\n\n\n\n
Build Pipeline steps<\/h2>\n\n\n\n
Part 1: Installing Dependencies<\/h4>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-4.png?w=386\")
- task: Bash@3\n displayName: Install Docker\n enabled: False\n inputs:\n targetType: inline\n script: >-\n sudo apt update\n sudo apt install apt-transport-https ca-certificates curl software- properties-common -y\n curl -fsSL https:\/\/download.docker.com\/linux\/ubuntu\/gpg | sudo apt-key add -\n sudo add-apt-repository \"deb [arch=amd64] https:\/\/download.docker.com\/linux\/ubuntu bionic stable\n sudo add-apt-repository \"deb [arch=amd64] https:\/\/download.docker.com\/linux\/ubuntu bionic stable\"\n sudo apt update\n apt-cache policy docker-ce\n sudo apt install docker-ce -y\n sudo systemctl start docker\n\n - task: Bash@3\n displayName: Install Azure CLI\n enabled: False\n inputs:\n targetType: inline\n script: \"sudo apt-get update \\nsudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg -y\\ncurl -sL https:\/\/packages.microsoft.com\/keys\/microsoft.asc |\\n gpg --dearmor |\\n sudo tee \/etc\/apt\/trusted.gpg.d\/microsoft.gpg > \/dev\/null\\nAZ_REPO=$(lsb_release -cs)\\necho \\\"deb [arch=amd64] https:\/\/packages.microsoft.com\/repos\/azure-cli\/ $AZ_REPO main\\\" |\\n sudo tee \/etc\/apt\/sources.list.d\/azure-cli.list\\nsudo apt-get update\\nsudo apt-get install azure-cli -y\\nsudo apt install unzip -y\"\n\n - task: TerraformInstaller@0\n displayName: Install Terraform 1.1.8\n inputs:\n terraformVersion: 1.1.8\n\n - task: Bash@3\n displayName: Pulling Required Docker Images\n enabled: False\n inputs:\n targetType: inline\n script: >\n # TFSEC\n sudo docker pull tfsec\/tfsec:v1.13.2-arm64v8\n # TFLINT\n sudo docker pull ghcr.io\/terraform-linters\/tflint:v0.35.0\n # InfraCost\n sudo docker pull infracost\/infracost:0.<\/pre>\n\n\n\n
Part 2: Terraform Initializing & Planning<\/h4>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-5.png?w=470\")
-out=plan.out<\/code> flag.<\/p>\n\n\n\n - task: TerraformTaskV2@2\n displayName: 'Terraform : INIT'\n inputs:\n backendServiceArm: a575**********************4bcc71\n backendAzureRmResourceGroupName: ADOagent_rg\n backendAzureRmStorageAccountName: terrastoragestatesreport\n backendAzureRmContainerName: statefile\n backendAzureRmKey: terraform.tfstate\n - task: TerraformTaskV2@2\n displayName: 'Terraform : VALIDATE'\n inputs:\n command: validate\n - task: TerraformTaskV2@2\n displayName: 'Terraform : PLAN ( For Cost Optimization )'\n inputs:\n command: plan\n commandOptions: -lock=false -out=plan.out\n environmentServiceNameAzureRM: a575**********************4bcc71<\/pre>\n\n\n\n
Part 3: Heart of Our Pipeline: Terraform Security Compliance, Linting, Cost Estimation & Cost Difference<\/strong><\/h4>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-6.png?w=335\")
---\nchecks:\n - code: CUS001\n description: Custom check to ensure the Name tag is applied to Resources Group Module \n impact: By not having Name Tag we can't keep track of our Resources\n requiredTypes:\n - module\n requiredLabels:\n - resource_group\n severity: MEDIUM\n matchSpec:\n name: tag_map\n action: contains\n value: Name\n errorMessage: The required Name tag was missing\n\n - code: CUS002\n description: Custom check to ensure the Name tag is applied to Resources Group Module\n impact: By not having Environment Tag we can't keep track of our Resources\n requiredTypes:\n - module\n requiredLabels:\n - resource_group\n severity: CRITICAL\n matchSpec:\n name: tag_map\n action: contains\n value: Environment\n errorMessage: The required Environment tag was missing\n\n - code: CUS003\n description: Custom check to ensure Resource Group is going to be created in Australia East region\n impact: By not having our resource in Australia East we might get some latency\n requiredTypes:\n - module\n requiredLabels:\n - resource_group\n severity: MEDIUM\n matchSpec:\n name: resource_group_location\n action: equals\n value: \"Australia East\"\n errorMessage: The required \"Australia East\" location was missing<\/pre>\n\n\n\n
- task: Bash@3\n displayName: 'Terraform : TFSEC'\n condition: succeededOrFailed()\n enabled: False\n inputs:\n targetType: inline\n script: sudo docker run --rm -v \"$(pwd):\/src\" aquasec\/tfsec \/src --tfvars-file \/src\/terraform.tfvars\n - task: Bash@3\n displayName: 'Terraform : Linting'\n condition: succeededOrFailed()\n enabled: False\n inputs:\n targetType: inline\n script: >\n sudo docker run --rm -v $(pwd):\/data -t ghcr.io\/terraform-linters\/tflint\n - task: Bash@3\n displayName: 'Terraform : Cost Estimation'\n condition: succeededOrFailed()\n enabled: False\n inputs:\n targetType: inline\n script: \"terraform show -json plan.out > plan.json\\n\\nsudo docker run --rm -e INFRACOST_API_KEY=$(INFRACOST_API_KEY) -v \\\"$(pwd):\/src\\\" infracost\/infracost breakdown --path \/src\/plan.json --show-skipped \\n\"\n - task: Bash@3\n displayName: 'Terraform : Cost Difference'\n condition: succeededOrFailed()\n enabled: False\n inputs:\n targetType: inline\n script: >\n sudo docker run --rm -e INFRACOST_API_KEY=$(INFRACOST_API_KEY) -v \"$(pwd):\/src\" infracost\/infracost diff --path \/src\/plan.json --show-skipped > $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber)<\/pre>\n\n\n\n
Part 4: Generating & Uploading Logs<\/h4>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-7.png?w=325\")
- task: Bash@3\n displayName: Generating Logs\n condition: succeededOrFailed()\n enabled: False\n inputs:\n targetType: inline\n script: >\n # Creating Logs Of Tfsec\n\n sudo docker run --rm -v \"$(pwd):\/src\" aquasec\/tfsec \/src --tfvars-file \/src\/terraform.tfvars > $(Build.DefinitionName)-tfsec-$(Build.BuildNumber)\n\n cat $(Build.DefinitionName)-tfsec-$(Build.BuildNumber)\n\n\n #Creating Logs Of Cost Estimation\n\n sudo docker run --rm -e INFRACOST_API_KEY=$(INFRACOST_API_KEY) -v \"$(pwd):\/src\" infracost\/infracost breakdown --path \/src\/plan.json --show-skipped --format html > $(Build.DefinitionName)-cost-$(Build.BuildNumber).html\n\n cat $(Build.DefinitionName)-cost-$(Build.BuildNumber).html\n\n\n #Creating Logs Of Cost Diffrence\n\n sudo docker run --rm -e INFRACOST_API_KEY=$(INFRACOST_API_KEY) -v \"$(pwd):\/src\" infracost\/infracost diff --path \/src\/plan.json --show-skipped > $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber)\n\n cat $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber)\n - task: AzureCLI@2\n displayName: 'Upload tfsec file '\n condition: succeededOrFailed()\n enabled: False\n inputs:\n connectedServiceNameARM: a575*********************c71\n scriptType: bash\n scriptLocation: inlineScript\n inlineScript: >\n az storage blob upload --file $(Build.DefinitionName)-tfsec-$(Build.BuildNumber) --name $(Build.DefinitionName)-tfsec-$(Build.BuildNumber) --account-name terrastoragestatesreport --container-name report\n cwd: $(Pipeline.Workspace)\/s\n - task: AzureCLI@2\n displayName: 'Upload cost file '\n condition: succeededOrFailed()\n enabled: False\n inputs:\n connectedServiceNameARM: a575*********************c71\n scriptType: bash\n scriptLocation: inlineScript\n inlineScript: az storage blob upload --file $(Build.DefinitionName)-cost-$(Build.BuildNumber).html --name $(Build.DefinitionName)-cost-$(Build.BuildNumber).html --account-name terrastoragestatesreport --container-name report\n cwd: $(Pipeline.Workspace)\/s\n - task: AzureCLI@2\n displayName: Upload cost diff file\n condition: succeededOrFailed()\n enabled: False\n inputs:\n connectedServiceNameARM: a575*********************c71\n scriptType: bash\n scriptLocation: inlineScript\n inlineScript: az storage blob upload --file $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber) --name $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber) --account-name terrastoragestatesreport --container-name report\n cwd: $(Pipeline.Workspace)\/s<\/pre>\n\n\n\n
Part 5: Generating Artifacts<\/h4>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-8.png?w=312\")
- task: Bash@3\n displayName: Generating Logs\n condition: succeededOrFailed()\n enabled: False\n inputs:\n targetType: inline\n script: >\n # Creating Logs Of Tfsec\n\n sudo docker run --rm -v \"$(pwd):\/src\" aquasec\/tfsec \/src --tfvars-file \/src\/terraform.tfvars > $(Build.DefinitionName)-tfsec-$(Build.BuildNumber)\n\n cat $(Build.DefinitionName)-tfsec-$(Build.BuildNumber)\n\n\n #Creating Logs Of Cost Estimation\n\n sudo docker run --rm -e INFRACOST_API_KEY=$(INFRACOST_API_KEY) -v \"$(pwd):\/src\" infracost\/infracost breakdown --path \/src\/plan.json --show-skipped --format html > $(Build.DefinitionName)-cost-$(Build.BuildNumber).html\n\n cat $(Build.DefinitionName)-cost-$(Build.BuildNumber).html\n\n\n #Creating Logs Of Cost Diffrence\n\n sudo docker run --rm -e INFRACOST_API_KEY=$(INFRACOST_API_KEY) -v \"$(pwd):\/src\" infracost\/infracost diff --path \/src\/plan.json --show-skipped > $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber)\n\n cat $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber)\n - task: AzureCLI@2\n displayName: 'Upload tfsec file '\n condition: succeededOrFailed()\n enabled: False\n inputs:\n connectedServiceNameARM: a575*********************c71\n scriptType: bash\n scriptLocation: inlineScript\n inlineScript: >\n az storage blob upload --file $(Build.DefinitionName)-tfsec-$(Build.BuildNumber) --name $(Build.DefinitionName)-tfsec-$(Build.BuildNumber) --account-name terrastoragestatesreport --container-name report\n cwd: $(Pipeline.Workspace)\/s\n - task: AzureCLI@2\n displayName: 'Upload cost file '\n condition: succeededOrFailed()\n enabled: False\n inputs:\n connectedServiceNameARM: a575*********************c71\n scriptType: bash\n scriptLocation: inlineScript\n inlineScript: az storage blob upload --file $(Build.DefinitionName)-cost-$(Build.BuildNumber).html --name $(Build.DefinitionName)-cost-$(Build.BuildNumber).html --account-name terrastoragestatesreport --container-name report\n cwd: $(Pipeline.Workspace)\/s\n - task: AzureCLI@2\n displayName: Upload cost diff file\n condition: succeededOrFailed()\n enabled: False\n inputs:\n connectedServiceNameARM: a575*********************c71\n scriptType: bash\n scriptLocation: inlineScript\n inlineScript: az storage blob upload --file $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber) --name $(Build.DefinitionName)-cost-diff-$(Build.BuildNumber) --account-name terrastoragestatesreport --container-name report\n cwd: $(Pipeline.Workspace)\/s<\/pre>\n\n\n\n
Release Pipeline Steps<\/h2>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-10.png?w=792\")
Part 1: Auto Approval For Terraform Apply<\/h4>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-9.png?w=523\")
steps:\n- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0\n displayName: 'Install Terraform 1.1.8'\n inputs:\n terraformVersion: 1.1.8\n\n- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2\n displayName: 'Terraform : INIT'\n inputs:\n workingDirectory: '$(System.DefaultWorkingDirectory)\/_IAC-CI\/release'\n backendServiceArm: 'Opstree-PoCs (4c9***************************f3c)'\n backendAzureRmResourceGroupName: 'ADOagent_rg'\n backendAzureRmStorageAccountName: terrastoragestatesreport\n backendAzureRmContainerName: statefile\n backendAzureRmKey: terraform.tfstate\n\n- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2\n displayName: 'Terraform : APPLY'\n inputs:\n command: apply\n workingDirectory: '$(System.DefaultWorkingDirectory)\/_IAC-CI\/release'\n commandOptions: '--auto-approve'\n environmentServiceNameAzureRM: 'Opstree-PoCs (4c9***************************f3c)'<\/pre>\n\n\n\n
Part 2: Manual Approval For Terraform Destroy<\/h4>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-11.png?w=382\")
steps:\n- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0\n displayName: 'Install Terraform 1.1.8'\n inputs:\n terraformVersion: 1.1.8\n\n- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2\n displayName: 'Terraform : INIT'\n inputs:\n workingDirectory: '$(System.DefaultWorkingDirectory)\/_IAC-CI\/release'\n backendServiceArm: 'Opstree-PoCs (4c9***************************f3c)'\n backendAzureRmResourceGroupName: 'ADOagent_rg'\n backendAzureRmStorageAccountName: terrastoragestatesreport\n backendAzureRmContainerName: statefile\n backendAzureRmKey: terraform.tfstate\n\n- task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2\n displayName: 'Terraform : APPLY'\n inputs:\n command: apply\n workingDirectory: '$(System.DefaultWorkingDirectory)\/_IAC-CI\/release'\n commandOptions: '--auto-approve'\n environmentServiceNameAzureRM: 'Opstree-PoCs (4c9***************************f3c)'<\/pre>\n\n\n\n
Bonus: Branching Policy<\/h2>\n\n\n\n
![\"\"](\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2022\/06\/image-12.png?w=1024\")
Conclusion<\/h2>\n\n\n\n
Image References<\/strong> – Image 1<\/a>, Image 2<\/a><\/p>\n\n\n\n
Blog Pundit:<\/strong> Bhupender Rawat<\/strong><\/a> and Sandeep Rawat<\/strong><\/a><\/p>\n\n\n\n