Know How to Use Velero to Backup and Migrate Kubernetes Resources and Persistent Volumes

“Murphy’s law doesn’t mean that something bad will happen. It means that whatever can happen, will happen.” This is something related to destiny but we should not totally rely upon it and should be prepared for the worst.
The same philosophy referenced above applies to the tech world too. That’s the reason we should be prepared with our backup options choices possibly, a data set or Kubernetes cluster.

Kubernetes backup solutions bring down the risk and empower faster recovery time while providing key benefits like: disaster recovery and backup & restore. Now we have to explore some simple and convenient options to take Kubernetes backup. While working on a similar project I came to know about Velero which can fulfil our needs to take Kubernetes backup and restore and it is easy to use.

https://c.tenor.com/wtz8za_yyGoAAAAM/darla-little-rascals.gif

Velero is an open-source tool for securely backing up and restoring resources in a Kubernetes cluster, performing disaster recovery, and moving resources and persistent volumes to another Kubernetes cluster.

Velero lets you:

  • Take backups of our cluster and restore in case of loss.
  • Migrate cluster resources to other clusters.
  • Replicate our production cluster with development and testing clusters.

Agenda Of this blog:

  • To set up Velero on AlibabaCloud:
    • Download the official release of velero
    • Create our OSS bucket
    • Create a RAM user and secrets file for Velero
    • Install the velero and velero-plugin for alibabacloud
    • Create the backup into OSS bucket
  • Restore the backup in the same cluster or another cluster
  • Run velero on AzureCloud to migrate the Kubernetes resources
    • Migrate the OSS bucket to the Azure storage account
    • Create Credentials file for azure Velero
    • Install the velero and velero-plugin for azurecloud
    • Test the backup and restore/migrate from the storage account

To set up Velero on AlibabaCloud:

Run velero on AlibabaCloud

To do backup/restore on Alibaba Cloud via Valero utility, you need to install and configure Valero-plugin for Alibaba Cloud.

Download the official release of velero & install

  • Download tar file of velero
https://github.com/vmware-tanzu/velero/releases/tag/v<velero_version>
  • Extract tar File :

tar -xvf <RELEASE-TARBALL-NAME>.tar.gz

Note: Move the extracted velero binary to somewhere in your $PATH (e.g. /usr/local/bin)

Create OSS bucket on Alibaba cloud

  • Firstly, you need to create the env variable of bucket name and region :
BUCKET=<YOUR_BUCKET>  
REGION=<YOUR_REGION>
  • Create OSS bucket using ossutil utility
ossutil mb oss://$BUCKET \
--storage-class Standard \
--acl=private

Create an RAM user and secrets file for Velero

  • Create the RAM user and attach the below policies to give velero the necessary permissions:
{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "ecs:DescribeSnapshots",
                "ecs:CreateSnapshot",
                "ecs:DeleteSnapshot",
                "ecs:DescribeDisks",
                "ecs:CreateDisk",
                "ecs:Addtags",
                "oss:PutObject",
                "oss:GetObject",
                "oss:DeleteObject",
                "oss:GetBucket",
                "oss:ListObjects"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        }
    ]
}
  • Create an access key and secret key for the user.
  • Create a Velero-specific credentials file example: credentials-velero
ALIBABA_CLOUD_ACCESS_KEY_ID=<ALIBABA_CLOUD_ACCESS_KEY_ID>
ALIBABA_CLOUD_ACCESS_KEY_SECRET=<ALIBABA_CLOUD_ACCESS_KEY_SECRET>
ALIBABA_CLOUD_OSS_ENDPOINT=<ALIBABA_CLOUD_OSS_ENDPOINT>

Note: oss endpoint is the value oss-$REGION.aliyuncs.com

Install the velero and velero-plugin for alibabacloud

  • Create a namespace velero and secret cloud-credentials
kubectl create namespace velero
  • Run the following command to create and run velero and velero-plugin for alibabacloud
velero install \
  --provider alibabacloud \
    --namespace velero \ 
  --image registry.$REGION.aliyuncs.com/acs/velero:1.4.2-2b9dce65-aliyun \
  --bucket $BUCKET \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=false \
  --backup-location-config region=$REGION \
  --use-restic \
  --plugins registry.$REGION.aliyuncs.com/acs/velero-plugin-alibabacloud:v1.0.0-2d33b89 \
  --wait
  • We should see Velero’s pods come up.
kubectl logs deploy/velero -n velero

Create the backup into the OSS bucket

  • Use the below commands to create the backup:
1. velero backup create my-backup
2. velero backup logs my-backup

  • Restore the backup in the same cluster or another cluster

Create a new cluster and restore the backup using velero

Note: If we will create a new cluster, follow the above steps to Install the velero and velero-plugin for alibabacloud k8s cluster

  • To restore the backup from the OSS bucket 
velero get backup

  • Restore the backup in the cluster.

Run velero on AzureCloud to migrate the Kubernetes resources

Run velero on AzureCloud

To do backup/migrate on Alibaba Cloud through Velero utility, you need to install and configure velero and velero-plugin for azurecloud.

Migrate the OSS bucket to the Azure storage account

  • First, create a storage account, this will store the backups.
1. STORAGE_ACCOUNT='Name'
2. STORAGE_RESOURCE_GROUP='RG_name'
3. az storage account create -n $STORAGE_ACCOUNT -g $STORAGE_RESOURCE_GROUP
  • Then, create the container to place the backups.
1. STORAGE_CONTAINER_NAME='Container_name'
2. az storage container create --account-name $STORAGE_ACCOUNT -n $STORAGE_CONTAINER_NAME
  • Using the azure utility, run below command from alibaba bastion host to migrate the bucket from alibaba to azure storage.
1. azcopy cp "/root/uat-noon1/*" "https://storageaccount.blob.core.windows.net/new1?XXXXXXXXXXXX --recursive=true.
  • We should see all backups in the Azure storage account.

Create Credentials file for azure Velero

  • Start by getting the resource group where the virtual machines for our AKS cluster.
1. AZURE_RESOURCE_GROUP=$(az aks show -n mycluster -g myresourcegroup --query "nodeResourceGroup" -o tsv)
  • Then, get the Azure subscription and Azure AD tenant information.
1. AZURE_SUBSCRIPTION_ID=$(az account list --query '[?isDefault].id' -o tsv)
2. AZURE_TENANT_ID=$(az account list --query '[?isDefault].tenantId' -o tsv)
  • Now create an Azure service principal for Velero to authenticate and create env variable for client id & client secret.
1. AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac --name "mycluster-velero" --role "Contributor" --query 'password' -o tsv)
2. AZURE_CLIENT_ID=$(az ad sp list --display-name "mycluster-velero" --query '[0].appId' -o tsv)
  • Create a file called credentials-velero-azure
cat << EOF  > ./credentials-velero-azure
AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
AZURE_TENANT_ID=${AZURE_TENANT_ID}
AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
AZURE_CLOUD_NAME=AzurePublicCloud
EOF

Install the velero and velero-plugin for azurecloud

  • Using kubectl, upload the credentials to a Kubernetes secret in a new velero namespace.
1. kubectl create ns velero
2. kubectl create secret generic velero-credentials -n velero --from-literal="cloud=$(cat ./credentials-velero-azure)"
  • Now, we install Velero using Helm and using Velero’s Azure plugin.
    – First, add the VMware Helm repo.
helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
  • Install the velero using the helm chart.
helm install velero vmware-tanzu/velero --namespace velero --version 2.13.2 \
--set "initContainers[0].image=velero/velero-plugin-for-microsoft-azure:v1.1.0" \
--set "initContainers[0].imagePullPolicy=IfNotPresent" \
--set "initContainers[0].volumeMounts[0].mountPath=/target" \
--set "initContainers[0].volumeMounts[0].name=plugins" \
--set "initContainers[0].name=velero-plugin-for-azure" \
--set credentials.existingSecret='velero-credentials' \
--set configuration.provider='azure' \
--set configuration.backupStorageLocation.bucket=$STORAGE_CONTAINER_NAME \
--set configuration.backupStorageLocation.config.resourceGroup=$STORAGE_RESOURCE_GROUP \
--set configuration.backupStorageLocation.config.storageAccount=$STORAGE_ACCOUNT \
--set configuration.backupStorageLocation.config.subscriptionId=$AZURE_SUBSCRIPTION_ID \
--set configuration.volumeSnapshotLocation.name='azure-eastus' \
--set configuration.volumeSnapshotLocation.config.resourceGroup=$STORAGE_RESOURCE_GROUP \
--set configuration.volumeSnapshotLocation.config.subscriptionId=$AZURE_SUBSCRIPTION_ID
  • You should see Velero’s pods come up.
kubectl get pods -n velero

Testing the backup and restore/migrate from the storage account

  • List the backup from the storage account.

  • Restore the Kubernetes backup in azure cluster

Note: Backup restore starts and it will take some time

  • After backup restores successfully, check all namespaces, secrets & PV of Kubernetes cluster:

Conclusion:

Valero is an open-source tool for securely backing up and restoring resources in the Kubernetes cluster, performing disaster recovery, moving resources and persistent volumes to another Kubernetes cluster. The best part about the Valero tool is that you need not worry about your Kubernetes cluster, it will take auto backup on a timely basis. Last but not least it has the capabilities to restore the complete Kubernetes cluster or the basis of namespaces in another cluster if something goes wrong with your Kubernetes cluster.


Blog Pundit: Naveen Verma and Sanjeev Pandey

Opstree is an End to End DevOps solution provider

CONTACT US

Connect Us

One thought on “Know How to Use Velero to Backup and Migrate Kubernetes Resources and Persistent Volumes”

  1. when I copy the backup from gcs to s3 , I can’t see backup from target, used velero bakcup get.

    Reply

Leave a Reply