No matter where organizations are in their cloud journey, cloud cost optimization remains an essential and top-priority concern. As organizations align increasing workloads to the cloud in terms of size there is a high possibility to lose sight of the overall cloud environment and the costs associated with it. Therefore, it’s essential to optimize cloud costs so as to maximise return over investment (ROI). With the same concern we were working towards reducing the cost of provisioned storage for one our clients. This post covers a particularly interesting issue around the same.
When we found that gp3 volumes are cheaper than gp2 by looking at price comparison given by aws, we decided to migrate. Once our migration was complete w.r.t. volumes under cost optimization and mail was circulated as well stating that we would be launching only gp3 ebs volumes from now onwards, we did not expect one particular issue to creep up on us. After 2-3 days, we saw that gp2 volumes were still launching.
We checked the volumes that were launched recently and found the culprit “storage class”. We were using a storage class for dynamic volume provisioning of aws ebs volumes. Turns out, we forgot to change storage class in our eks cluster.
Warning ProvisioningFailed 9s (x5 over 59s) persistentvolume-controller Failed to provision volume with StorageClass "gp3": invalid AWS VolumeType "gp3".
With the above error we came to know that we need to change the underlying provisioner of the storage class because provisioner: kubernetes.io/aws-ebs only supported gp2 volume type as per official documentation. On further Googling to get out of this sticky situation, we found a CSI provisioner could fix this problem. But implementing it wasn’t exactly a straight shot.
Problem Statement
Unable to launch gp3 volumes with in-tree kubernetes plugin. Followed a blog, but ended up with another error, “not able to provision persistent volume using csi provisioner”
Steps to reproduce
Delete storage class with default provisioner i.e. Fig. 1 belowkubectl delete sc buildpiper-storage
Create storage class with csi provisioner i.e. Fig. 2 belowkubectl apply -f storage-class-csi.yaml
Get eks worker node role name i.e. Fig. 3 belowkubectl get configmap aws-auth -n kube-system
copy the ROLE-NAME
create IAM policy with name Amazon_EBS_CSI_Driver
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteSnapshot",
"ec2:DeleteTags",
"ec2:DeleteVolume",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DetachVolume"
],
"Resource": "*"
}
]
}
Associate IAM Policy to Worker Node IAM Role:
1. Open AWS console -> Go to Services -> IAM -> Roles
2. Search for role with name ROLE-NAME and open it
3. Click on Permissions tab – Click on Attach Policies
4. Search for Amazon_EBS_CSI_Driver and click on Attach Policy
Install EBS CSI Driverkubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-0.1
Resolution
Continuing to struggle with this issue, we looked up compatible version at Kubernetes Version Compatibility Matrix. Following this, upgraded k8s cluster from version 1.16 to 1.17 and it started working with version “release-0.10” for me.
Conclusion
We sought cost optimization and during that journey, came across a completely different issue. Got to know about CSI plugin that helped to launch gp3 ebs volumes via storage class. We had also faced version compatibility issue that leads to failure in achieving intended result. Overall it was a satisfying experience for me.
I have tried to keep it simple and frame the content based on my experience. If you feel it needs any correction or feedback, feel free to comment.
Keep learning and stay safe !
Image Reference
Blog Pundit: Adeel Ahmad & Naveen Verma
Opstree is an End to End DevOps solution provider
Connect Us
