{"id":11932,"date":"2022-10-25T11:22:16","date_gmt":"2022-10-25T05:52:16","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=11932"},"modified":"2024-09-24T17:22:36","modified_gmt":"2024-09-24T11:52:36","slug":"on-premise-setup-of-kubernetes-cluster-using-kubespray-offline-mode-part-1","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2022\/10\/25\/on-premise-setup-of-kubernetes-cluster-using-kubespray-offline-mode-part-1\/","title":{"rendered":"On-Premise Setup of Kubernetes Cluster using KubeSpray (Offline Mode) – PART 1"},"content":{"rendered":"\r\n

Today, most organizations are moving to Managed Services<\/strong> like EKS<\/strong> (Elastic Kubernetes Services), and AKS<\/strong> (Azure Kubernetes Services), for easier handling of the Kubernetes Cluster. With Managed Kubernetes we do not have to take care of our Master Nodes, cloud providers will be responsible for all Master Nodes and Worker Nodes, freeing up our time. We just need to deploy our Microservices over the Worker nodes. You can pay extra to achieve an uptime of 99.95%. Node repair ensures that a cluster remains healthy and reduces the chances of possible downtime. This is good in many cases but it makes it an expensive ordeal as AKS costs $0.10 per cluster per hour. You have to install upgrades for the VPC CNI yourself and also, install Calico CNI. There is no IDE extension for developing EKS code. it also creates a dependency on the particular Cloud Provider.<\/p>\r\n\r\n\r\n\r\n

To skip the dependency on any Cloud Provider we have to create a Vanilla<\/strong> Kubernetes Cluster<\/strong>. This means we have to take care of all the components – all the Master and Worker Nodes of the Cluster by ourselves.<\/p>\r\n\r\n\r\n\r\n

Here we got a scenario in which one of our client’s requirements was to set up a Kubernetes cluster over On-premises Servers, under the condition of no Internet connectivity. So I choose to perform the setup of the Kubernetes Cluster via Kubespray<\/strong>.<\/p>\r\n\r\n\r\n\r\n

Why Kubespray?<\/h4>\r\n\r\n\r\n\r\n

Kubespray<\/strong> is a composition of Ansible playbooks, inventory,
provisioning tools, and domain knowledge for generic
OS\/Kubernetes clusters configuration management tasks.
Kubespray provides a highly available cluster<\/em><\/strong>, composable
(choice of the network plugin for instance), supports most popular Linux distributions, and continuous integration tests<\/em><\/strong>.<\/p>\r\n\r\n\r\n\r\n

<\/p>\r\n\r\n\r\n\r\n

Creating a cluster<\/em><\/strong><\/h3>\r\n\r\n\r\n\r\n


\u25cf Minimum required version of Kubernetes is v1.22
\u25cf Ansible v2.11+, Jinja 2.11+ and python-netaddr is installed on the machine that will run Ansible commands
\u25cf The target servers must have access to the Internet in order to pull docker images. Otherwise, additional configuration is required. Checkout offline_environment.md file.
\u25cf The target servers are configured to allow IPv4 forwarding.
\u25cf If using IPv6 for pods and services, the target servers are configured to allow IPv6 forwarding.
\u25cf The firewalls are not managed, you have to implement your own rules the way you are used to. In order to avoid any issues during the deployments you should disable your firewall.
\u25cf If Kubespray is run from a non-root user account, the correct privilege escalation method should be configured in the target servers. Then the ansible_become flag or command parameters –become or -b should be specified.<\/p>\r\n\r\n\r\n\r\n

Setup K8s Cluster using Kubespray(Offline Mode):<\/h2>\r\n\r\n\r\n\r\n

Prerequisite:<\/h3>\r\n\r\n\r\n\r\n

\u25cf All required binaries should be downloaded Offline.
\u25cf All binaries are pushed to whitelisted Storage to K8s Servers as a defined directory structure.
\u25cf Ansible over jump host.
\u25cf Passwordless connectivity b\/w jump host and all K8s nodes.
\u25cf Python over all K8s nodes.
\u25cf ACR connectivity b\/w all K8s nodes and decided ACR.
\u25cf YUM Proxy setup from K8s nodes to nexus.
\u25cf Get the Kubespray Ansible Role over Jump host.<\/p>\r\n\r\n\r\n\r\n

After completing all the prerequisites We can continue further<\/h3>\r\n\r\n\r\n\r\n

1) <\/strong>First of all we need to pull the required code from the official GitHub repository.<\/p>\r\n\r\n\r\n\r\n

git clone https:\/\/github.com\/kubernetes-sigs\/kubespray.git<\/pre>\r\n\r\n\r\n\r\n
As we are setting up an Offline Kubernetes cluster, we have to do
some required changes in the below file and we have to update
all the offline binaries over the location.<\/strong><\/p>\r\n\r\n\r\n\r\n
vi kubespray\/roles\/download\/defaults\/main.yml<\/pre>\r\n\r\n\r\n\r\n
\"\"\r\n
Previous Download file<\/strong><\/figcaption>\r\n<\/figure>\r\n\r\n\r\n\r\n
\"\"\r\n
Updated Download File<\/strong><\/figcaption>\r\n<\/figure>\r\n\r\n\r\n\r\n
\"\"\r\n
Nexus View<\/strong><\/figcaption>\r\n<\/figure>\r\n\r\n\r\n\r\n
We have to modify all the required URLs as per the directory structure given over Nexus.<\/strong><\/p>\r\n\r\n\r\n\r\n
2)<\/strong> After Modifying all the Binary URLS we will provide Root Privileges<\/strong> to all the tasks mentioned in cluster.yml<\/strong> file.
cluster.yml<\/p>\r\n\r\n\r\n\r\n
cd kubespray\/<\/pre>\r\n\r\n\r\n\r\n
vi cluster.yml<\/pre>\r\n\r\n\r\n\r\n
\"\"<\/figure>\r\n\r\n\r\n\r\n
3) After editing cluster.yml we have to set SELINUX Disabled over each Kubernetes node. We have to run the below command over each server.<\/p>\r\n\r\n\r\n\r\n
sudo vi \/etc\/selinux\/config<\/pre>\r\n\r\n\r\n\r\n
We have to uncomment the line and set<\/strong><\/p>\r\n\r\n\r\n\r\n
SELINUX=disabled<\/pre>\r\n\r\n\r\n\r\n
4) After the previous step, we have to turn off the SWAP Memory. For that, we need to edit the \u2018fstab\u2019 file over each K8s node. We have to run the below command over each server.<\/p>\r\n\r\n\r\n\r\n
sudo vi \/etc\/fstab<\/pre>\r\n\r\n\r\n\r\n
Here we have to comment on the line which is having below content:<\/p>\r\n\r\n\r\n\r\n
#\/dev\/mapper\/rootVG-swapLV swap swap default     0        0<\/pre>\r\n\r\n\r\n\r\n
5) After setting up all the K8s nodes We have to restart the server.
We have to run this command over all the servers.<\/p>\r\n\r\n\r\n\r\n
sudo init 6<\/pre>\r\n\r\n\r\n\r\n
6) Now we will come back to our jump host and run the below
commands one by one.<\/p>\r\n\r\n\r\n\r\n
a) Copy ‘inventory\/sample’ as ‘inventory\/mycluster’:<\/strong><\/p>\r\n\r\n\r\n\r\n
cp -rfp inventory\/sample inventory\/mycluster<\/pre>\r\n\r\n\r\n\r\n
b) Update Ansible inventory file with inventory builder:<\/strong><\/p>\r\n\r\n\r\n\r\n
declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)<\/strong><\/pre>\r\n\r\n\r\n\r\n
CONFIG_FILE=inventory\/mycluster\/hosts.yaml python3 contrib\/inventory_builder\/inventory.py ${IPS[@]}<\/pre>\r\n\r\n\r\n\r\n
c) Review and change parameters:<\/strong><\/p>\r\n\r\n\r\n\r\n
cat inventory\/mycluster\/group_vars\/all\/all.yml<\/strong><\/pre>\r\n\r\n\r\n\r\n
cat inventory\/mycluster\/group_vars\/k8s_cluster\/k8s-cluster.yml<\/pre>\r\n\r\n\r\n\r\n
7) Deploy Kubespray with Ansible Playbook<\/p>\r\n\r\n\r\n\r\n
ansible-playbook -i inventory\/mycluster\/hosts.yaml --become\r\n--become-user=root cluster.yml<\/pre>\r\n\r\n\r\n\r\n
Note: It will take 25-30 mins to run the complete playbook.<\/strong><\/p>\r\n\r\n\r\n\r\n
8) After the completion of the playbook we can run the below command over K8s Master Node to check whether all nodes are connected up and running.<\/p>\r\n\r\n\r\n\r\n
sudo \/usr\/local\/bin\/kubectl get nodes -o wide<\/pre>\r\n\r\n\r\n\r\n
It will give you the detail of your Cluster Nodes.<\/p>\r\n\r\n\r\n\r\n
Conclusion<\/h2>\r\n\r\n\r\n\r\n
This is how we can set up Kubernetes Cluster<\/strong> via Kubespray<\/strong>. With all that set, we need to add monitoring, tracing, logging, and all the associated operations for troubleshooting. Now as we have reached the end of the blog – what do you think I could have done something differently? Do comment. Also If you face any sort of issue in setting up the Kubernetes cluster drop a comment. I’ll be back with Part – 2 of the blog series where I’ll discuss setting up the Metrics Server, and Redis Setup.<\/p>\r\n\r\n\r\n\r\n
Blog Pundits: Mehul Sharma<\/a> and Sandeep Rawat<\/a><\/strong><\/p>\r\n\r\n\r\n\r\n
OpsTree<\/a><\/strong> is an End-to-End DevOps Solution Provider.<\/p>\r\n\r\n\r\n\r\n
\r\n
Contact Us<\/a><\/div>\r\n<\/div>\r\n\r\n\r\n\r\n
Connect with Us<\/strong><\/p>\r\n\r\n\r\n\r\n