A blog site on our Real life experiences with various phases of DevOps starting from VCS, Build & Release, CI/CD, Cloud, Monitoring, Containerization.
All we could think of imagining a routine day of a NOC guy is looking at all the fancy and colorful multiple screens around, but is this all it?
The answer to the above statement is NO! As a NOC, We have access to the information that is critical to analyze and plot company infra strength and on top of that, access to the servers and protected network makes the situation more critical if thing happens to be in wrong hands.
Hello friends, if you are reading this blog, I assume that you have gone through the first part of my blog . However, if you haven’t, I suggest you to go through the link before reading this blog.
Continue reading “Perfect Spot Instance’s Imperfections | part-II”
Recently I was working on a project which includes Terraform and AWS stuff. While working on that I was using my local machine for terraform code testing and luckily everything was going fine. But when we actually want to test it for the production environment we got some issues there. Then, as usual, we started to dig into the issue and finally, we got the issue which was quite a silly one 😜. The production server Terraform version and my local development server Terraform version was not the same.
After wasting quite a time on this issue, I decided to come up with a solution so this will never happen again.
But before jumping to the solution, let’s think is this problem was only related to Terraform or do we have faced the similar kind of issue in other scenarios as well.
Well, I guess we face a similar kind of issue in other scenarios as well. Let’s talk about some of the scenario’s first.
Suppose you have to create a CI pipeline for a project and that too with code re-usability. Now pipeline is ready and it is working fine in your project and then after some time, you have to implement the same kind of pipeline for the different project. Now you can use the same code but you don’t know the exact version of tools which you were using with CI pipeline. This will lead you to error elevation.
Let’s take another example, suppose you are developing something in any of the programming languages. Surely that utility or program will have some dependencies as well. While installing those dependencies on the local system, it can corrupt your complete system or package manager for dependency management. A decent example is Pip which is a dependency manager of Python😉.
These are some example scenarios which we have faced actually and based on that we got the motivation for writing this blog.
To resolve all this problem we just need one thing i.e. containers. I can also say docker as well but container and docker are two different things.
But yes for container management we use docker.
So let’s go back to our first problem the terraform one. If we have to solve that problem there are multiple ways to solve this. But we tried it to solve this using Docker.
As Docker says
Build Once and Run Anywhere
So based on this statement what we did, we created a Dockerfile for required Terraform version and stored it alongside with the code. Basically our Dockerfile looks like this:-
FROM alpine:3.8
MAINTAINER OpsTree.com
ENV TERRAFORM_VERSION=0.11.10
ARG BASE_URL=https://releases.hashicorp.com/terraform
RUN apk add --no-cache curl unzip bash \
&& curl -fsSL -o /tmp/terraform.zip ${BASE_URL}/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip \
&& unzip /tmp/terraform.zip -d /usr/bin/
WORKDIR /opstree/terraform
USER opstree
In this Dockerfile, we are defining the version of Terraform which needs to run the code.
In a similar fashion, all other above listed problem can be solved using Docker. We just have to create a Dockerfile with exact dependencies which are needed and that same file can work in various environments and projects.
To take it to the next level you can also dump a Makefile as well to make everyone life easier. For example:-
IMAGE_TAG=latest
build-image:
docker build -t opstree/terraform:${IMAGE_TAG} -f Dockerfile .
run-container:
docker run -itd --name terraform -v ~/.ssh:/root/.ssh/ -v ~/.aws:/root/.aws -v ${PWD}:/opstree/terraform opstree/terraform:${IMAGE_TAG}
plan-infra:
docker exec -t terraform bash -c "terraform plan"
create-infra:
docker exec -t terraform bash -c "terraform apply -auto-approve"
destroy-infra:
docker exec -t terraform bash -c "terraform destroy -auto-approve"
And trust me after making this utility available the reactions of the people who will be using this utility will be something like this:-
Now I am assuming you guys will also try to simulate the Docker in multiple scenarios as much as possible.
There are a few more scenarios which yet to be explored to enhance the use of Docker if you find that before I do, please let me know.
Thanks for reading, I’d really appreciate any and all feedback, please leave your comment below if you guys have any feedback.
Cheers till the next time.
For those who have surfed straight to this blog, please check out the previous part of this series Unix File Tree Part-1 and those who have stayed tuned for this part, welcome back.In the previous part, we discussed the philosophy and the need for file tree. In this part, we will dive deep into the significance of each directory.
Dayum!! that’s a lot of stuff to gulp at once, we’ll kick out things one after the other.
Let’s talk about the crucial directories which play a major role.
We’ll be discussing those directories which confuse us always, which have almost a similar purpose but still are in separate locations and when asked about them we go like ummmm…….
/bin vs /usr/bin vs /sbin vs /usr/local/bin
This might get almost clear out when I explained the significance of /usr in the above paragraph. Since Unix designers planned /usr to be the local directories of individual users so it contained all of the sub-directories like /usr/bin, /usr/sbin, /usr/local/bin. But the question remains the same how the content is different?
/usr/bin:
/usr/sbin:
I hope I have covered most of the directories which you might come across frequently and your questions must have been answered.
Now that we know about the significance of each UNIX directory, It’s time to use them wisely the way they are supposed to be.
Please feel free to reach me out for any suggestions.
Goodbye till next time!
References: https://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/usr.htmlhttps://askubuntu.com/questions/130186/what-is-the-rationale-for-the-usr-directoryhttps://askubuntu.com/questions/308045/differences-between-bin-sbin-usr-bin-usr-sbin-usr-local-bin-usr-localhttp://index-of.es/Varios-2/How%20Linux%20Works%20What%20Every%20Superuser%20Should%20Know.pdf
https://imgflip.com/memegenerator
When we say CI/CD as code, it should have modularity and reusability which results in Reducing integration problems and allowing you to deliver software more rapidly.
Jenkins Shared library is the concept of having a common pipeline code in the version control system that can be used by any number of pipelines just by referencing it. In fact, multiple teams can use the same library for their pipelines.
Our thought is putting all pipeline functions in vars is much more practical approach, while there is no other good way to do inheritance, we wanted to use Jenkins Pipelines the right way but it has turned out to be far more practical to use vars for global functions.
Practical Strategy
As we know Jenkins Pipeline’s shared library support allows us to define and develop a set of shared pipeline helpers in this repository and provides a straightforward way of using those functions in a Jenkinsfile.This simple example will just illustrate how you can provide input to a pipeline with a simple YAML file so you can centralize all of your pipelines into one library. The Jenkins shared library example:And the example app that uses it:
You would have the following folder structure in a git repo:
└── vars
├── opstreePipeline.groovy
├── opstreeStatefulPipeline.groovy
├── opstreeStubsPipeline.groovy
└── pipelineConfig.groovy
This repo would be configured in under Manage Jenkins > Configure System in the Global Pipeline Libraries section. In that section Jenkins requires you give this library a Name. Example opstree-library
Let’s assume that project repository would have a pipeline.yaml file in the project root that would provide input to the pipeline:Pipeline.yaml
ENVIRONMENT_NAME: test SERVICE_NAME: opstree-service DB_PORT: 3079 REDIS_PORT: 6079
Then, to utilize the shared pipeline library, the Jenkinsfile in the root of the project repo would look like:
@Library ('opstree-library@master') _
opstreePipeline()
So how does it all work? First, the following function is called to get all of the configuration data from the pipeline.yaml file:
def call() {
Map pipelineConfig = readYaml(file: "${WORKSPACE}/pipeline.yaml")
return pipelineConfig
}
You can see the call to this function in opstreePipeline(), which is called by the Jenkinsfile.
def call() {
node('Slave1') {
stage('Checkout') {
checkout scm
}
def p = pipelineConfig()
stage('Prerequistes'){
serviceName = sh (
script: "echo ${p.SERVICE_NAME}|cut -d '-' -f 1",
returnStdout: true
).trim()
}
stage('Build & Test') {
sh "mvn --version"
sh "mvn -Ddb_port=${p.DB_PORT} -Dredis_port=${p.REDIS_PORT} clean install"
}
stage ('Push Docker Image') {
docker.withRegistry('https://registry-opstree.com', 'dockerhub') {
sh "docker build -t opstree/${p.SERVICE_NAME}:${BUILD_NUMBER} ."
sh "docker push opstree/${p.SERVICE_NAME}:${BUILD_NUMBER}"
}
}
stage ('Deploy') {
echo "We are going to deploy ${p.SERVICE_NAME}"
sh "kubectl set image deployment/${p.SERVICE_NAME} ${p.SERVICE_NAME}=opstree/${p.SERVICE_NAME}:${BUILD_NUMBER} "
sh "kubectl rollout status deployment/${p.SERVICE_NAME} -n ${p.ENVIRONMENT_NAME} "
}
}
You can see the logic easily here. The pipeline is checking if the developer wants to deploy on which environment what db_port needs to be there.
The benefits of this approach are many, some of them are as mentioned below:
Jenkinsfiles could actually just look more commonly, like this:
@Library ('opstree-library@master') _
opstreePipeline()
and opstreePipeline() would just read the the project type from pipeline.yaml and dynamically run the exact function, like opstreeStatefulPipeline(), opstreeStubsPipeline.groovy() . since pipeline are not exactly groovy, this isn’t possible. So one of the drawback is that each project would have to have a different-looking Jenkinsfile. The solution is in progress!So, what do you think?
Reference links:
Image: Google image search (jenkins.io)