Jenkins Pipeline Global Shared Libraries

When we say CI/CD as code, it should have modularity and reusability which results in Reducing integration problems and allowing you to deliver software more rapidly.

Jenkins Shared library is the concept of having a common pipeline code in the version control system that can be used by any number of pipelines just by referencing it. In fact, multiple teams can use the same library for their pipelines.

Our thought is putting all pipeline functions in vars is much more practical approach, while there is no other good way to do inheritance, we wanted to use Jenkins Pipelines the right way but it has turned out to be far more practical to use vars for global functions.

Practical Strategy
As we know Jenkins Pipeline’s shared library support allows us to define and develop a set of shared pipeline helpers in this repository and provides a straightforward way of using those functions in a Jenkinsfile.This simple example will just illustrate how you can provide input to a pipeline with a simple YAML file so you can centralize all of your pipelines into one library. The Jenkins shared library example:And the example app that uses it:

Directory Structure

You would have the following folder structure in a git repo:

└── vars
    ├── opstreePipeline.groovy
    ├── opstreeStatefulPipeline.groovy
    ├── opstreeStubsPipeline.groovy
    └── pipelineConfig.groovy

Setting up Library in Jenkins Console.

This repo would be configured in under Manage Jenkins > Configure System in the Global Pipeline Libraries section. In that section Jenkins requires you give this library a Name. Example opstree-library

Pipeline.yaml

Let’s assume that project repository would have a pipeline.yaml file in the project root that would provide input to the pipeline:Pipeline.yaml

ENVIRONMENT_NAME: test
SERVICE_NAME: opstree-service
DB_PORT: 3079
REDIS_PORT: 6079

Jenkinsfile

Then, to utilize the shared pipeline library, the Jenkinsfile in the root of the project repo would look like:

@Library ('opstree-library@master') _
opstreePipeline()

PipelineConfig.groovy

So how does it all work? First, the following function is called to get all of the configuration data from the pipeline.yaml file:

def call() {
  Map pipelineConfig = readYaml(file: "${WORKSPACE}/pipeline.yaml")
  return pipelineConfig
}

opstreePipeline.groovy

You can see the call to this function in opstreePipeline(), which is called by the Jenkinsfile.

def call() {
    node('Slave1') {

        stage('Checkout') {
            checkout scm
        }

         def p = pipelineConfig()

        stage('Prerequistes'){
            serviceName = sh (
                    script: "echo ${p.SERVICE_NAME}|cut -d '-' -f 1",
                    returnStdout: true
                ).trim()
        }

        stage('Build & Test') {
                sh "mvn --version"
                sh "mvn -Ddb_port=${p.DB_PORT} -Dredis_port=${p.REDIS_PORT} clean install"
        }

        stage ('Push Docker Image') {
            docker.withRegistry('https://registry-opstree.com', 'dockerhub') {
                sh "docker build -t opstree/${p.SERVICE_NAME}:${BUILD_NUMBER} ."
                sh "docker push opstree/${p.SERVICE_NAME}:${BUILD_NUMBER}"
            }
        }

        stage ('Deploy') {
            echo "We are going to deploy ${p.SERVICE_NAME}"
            sh "kubectl set image deployment/${p.SERVICE_NAME} ${p.SERVICE_NAME}=opstree/${p.SERVICE_NAME}:${BUILD_NUMBER} "
            sh "kubectl rollout status deployment/${p.SERVICE_NAME} -n ${p.ENVIRONMENT_NAME} "

    }
}

You can see the logic easily here. The pipeline is checking if the developer wants to deploy on which environment what db_port needs to be there.

Benefits

The benefits of this approach are many, some of them are as mentioned below:

  • How to write groovy code is now none of the developer’s perspective.
  • Structure of the Pipeline.yaml is really flexible, where entire data structures can be passed as input to the pipeline.
  • Code redundancy saved to a large extent.

 Jenkinsfiles could actually just look more commonly, like this:

@Library ('opstree-library@master') _
opstreePipeline()

and opstreePipeline() would just read the the project type from pipeline.yaml and dynamically run the exact function, like opstreeStatefulPipeline(), opstreeStubsPipeline.groovy() . since pipeline are not exactly groovy, this isn’t possible. So one of the drawback is that each project would have to have a different-looking Jenkinsfile. The solution is in progress!So, what do you think?

Reference links: 
Image: Google image search (jenkins.io)

What Without Internet

What without Internet?

I had a dream a few days ago in which the existence of the internet was gone, When I woke up I thought about what would happen if there is no Internet for a day?


Sure, it would cause quite a bit of panic and uproar and it would be havoc for an organization to work without the internet, but if the internet resumed normally after 24 hours are over, things would return to normal pretty quickly.


Now, switch it off for a longer time, possibly a week or a month, that would have a more lasting impact, since, in that time, a significant number of people would find themselves unable to meet their obligations or do their business at all. This would be somewhat mitigated by the fact that the situation is a sort of a ‘natural disaster’, but still, those who really depend on the internet for their business would likely feel a lasting negative impact.
               
What if I say there are some organizations that work in a situation like there is no internet, yes it’s right due to some sort of security reasons they don’t prefer to use the public internet. Banks, space organizations, and many security agencies fall under this category.


Now, a question arises here how they manage to do regular updates and the installation of different packages in their different systems? The answer is quite simple: “the use of satellite server”.


Recently I got a task in relation to this context, in which:

1. A prerequisite here is that you don’t have internet connectivity in your system but one of the systems with which you can connect has internet connectivity.
2. Setup individual satellite server in your local network.
3. Install packages and regular updates.


To do so here I prefer to use the FTP satellite server

How to implement Ftp satellite server

Pre-requisites

An Ubuntu Server, and a non-root user with sudo privileges.
The system is configured with vsftpd

Suppose we are doing the installation of Jenkins

Make a directory pkg.jenkins.io in /var/www/html/



 Contents of pkg.jenkins.io


 

 

 

 

 

 

 

 

 

 

 

Paste the host link of debian file in /etc/apt/source.list.d

 

 Run the command sudo apt-get update

 Now run the command for installation of the package

 


“ The internet made fame wack and anonymity cool ”

So, far from the above context, we have learned about setting up FTP for users with a local account. If you need to use an external authentication source, you might want to look into vsftpd’s support of virtual users. This offers a rich set of options through the use of PAM, the Pluggable Authentication Modules, and is a good choice if you manage users in another system such as LDAP or Kerberos.

I hope I explained everything clearly enough to understand. If you have some better way of implementing a satellite server please help me to improve this blog.

Thanks for reading my writing. I’d really appreciate any kind of feedback in the comments.

Cheers till next time!!!!

Can you integrate a GitHub Webhook with Privately hosted Jenkins No? Think again

Introduction

Triggering Jenkins builds automatically after every code commit is a core requirement in any continuous integration setup. Jenkins supports automated triggers through repository polling or event based notifications. While polling works, it consumes resources and introduces delays. Push based triggering through webhooks is far more efficient.

The difficulty appears when Jenkins is hosted inside a private network and the version control system is hosted on a cloud platform such as GitLab. In this scenario, GitLab cannot directly reach the Jenkins endpoint, making webhook based triggering difficult without exposing Jenkins publicly.

Webhook Relay solves this problem by acting as a secure bridge between GitLab and a privately hosted Jenkins server. This article explains how GitLab webhooks can trigger Jenkins jobs using Webhook Relay, based on real implementation experience.

Installing the Webhook Relay Agent

The Webhook Relay agent needs to run on the same machine where Jenkins is hosted or where Jenkins is reachable internally.

Below is the installation process shown as step based instructions.

# download the relay binary
curl -sSL https://storage.googleapis.com/webhookrelay/downloads/relay-linux-amd64 > relay
# make the binary executable
chmod +x relay

# move it to a directory in system path
sudo mv relay /usr/local/bin/relay

Webhook Relay service runs on a public endpoint, while this agent runs locally and listens for forwarded webhook events.

Creating a Webhook Relay Account

Create an account on the official Webhook Relay platform using the registration page shown below.

https://my.webhookrelay.com/register

After signing up, access to the Webhook Relay dashboard is provided, where authentication tokens can be generated.

Authenticating the Relay Agent

From the dashboard, create an access token. This generates a key and secret pair.

Use those credentials to authenticate the relay agent.

relay login \
-k
<your_token_key> \
-s <your_token_secret>

A successful login message confirms that the agent is connected and ready.

Creating the GitLab Repository

Create a GitLab repository for testing webhook integration. To keep the setup simple, a public repository can be used.

For reference, assume the repository name is WebhookProject.

Preparing Jenkins for GitLab Webhooks

Install the required Jenkins plugins from the plugin manager.

Navigate through the Jenkins dashboard to install
GitLab Plugin
GitLab Hook Plugin

Once installed, Jenkins becomes capable of receiving GitLab webhook events.

Creating the Jenkins Job

Create a new Jenkins job and configure it to pull source code from the GitLab repository.

Enable the option that allows Jenkins to be triggered by GitLab webhooks.

After enabling this option, Jenkins generates a webhook endpoint associated with the job. It usually follows this pattern.

http://<jenkins-host>:8080/project/<job-name>

Example shown for reference only.

Copy this endpoint, as it will be used in the forwarding configuration.

Forwarding Webhooks Using Webhook Relay

Start webhook forwarding by creating a relay bucket. This bucket acts as a routing channel between GitLab and Jenkins.

relay forward \
--bucket gitlab-jenkins \
http://<jenkins-host>:8080/project/<job-name>

Important note
Do not stop this process. Keep it running in the background.
Open a new terminal tab for further steps.

Once this command starts, the relay agent generates a public forwarding URL.

Configuring GitLab Webhook

Open the GitLab repository settings and navigate to the integrations or webhook section.

Paste the forwarding URL generated by Webhook Relay into the webhook URL field.

For initial testing, SSL verification can be disabled to avoid certificate related issues.

Save the webhook configuration.

Testing the Integration

Clone the GitLab repository locally and push a new commit.

git add .
git commit -m "test webhook trigger"
git push origin main

As soon as the push is completed, GitLab sends a webhook event. Webhook Relay receives it and forwards it to the local agent, which triggers the Jenkins job internally.

You can verify this by checking the Jenkins job build history.

Viewing Logs

GitLab webhook logs can be viewed from
Repository settings
Integrations
Webhook edit section

Webhook Relay logs are available in the Relay Logs section of the Webhook Relay dashboard.

Jenkins build logs confirm successful job execution.

Conclusion

Webhook Relay makes it possible to trigger Jenkins builds through GitLab webhooks even when Jenkins is hosted inside a private network. This approach avoids exposing Jenkins publicly while still enabling real time CI automation.

The same pattern works for GitHub and other webhook enabled platforms. With proper configuration, secure and efficient CI workflows can be achieved in restricted network environments.

The Reset Button !!!!

!!!! The Reset button !!!!

Anyone who has recently used the Google Compute Engine for creating the VM instances will be aware of the reset button available.

Since I wasn’t very much sure of it , I just clicked it without much know-how . This resulted in making all the servers to their original state as they were freshly build and which is certainly a very bad thing for us.

But , we had  puppet that we used to create the whole infrastructure as it is . All the modules we had used and changes we made were committed to GitHub repo and this certainly was a boon to us, else we have to sit whole day long for making those changes on the servers.

Just in couple of minutes the new  instances were created using the compute engine-create group instance feature. We did  installation of the foreman  and git on one of the servers and set up the  puppet clients agents accordingly . This took around 15 more crucial minutes and then cloned our GitHub repo which contains all the  necessary modules and configurations required for the rest of infrastructure.

These are the conditions where Configuration Management Tools like Puppet come in picture and help us get on the track in the shortest possible manner.
It was a hectic day but definitely made us learn several important aspects. Using puppet for maintain the infrastructure is really important now days. It is reliable,efficient and fast for deploying configurations on the servers  and making ready for the production work load.

Setup Jenkins using Ansible

In this document I’ll walk you through how you can setup Jenkins using Ansible.

Prerequisites

  •  OS – Ubuntu {at least two machine required in production}
  •  First machine for Ansible  installation
  •  Second machine where we will install jenkins server
  • You should have basic understanding of ansible workflow.
Note :  You should have password less login enabled in second machine. use this link 
http://www.linuxproblem.org/art_9.html

Ansible Installation
Before starting with installing Jenkins using Ansible, you need to have Ansible installed in your system.

$ curl https://raw.githubusercontent.com/OpsTree/AnsiblePOC/alok/scripts/Setup/setup_ansible.sh | sudo bash

Setup jenkins using Ansible
Install jenkins ansible roles
Once we have ansible installed in our system, we can start installing the jenkins using ansible. To install we will use an already available ansible role to setup jenkins

$ ansible-galaxy install geerlingguy.jenkins
to know more about the jenkins role hit this link https://galaxy.ansible.com/detail#/role/440

Ansible roles default directory path is /etc/ansible/roles
Make ansible playbook file

Now the next step is to use the installed jenkins roles to install the jenkins. For this purpose we will create a playbook  and hosts file with below content

$ cd ~/MyPlaybook/jenkins
create a file hosts and add below content
[jenkins_hosts]
192.168.33.15

Screenshot from 2015-11-30 12:55:41.png
Next create  a file site.yml and add below content
– hosts: jenkins_hosts
 roles:
     – { role: geerlingguy.jenkins }

Screenshot from 2015-11-30 12:59:08.png
so configuration file is done, the next step is to run ansible playbook command

$ ansible-playbook -i hosts site.yml

Now that Jenkins is running, go to http://192.168.33.15:8080. You’ll be welcome by the default Jenkins screen.