Build and Release Archives - DEVOPS DONE RIGHT.

Can you integrate a GitHub Webhook with Privately hosted Jenkins No? Think again

Introduction

Triggering Jenkins builds automatically after every code commit is a core requirement in any continuous integration setup. Jenkins supports automated triggers through repository polling or event based notifications. While polling works, it consumes resources and introduces delays. Push based triggering through webhooks is far more efficient.

The difficulty appears when Jenkins is hosted inside a private network and the version control system is hosted on a cloud platform such as GitLab. In this scenario, GitLab cannot directly reach the Jenkins endpoint, making webhook based triggering difficult without exposing Jenkins publicly.

Webhook Relay solves this problem by acting as a secure bridge between GitLab and a privately hosted Jenkins server. This article explains how GitLab webhooks can trigger Jenkins jobs using Webhook Relay, based on real implementation experience.

Installing the Webhook Relay Agent

The Webhook Relay agent needs to run on the same machine where Jenkins is hosted or where Jenkins is reachable internally.

Below is the installation process shown as step based instructions.

Webhook Relay service runs on a public endpoint, while this agent runs locally and listens for forwarded webhook events.

Creating a Webhook Relay Account

Create an account on the official Webhook Relay platform using the registration page shown below.

After signing up, access to the Webhook Relay dashboard is provided, where authentication tokens can be generated.

Authenticating the Relay Agent

From the dashboard, create an access token. This generates a key and secret pair.

Use those credentials to authenticate the relay agent.

A successful login message confirms that the agent is connected and ready.

Creating the GitLab Repository

Create a GitLab repository for testing webhook integration. To keep the setup simple, a public repository can be used.

For reference, assume the repository name is WebhookProject.

Preparing Jenkins for GitLab Webhooks

Install the required Jenkins plugins from the plugin manager.

Navigate through the Jenkins dashboard to install
GitLab Plugin
GitLab Hook Plugin

Once installed, Jenkins becomes capable of receiving GitLab webhook events.

Creating the Jenkins Job

Create a new Jenkins job and configure it to pull source code from the GitLab repository.

Enable the option that allows Jenkins to be triggered by GitLab webhooks.

After enabling this option, Jenkins generates a webhook endpoint associated with the job. It usually follows this pattern.

Example shown for reference only.

Copy this endpoint, as it will be used in the forwarding configuration.

Forwarding Webhooks Using Webhook Relay

Start webhook forwarding by creating a relay bucket. This bucket acts as a routing channel between GitLab and Jenkins.

Important note
Do not stop this process. Keep it running in the background.
Open a new terminal tab for further steps.

Once this command starts, the relay agent generates a public forwarding URL.

Configuring GitLab Webhook

Open the GitLab repository settings and navigate to the integrations or webhook section.

Paste the forwarding URL generated by Webhook Relay into the webhook URL field.

For initial testing, SSL verification can be disabled to avoid certificate related issues.

Save the webhook configuration.

Testing the Integration

Clone the GitLab repository locally and push a new commit.

As soon as the push is completed, GitLab sends a webhook event. Webhook Relay receives it and forwards it to the local agent, which triggers the Jenkins job internally.

You can verify this by checking the Jenkins job build history.

Viewing Logs

GitLab webhook logs can be viewed from
Repository settings
Integrations
Webhook edit section

Webhook Relay logs are available in the Relay Logs section of the Webhook Relay dashboard.

Jenkins build logs confirm successful job execution.

Conclusion

Webhook Relay makes it possible to trigger Jenkins builds through GitLab webhooks even when Jenkins is hosted inside a private network. This approach avoids exposing Jenkins publicly while still enabling real time CI automation.

The same pattern works for GitHub and other webhook enabled platforms. With proper configuration, secure and efficient CI workflows can be achieved in restricted network environments.

jgit-flow maven plugin to Release Java Application

Introduction

As a DevOps I need a smooth way to release the java application, so I compared two maven plugin that are used to release the java application and in the end I found that Jgit-flow plugin is far better than maven-release plugin on the basis of following points:

Maven-release plugin creates .backup and release.properties files to your working directory which can be committed mistakenly, when they should not be. jgit-flow maven plugin doesn’t create these files or any other file in your working directory.
Maven-release plugin create two tags.
Maven-release plugin does a build in the prepare goal and a build in the perform goal causing tests to run 2 times but jgit-flow maven plugin builds project once so tests run only once.
If something goes wrong during the maven plugin execution, It become very tough to roll it back, on the other hand jgit-flow maven plugin makes all changes into the branch and if you want to roll back just delete that branch.
jgit-flow maven plugin doesn’t run site-deploy
jgit-flow maven plugin provides option to turn on/off maven deployment
jgit-flow maven plugin provides option to turn on/off remote pushes/tagging
jgit-flow maven plugin keeps the master branch always at latest release version.

Now let’s see how to integrate Jgit-flow maven plugin and use it

How to use Jgit-flow maven Plugin for Release

Follow the flowing steps

Add the following lines in your pom.xml for source code management access
```
   scm:git:
  scm:git:git:
```

Add these line to resolve the Jgit-flow maven plugin and put the other option that will be required during the build

com.atlassian.maven.plugins
maven-jgitflow-plugin
1.0-m4.3
        
true
false
true
true
true
true
true
true
          
master-test
deploy-test

Above code snippet will perform following steps:

- Maven will resolve the jgitflow plug-in dependency
- In the configuration section, we describe how jgit-flow plug-in will behave.
- pushRelease XML tag to enable and disable jgit-flow from releasing the intermediate branches into the git or not.
- keepBranch XML tag to enable and disable the plug-in for keep the intermediate branch or not.
- noTag XMl tag to enable and disable the plug-in to create the that tag in git.
- allowUntracked XML tag to whether allow untracked file during the checking.
- flowInitContext XML tag is used to override the default and branch name of the jgit-flow plug-in
- In above code snippet, there is only two branches, master from where that code will be pulled and a intermediate branch that will be used by the jgit-flow plug-in. as I have discussed that jgit-flow plug-in uses the branches to keep it records. so development branch will be created by the plug-in that resides in the local not remotely, to track the release version etc.

To put your releases into the repository manager add these lines

<distributionManagement>
  <repository>
    <id><auth id></id>
    <url><repo url of repository managers></url>
  </repository>
  <snapshotRepository>
    <id><auth id></id>
    <url><repo url of repository managers></url>
  </snapshotRepository>
</distributionManagement>

Put the following lines into your m2/settings.xml with your repository manager credentials

<settings>
  <servers>
    <server>
        <id><PUT THE ID OF THE REPOSITORY OR SNAPSHOTS ID HERE></id>
       <username><USERNAME></username>
       <password><PASSWORD></password>
    </server>
  </servers>
</settings>

Start Release jgit-flow maven plugin command

To start the new release execute jgitflow:release-start.

Finish Release jgit-flow maven plugin command

To finish new release, execute mvn jgitflow:release-finish.
For a example I have created a repository in github.com. for testing and two branch master-test and deploy-test. It is assumed that you have configured maven and git your system.

In the deploy-test branch run following command
$ mvn clean -Dmaven.test.skip=true install jgitflow:release-start

This command will take input from you for release version and create a release branch with release/. then it will push this release branch into github repository for temporarily because we are not saving the intermediate branched

Now At the end run this command
$ mvn -Dmaven.test.skip=true jgitflow:release-finish
after finishing this command it will delete release/ from local and remote.

Now you can check the changes in pom file by jgitflow. in the above snapshot, it is master-test branch, you can see in the tag it has removed the snapshot and also increased the version. It hold the current version of the application.

And in the deploy-test branch it show you new branch on which developers are working on

Opstree SHOA Part 1: Build & Release

At Opstree we have started a new initiative called SHOA, Saturday Hands On Activity. Under this program we pick up a concept, tool or technology and do a hands on activity. At the end of the day whatever we do is followed by a blog or series of blog that we have understood during the day.

Since this is the first Hands On Activity so we are starting with Build & Release.

What we intend to do

Setup Build & Release for project under git repository https://github.com/OpsTree/ContinuousIntegration.

What all we will be doing to achieve it

Finalize a SCM tool that we are going to use puppet/chef/ansible.
Automated setup of Jenkins using SCM tool.
Automated setup of Nexus/Artifactory/Archiva using SCM tool.
Automated setup of Sonar using SCM tool.
Dev Environment setup using SCM tool: Since this is a web app project so our Devw443 environment will have Nginx & tomcat.
QA Environment setup using SCM tool: Since this is a web app project so our QA environment will have Nginx & tomcat.
Creation of various build jobs
- Code Stability Job.
- Code Quality Job.
- Code Coverage Job.
- Functional Test Job on dev environment.
Creation of release Job.
Creation of deployment job to do deployment on Dev & QA environment.

This activity is open for public as well so if you have any suggestion or you want to attend it you are most welcome

Revert a patch in most awesome way

If you are a Release Engineer, System Admin or Support Engineer you have definitely come across a requirement where you have to apply patches to the target systems be it production or non-production. I’m assuming that you are using some automated system to manage the patches i.e applying them and reverting them. In this blog I would be discussing about the standard way of patch management and how you can have an out of the box solution to revert your patch in most simplistic way and without much fuss. At the end of the blog I would like to see an expression where you will say what the hell it’s so awesome yet so simple :).

People usually use some tool to apply patch to a target system which in addition to applying a patch also manage the history the patches so that it can be reverted in case the patch goes wrong. The patch history usually contains below details:

The new files that were added in the patch, while reverting the patch those files should be deleted.
The files that were deleted by the patch, while reverting the patch the deleted files should be restored back.
The files that were modified by the patch, while reverting the patch the modified files should be restored back.

You can definitely create a tool that can revert the patch for you as the use cases are not much, but do you really need to put this much effort if you can have an out of the box solution for this. What if I tell you that we use git for managing our patch history and reverting them. As git comes with a local repository concept so we created a local git repository at our app server codebase location only. Git comes with all the file level tracking we map each patch with one git commit, so at the time of reverting a specific patch you can ask git to simply revert the commit for you.

Extra steps to be done after applying patch:
To make git track the changes done in patch, you just need to perform 2 extra commands

git add . : This command will track all the files that have been modififed, added or deleted in the system.
git commit -m “Applying Patch” : This command actually adds the files information tracked by previous command with a message in the git system

Steps to be done in reverting changes done by a patch:
Once you have all the information tracked in git it will become no-brainer to revert the patches.

To view the details of all the patches: You can use git log command that will provide you the list of all the patches that you have applied or reverts that you have done

sandy@sandy:~/test/app1$ git log commit f622f1f97fc44f6897f9edc25f9c6aab8e425049 Author: sandy Date: Thu Jun 19 15:19:53 2014 +0530

Patch 1 on release2

commit 9a1dd81c7799c2f83d897eed85914eecef304bf0
Author: sandy
Date: Thu Jun 19 15:16:52 2014 +0530

Release 2

commit 135e04c00b3c3d5bc868f7774a5f284c3eb8cb29
Author: sandy
Date: Thu Jun 19 15:16:28 2014 +0530

Release 1

Now Reverting a patch is as simple as executing a simple command git revert, with the commit id of the patch

git revert f622f1f97fc44f6897f9edc25f9c6aab8e425049 [master 0ba533f] q Revert "Patch 1 on release2" 1 file changed, 1 deletion(-)

If you run git log command, you will see the patch revert history as well

sandy@sandy:~/test/app1$ git log commit 0ba533fda95ed4d7fcf0b7e6b23cd1a5589946a7 Author: sandy Date: Thu Jun 19 15:20:24 2014 +0530

Revert "Patch 1 on release2"

This reverts commit f622f1f97fc44f6897f9edc25f9c6aab8e425049.commit f622f1f97fc44f6897f9edc25f9c6aab8e425049
Author: sandy
Date: Thu Jun 19 15:19:53 2014 +0530

Patch 1 on release2

commit 9a1dd81c7799c2f83d897eed85914eecef304bf0
Author: sandy
Date: Thu Jun 19 15:16:52 2014 +0530

Release 2

commit 135e04c00b3c3d5bc868f7774a5f284c3eb8cb29
Author: sandy
Date: Thu Jun 19 15:16:28 2014 +0530

Release 1

I hope this blog has given you a very different perspective of managing the patches, let me know your thoughts about this. Also if you have such more ideas do share with me.

A wrapper over linode python API bindings

Recently I’ve been working on automating the nodes creation on our Linode infrastructure, in the process I came across the Linode API and it’s bindings. Though they were powerful but lacks at some places i.e:

In case of Linode CLI, while creating a linode you have to enter the root password so you can’t achieve full automation. Also I was not able to find an option to add private ip to the linode
In case of Linode API python binding you can’t straight away create a running linode machine.

Recently I’ve launched a new GitHub project, this project is a wrapper over existing python bindings of linode and will try to ease out the working with linode api. Currently using this project you can create a linode with 3 lines of code
from linode import Linode
linode=Linode(‘node_identifier’)
linode.create()

You just need to have a property file,/data/linode/linode.properties:

[DEFAULT]
UBUNTU_DIST=Ubuntu 12.04
KERNEL_LABEL=Latest 64 bit
DATACENTER_LABEL=Dallas
PLAN_ID=1024
ROOT_SSH_KEY=
LINODE_API=

The project is still in development, if someone wants to contribute or have any suggestions you are most welcome.