Author: Sandeep Rawat

Gitolite

 

Requirement

We need private git repositories for internally use in our project so we use Gitolite for this requirement. Our client has a lot of consultants, partners and short term employees working with their code so they needed a good way of controlling access to the repos and preferably without giving each of them a unix user on the server where the repo is hosted.

What is Gitolite?

Gitolite is basically an access layer on top of Git. Users are granted access to repos via a simple config file and we as an admin only needs the users public SSH key and a username from the user. Gitolite uses this to grant or deny access to our Git repositories. And it does this via a git repository named gitolite-admin.

Installation

We need a public key and a Gitolite user through which we will setup the Gitolite.

In this case I have used my base machine(Ubuntu) public key so that only my machine can manage Gitolite.

Now we will copy this public key to a virtual machine

$ scp ~/.ssh/gitolite.pub [email protected]:/home/git

 

where vagrant is the user of my virtual machine & its IP is 192.168.0.20

Now we will install & create a gitolite user on remote machine which will be hosting gitolite.

root@git:~# apt-get install gitolite3
 
root@git:~# adduser gitolite
 
Now we need to remove password of gitolite user from below command
 
root@git:~# passwd -d gitolite
 
 
Let’s move & change the ownership of this public key.
root@git:~# mv gitolite.pub /home/gitolite/
root@git:~# chown gitolite:gitolite /home/gitolite/gitolite.pub
 
Become the gitolite user
 
root@git:~# su – gitolite
 
Now setup the gitolite with the public key
 
gitolite@git:~# gitolite setup -pk gitolite.pub
 
Now to manage the repositories, users and access-rights we will download the gitolite-admin(git repository) to our base machine.
 
$ git clone [email protected]:gitolite-admin
$ cd gitolite-admin
$ ls -l
total
8
drwxr-xr-x
2 nitin nitin 4096 Jan 10 17:52 conf/
drwxr-xr-x
2 nitin nitin 4096 Jan  9 13:43 keydir/
 
where “keydir” is the directory where we store our user’s keys and that key name must be same as existing username on the system.
 
In conf directory there is a “gitolite.conf” file which controls which repositories are available on the system and who has which rights to those repositories.
We just need to add new repository name & users who will access it and this file will create the repo & grant the permission on it accordingly.
Let us explore my gitolite.conf file in which I have added a new repository called “opstreeblog”
$ cat conf/gitolite.conf

# Group name & members

@admin = nitin
@staff    = jatin james
 
# Gitolite admin repository

repo gitolite-admin
RW+   = gitolite @admin
 
# Read-Write permission to all the users on testing repo

repo testing
RW+    = @all
 
# Read-Write permission to user sandy & the admin group. And Read-Only access to staff group

repo opstreeblog
   RW+   = sandy @admin
   R         = @staff

where ‘@’ denotes the user group i.e @staff is a group & jatin, james are the users of this group and these names must be similar to the key name stored in keydir directory.
For example “jatin” user must have the public key named “jatin.pub”

Let’s have a quick test of our setup

$ git commit conf/gitolite.conf -m “added opstreeblog repo”
 
[master 357bbc8] added “opstreeblog” repo
 
1 files changed, 9 insertions(+), 1 deletions(-)
 
nitin@Latitude-3460:~/gitolite-admin$ git push origin master
 
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 428 bytes, done.
Total
4 (delta 0), reused 0 (delta 0)
remote: Initialized empty Git repository in /home/gitolite/repositories/opstreeblog.git/
To gitbox:gitolite-admin d595439..357bbc8
master -> master
 
I hope that gives you a good overview of how to install and manage Gitolite.

Stunnel a Proxy to ship the log on SSL

Introduction

p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }

Few days ago I got a task to create the SSL connection with logstash redis plug-in with Azure Redis. As we are shipping the logs form the several data center to the Azure Redis. So logs must be shipped on SSL connection. There is no provision to create SSL connection through logstash redis plug-in thats why logstash redis plug-in is not able to make SSL connection with Azure redis.To resolve this problem we have to setup the stunnel as proxy front of the logstash redis plug-in. Stunnel can create SSL connection with Azure redis and Stunnel provide non-ssl connection for the logstash redis plug-in.

p { margin-bottom: 0.25cm; line-height: 120%; }a:link { Azure redis provide two type of connections, SSL on 6380 port and non SSL on 6379 port and also provide primary and secondary key(password).

Installation

Install Stunnel on the ubuntu.

$ sudo apt-get install stunnel 

Configuration

Create a configuration for stunnel /etc/stunnel/stunnel.conf and put the following lines into the configure file.

setuid = root
setgid = root
pid = /var/run/stunnel-azureredis.pid
debug = 7
output = /var/log/stunnel4/azureredis.log
options = NO_SSLv2
options = NO_SSLv3
[azureredis]
accept=127.0.0.1:6379
connect=:6380
client=yes
Timeout idle = 30

p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }

Restart the stunnel4
p { margin-bottom: 0.25cm; line-height: 120%; }a:link { }
$ service stunnel4 restart

Install Redis-cli to test the connection try to ping the redis azure it should reply as PONG

$ redis-cli -a
127.0.0.1:6379> ping
PONG

Now you can make SSL connection with azure redis on SSL.

Setup of Nginx Vhost

 

Introduction

This is an ancient way of setting up Vhost with nginx. As  we have Chef to automate everything. But before a kickoff with automation using chef, it’s crucial to interpret our problem statement by dealing with it manually.
 
“Choose Older Eggs For Hard Cooking Maria Simmons”

Problem Statement

NGINX is a free, open-source, high-performance HTTP server. Install nginx manually using package manager, and configure virtual host for opstree.com/blog/, chef.opstree.com.

Prerequisites

This exercise considers that you have a basic understanding of Git, and Vagrant. This blog deal with centos7.
 

Install Nginx

Clone our github repository and spin up a bare centos7 vagrant machine.
 
 
Go to nginxVhost directory.
 
$ cd  Chef/centos/nginxVhost
 
  • This directory have a Vagrantfile. Which can initiate a centos7 vagrant box with 512mb ram.
 
$ cat Vagrantfile
 
 
This file update and install some basic tools in your vagrant machine using vagrant shell provisioning.
 
  • Launch new vagrant machine and login into it via ssh.
$ vagrant up
 
$ vagrant ssh
 
  • Add nginx repo
As nginx is not available in default list of centos7, we add nginx repo to it.
 
$ sudo yum install -y epel-release
 
  • Install nginx
Install nginx using package manager “yum”.
 
$ sudo yum install -y nginx
 
  • Start nginx
Nginx do not start on its own. Type following to start it.
 
$ sudo service nginx start

Setup Vhost

Let’s go ahead with our problem statement of setting up vhost with nginx. This leads some dull steps to serve our webpages with opstree.com/blog/ and chef.opstree.com.
 
  • Replace nginx.conf file with given nginx.conf file.
 
$ sudo cp /vagrant/nginx.conf /etc/nginx/nginx.conf
 
  • Copy opstree.com/blog/ and chef.opstree.com into the /etc/nginx/conf.d directory
$ sudo cp /vagrant/opstree.com/blog/ /etc/nginx/conf.d/opstree.com/blog/
 
$ sudo cp /vagrant/chef.opstree.com /etc/nginx/conf.d/chef.opstree.com
 
  • Create home directory for vhost.
$ sudo mkdir /usr/share/nginx/blog
$ sudo mkdir /usr/share/nginx/chef
 
  • Create index files.
$ sudo su -c “echo \”Welcome, this is opstree.com/blog/\” > /usr/share/nginx/blog/index.html”
$ sudo su -c “echo \”Welcome, this is chef.opstree.com\” > /usr/share/nginx/chef/index.html”
 
  • Make entry in /etc/hosts
$ sudo vim /etc/hosts
 
127.0.0.1 opstree.com/blog/
127.0.0.1 chef.opstree.com
 
  • Restart nginx server
 
$ sudo service nginx restart
 
  • Access and test your  Vhost
 
$ curl opstree.com/blog/
 
$ curl  chef.opstree.com
You have done all the tiring stuff to set up Nginx Vhost. 
 
“Don’t let a bad day make you feel like you have a bad life.”
 
We understand your hard labor, so in next blog we go ahead to automate all the stuff.   

jgit-flow maven plugin to Release Java Application

Introduction

As a DevOps I need a smooth way to release the java application, so I compared two maven plugin that are used to release the java application and in the end I found that Jgit-flow plugin is far better than maven-release plugin on the basis of following points:
  • Maven-release plugin creates .backup and release.properties files to your working directory which can be committed mistakenly, when they should not be. jgit-flow maven plugin doesn’t create these files or any other file in your working directory.
  • Maven-release plugin create two tags.
  • Maven-release plugin does a build in the prepare goal and a build in the perform goal causing tests to run 2 times but jgit-flow maven plugin builds project once so tests run only once.
  • If something goes wrong during the maven plugin execution, It become very tough to roll it back, on the other hand jgit-flow maven plugin makes all changes into the branch and if you want to roll back just delete that branch.
  • jgit-flow maven plugin doesn’t run site-deploy
  • jgit-flow maven plugin provides option to turn on/off maven deployment
  • jgit-flow maven plugin provides option to turn on/off remote pushes/tagging
  • jgit-flow maven plugin keeps the master branch always at latest release version.
Now let’s see how to integrate Jgit-flow maven plugin and use it

How to use Jgit-flow maven Plugin for Release

Follow the flowing steps
    1. Add the following lines in your pom.xml for source code management access 
      
   scm:git:
  scm:git:git:
    2. Add these line to resolve the Jgit-flow maven plugin and put the other option that will be required during the build 
      com.atlassian.maven.plugins
maven-jgitflow-plugin
1.0-m4.3
        
true
false
true
true
true
true
true
true
          
master-test
deploy-test

Above code snippet will perform following steps:

    • Maven will resolve the jgitflow plug-in dependency
    • In the configuration section, we describe how jgit-flow plug-in will behave.
    • pushRelease XML tag to enable and disable jgit-flow from releasing the intermediate branches into the git or not.
    • keepBranch XML tag to enable and disable the plug-in for keep the intermediate branch or not.
    • noTag XMl tag to enable and disable the plug-in to create the that tag in git.
    • allowUntracked XML tag to whether allow untracked file during the checking.
    • flowInitContext XML tag is used to override the default and branch name of the jgit-flow plug-in
    • In above code snippet, there is only two branches, master from where that code will be pulled and a intermediate branch that will be used by the jgit-flow plug-in. as I have discussed that jgit-flow plug-in uses the branches to keep it records. so development branch will be created by the plug-in that resides in the local not remotely, to track the release version etc.
  2. To put your releases into the repository manager add these lines 
    <distributionManagement>
  <repository>
    <id><auth id></id>
    <url><repo url of repository managers></url>
  </repository>
  <snapshotRepository>
    <id><auth id></id>
    <url><repo url of repository managers></url>
  </snapshotRepository>
</distributionManagement>
  3. Put the following lines into your m2/settings.xml with your repository manager credentials 
    <settings>
  <servers>
    <server>
        <id><PUT THE ID OF THE REPOSITORY OR SNAPSHOTS ID HERE></id>
       <username><USERNAME></username>
       <password><PASSWORD></password>
    </server>
  </servers>
</settings>

Start Release jgit-flow maven plugin command

To start the new release execute jgitflow:release-start.

Finish Release jgit-flow maven plugin  command

To finish new release, execute mvn jgitflow:release-finish.
For a example I have created a repository in github.com. for testing and two branch master-test and deploy-test. It is assumed that you have configured maven and git your system.

In the deploy-test branch run following command
$ mvn clean -Dmaven.test.skip=true install jgitflow:release-start

This command will take input from you for release version and create a release branch with release/. then it will push this release branch into github repository for temporarily because we are not saving the intermediate branched

Now At the end run this command
$ mvn -Dmaven.test.skip=true jgitflow:release-finish
after finishing this command it will delete release/ from local and remote.

Now you can check the changes in pom file by jgitflow. in the above snapshot, it is master-test branch, you can see in the tag it has removed the snapshot and also increased the version.  It hold the current version of the application.

And in the deploy-test branch it show you new branch on which developers are working on

Kernel-based Virtualization Machine

What is virtualizatin?
“Virtualization is a technology that combines or divides computing resources to present one or many operating environments using methodologies like hardware and software partitioning or aggregation, partial or complete machine simulation, emulation, time-sharing, and many others.”

This means, that virtualization uses technology to abstract from the real hardware and provides isolated environments, so called Virtual Machines. They are capable to run various applications or even a whole operating system. A goal not mentioned in the definition is to have nearly to native performance for running VMS. This is a very important point, because the users always want to get the most out of their hardware. Most of them are not willing to introduce virtualiztion technology, if a huge amount of CPU power is wasted by managing VMs.

Kernel-based Virtual Machine :
KVM is the first virtualization solution that has been integration into the vanilla Linux kernel. KVM has been initially developed by Qumranet, a small company located in Isreal. Redhat acquired Qumranet in September 2008, when KVM become more production ready. They see KVM as next generation of virtualiztion technology.

KVM Architecture:
Linux has all the mechanism a VMM needs to operate several VMs. KVM is implemented as a kernel module that can be loaded to extend Linux by these capabilities.
In a normal Linux environment each process runs either in user-mode or in kernel-mode. KVM introduces a third, the guest-mode. Therefore it relies on a

virtualization capable CPU either Intel VT or AMD-SVM extensions. A process in guest-mode has its own kernel-mode and user-mode. Thus, it is able to run an opeating system. Such processes are representing the VMs running on a KVM host. What the modes are used for from a hosts point of view:

  • user-mode: I/O when guest needs to access devices.
  • kernel-node: switch into guest-mode and handle exits due to I/O operations
  • guest-mode: execute guest code, which is the guest OS except I/O

Resource Management:
To increase the reuse code as mush as possible they mainly modified the Linux memory management, to allow mapping physical memory into the VMs address space. Therefore they added shadow page tables, that were needed in the early days of x8 virtualization.
The scheduler of an operating system computers an order in that each process is assigned to one of the variable CPUs. In this way, all running process is assigned to one of the available CPUs. In this way, all running processes are share the computing time. Since the KVM developers wanted to reuse most of the mechanisms of Linux. They simply implemented each VM as a process, relying on its scheduler to assign computing power to the VMs.

The KVM control interface: 
Once the KVM kernel module has been loaded, the /dev/kvm device node appears in the file-system that represents the interface of KVM. It allows to control the hypervisor through a set of ioctls, commonly used in certain operating systems as an interface for processes running in user-mode to communicate with a driver. The ioctl() system call allows to execute several operations to create new virtual machines, assign memory to a virtual machine, assign and start virtual CPUs.

Emulation of Hardware:
To provide hardware like hard disks, cd drivers or networks  cards to the VMs, KVM uses a highly modified QEMU, This is a so called platform virtulization tool, which allows to emulate  an entire PC platform including graphics, net working, disk drives and many more.

Execution-Model:
Figure describe the execution model of KVM. This is the loop of the actions used to operate the VMs. These actions are separated by the three modes.

let see the which tasks are done in which mode:

  • user-mode: The KVM module is called using ioctl() to execute guest code until I/O operations initiated by the guest or an external event occurs. Such an event may be the arrival of a network package, which could be the reply of a network package sent by the host earlier. Such events are expressed as signals that leads to an interruption of guest code execution.
  • Kernel-mode: The kernel causes the hardware to execute guest code natively. IF the processor exits the guest due to pending memory or I/O operations, the kernel performs the necessary tasks and resumes the flow of execution. If external events such as signals or I/O operations initiated by the guest exists, it exits to the user-mode.
  • guest-mode: This is on the hardware level, where the extended instruction set of  virtualization capable CPU is used to execute the native code, until an instruction called that needs assistance by KVM, a fault or an external interrupt.

While a VM runs, There are plenty of switches between these modes.