Puppet module to setup nodejs deployment

I would like to share my puppet module to setup nodejs deployment infrastructure on a linux box. This module performs the basic setup required to facilitate the automated deployment of a nodejs app. Very soon I’ll be introducing another generic puppet module that will run on top of this module & provide a full fledged automatic deployment of any node app. To view the source code of this module you can refer my github repository.

Let’s talk about what this module actually does. First of all we create a nodejs user which we will use for all deployment related activities of all the node app’s, as a convention we have created a folder /home/nodejs/nodeapps this folder will contain all the code of our node applications.

This modules adds 2 scripts as well the first one is deployNodeApp.sh, deployNodeApp.sh is a generic script that assumes that node app code will be present in tar form at /home/nodejs it will clean existing code of nodeapp at /home/nodejs/nodeapps untar the code at corresponding directory of node app & restart the node app. As another convention we are using upstart for managing the node app i.e starting & stopping the node app I’ll talk about the upstart configuration in my next blog where I’ll talk about generic puppet module for a node app. Another script startNodeApp.sh will take care of starting the node app after doing some per-processing such as loading some environment specific properties of node app which we don’t want to commit in the codebase i.e want to separate it out from deployment process choosing a specific version of node.

This module also takes cares of installing nvm for nodejs user so that nodejs version can be managed locally for this user or app.

Though we already have a puppet module for nodejs, but I had some specific requirements which I wanted to handle that’s why I’ve created this module.

Let me know if you have some points of improvement in this module, one thing that I wanted to add in this module is to add npm installation but it had some other dependencies also I had some doubts whether I should have npm as part of nodejs module or not.

System Monitoring

One of the main task of a system administrator is system monitoring, system monitoring usually involves monitoring the ram & disk space usage of the system …. In this blog I’ll be talking about my experience as a system admin & how I do it.

Usually system monitoring is divided into 2 parts Continuous system monitoring and troubleshooting system issues when system crosses a threshold value & you have to figure out the issue & try to resolve it.

In continuous system monitoring a system is put under continuous monitoring i.e the system ram usage is within defined limit or not, the disk space occupancy should not cross a predefined threshold …. To achieve continuous monitoring you can use couple of tools available in market such as nagios, omd we are primarily using these tools their would be other tools available also for this purpose.

Continuous system monitoring serves one purpose where they notify about any deviation from the expected state of the system, the next step is to troubleshoot this issue & resolve it accordingly. As a first step I usually execute top command, top is a very powerful command apart from just viewing the processes activity in real you can do a lot of things i.e

  • If you want to add/remove fields : press f & then you can choose the fields to add/remove
  • If you want to change ordering of  fields : press O & then you can move fields
  • If you want to change the sort order : press F or O
there are lot of other options available as well, if you want to explore them pressing h will provide you a list of all the options.

You can also read about htop, htop is an advanced form of top where you can view some graphs as well though I’ven’t used htop so much but I’m planning to 🙂

One thing to note sometimes you are not able ot run top command due to high resource utilization, in that case you have to use cat /proc/loadavg to view the load on the system & cat /proc/meminfo to view current memory state of the system.

One of the useful command if top doesn’t work
ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5
This command will give the top 5 processes by memory usage.

Also there are couple of other commands that you can use
free : To view the memory usage of system
df : To view the file system information
du : To view the disk usage

One tip : To increase the memory of system you can create a swap memory & it is always recommended to create a swap on a partition only. Another best practice for swap area is if your system RAM is below 8 gb your swap area should be double of your ram otherwise it should be half of your RAM size

Tip : Setting up Git Jenkins integration on windows box

If you have ever tried setting up git as a version control system in a Jenkins installation on a windows box you would have faced an error message ssh key not available.

The reason behind this issue is that if you are using git with ssh protocol it tries to use your private key to perform git operations over ssh protocol & the location it expects is the .ssh folder at home directory of user. To fix this issue you have to create a HOME environment variable and point to your home directory where your .ssh folder exists after that restart Jenkins & now it should work fine.

Linux Utility to manage login to systems

One of the problem I used to have as build & release engineer is to manage login to huge number of boxes through my Linux system. At the scale of 5-10 machine it’s a not a big problem but once you have close to 100+ boxes then it is not humanly possible to remember the ip’s of those boxes.
The usual approach for this problem is to maintain a reference file, from where you map machine name with the ip & find the ip of the box from this file, but again after some time this solution seems to be not that efficient. Another solution is to have a DNS server where you can store such mappings & then you can access these machines using their names only, this is the idle solution but what if you don’t have DNS server also still you have to execute the ssh command ‘ssh user@machine”.

I developed a simple solution for this problem, I created a utility script connect.sh, this script takes machine name as an argument & then we have multiple conditions statements which checks which ssh command to be executed for the machine name.

#!/bin/bash
if [ “mc1” == $1 ]; then
    ssh user@
elif [ “mc2” == $1 ]; then
    ssh user@
elif [ “mc3” == $1 ]; then
    ssh user@
.
.
.
fi

This solution worked really well for me as now I’m saved from executing whole ssh command, also for machine name I’ve followed a convention i.e _ for example the entry for a machine for release environment that is hosting an application catalog the machine name would be release_catalog, similarly dev_catalog, staging_catalog, pt_catalog.. so you don’t have to remember machine names as well :).

Automated DB Updater Release First Release

Initial version of Automated DB Updater Release ADU

With this blog I’m releasing the intial version of a python utility to provide automated db updates across various environments for different components.

The code for this utility is hosted on github
https://github.com/sandy724/ADU

You can clone the read only copy of this codebase by url given below
https://github.com/sandy724/ADU.git

To understand the basic idea about this utility go thorugh this blog
http://sandy4blogs.blogspot.in/2013/07/automated-db-updater.html

How to use this utility
Checkout the code at some directory, add the path of this directory in PYTHONPATH environment variable
Create a database with a script’s metadata table with given below ddl

CREATE TABLE `script_metadata` (
  `name` varchar(100) DEFAULT NOT NULL,
  `version` int(11) DEFAULT NOT NULL,
  `executed` tinyint(1) NOT NULL DEFAULT ‘0’,
  `env` varchar(30) DEFAULT NOT NULL,
  `releas` varchar(30) DEFAULT NOT NULL,
  `component` varchar(30) DEFAULT NOT NULL
)
Create a database.properties, containing connection properties of each environment database

[common_db]
dbHost=localhost
dbPort=3306
dbUser=root
dbPwd=root
db=test
 
 
[env1]
dbHost=localhost
dbPort=3306
dbUser=root
dbPwd=root
db=test

Here common_db represents connection to database which will contain metadata of scripts for monitoring

Now execute the pythong utility
Copy the client(updateDB.py) to directory of your choice, make sure that property configration file should also be at this directory
python updateDB.py -f -r –env