Chef-Resources Easy as pie..

 

Introduction

Resources are the basal element of chef’s heap. In all the functioning of chef, resources are in crux. These are a very first statement of any chef recipe or cookbook. In this blog we only intensify on the practical approach of  working with resources. This article will help you to compose a primitive awareness towards the execution of chef.
 
“Pie Makes Everybody Happy!! L. H. Anderson”

Prerequisites

To follow this article you need a prior information about Git and Vagrant. This blog uses centos7   as platform.

Getting Started

Sooner, probably but it necessitous to talk about chef installation. For this and some of next blogs we use chef-standalone mode. Chef-standalone mode provides a basic learning environment for beginners. Chef-standalone mode allow you to configure a machine directly.
Chef cater a complete  package labeled as Chefdk (Chef Development Kit ). It encloses all the essential thing you need to start with chef.
 

Install Chefdk

Clone our git repo and spin up a vagrant box with it.
 
$ git clone git@github.com:OpsTree/Chef.git
 
  • Go to chefResources directory. This directory contains a Vagrantfile, which can launch a centos7 vagrant machine with Chefdk and other essential tools installed.
$ cat Vagrantfile
 
  • Launch a new vagrant machine and login into it.
$ vagrant up
 
$ vagrant ssh
 
Your learning environment is ready. Let’s start with chef resources.

Resource

Lets took our first resource package. As in our blog series we go through a common problem statement of installing nginx and then setup nginx vhost. This time we are going to do this with chef resources only. Chef resource have basic two parts properties and action. We will discuss both of these in our next article. Till then we start playing with  chef resources.
 
  • Package resource
Package resource is used to manage packages. This is the base resource for other package management resources for different platforms.
 
    • Add nginx repo by installing epel-release.
$ sudo chef-apply -l info -e “package ‘epel-release'”
 
    • Install nginx using package resource
$ sudo chef-apply -l info -e “package ‘nginx'”
 
  • Directory resource
Directory resource is responsible to maintain the directory lane in target node.This resource have multiple attributes to classify the permissions, owner and group.
 
    • Create home directory for vhosts.
$ sudo chef-apply -l info -e “directory ‘/usr/share/nginx/blog'”
$ sudo chef-apply -l info -e “directory ‘/usr/share/nginx/chef'”
 
This will create two directories blog and chef under ‘/usr/share/nginx directory’.

  • File resource
This resource is answerable for the file management on the node. This resource holds mode, content, owner, group and path attributes with their respective meaning for a file.
 
    • Create index files for nginx vhosts.
$ sudo chef-apply -l info -e “file ‘/usr/share/nginx/blog/index.html’ do content ‘Hello from blog server’ end”
 
$ sudo chef-apply -l info -e “file ‘/usr/share/nginx/chef/index.html’ do content ‘Hello from chef server’ end”
 
File resource creates a index.html file under chef and blog directories with the respective content available in resource.
 
    • Create opstree.com/blog/.conf  and chef.opstree.com.conf into the /etc/nginx/conf.d directory with files available for this.
$ sudo chef-apply -l info -e “file ‘/etc/nginx/conf.d/opstree.com/blog/.conf’ do content IO.read(‘/vagrant/resources/opstree.com/blog/.conf’) end”
 
$ sudo chef-apply -l info -e “file ‘/etc/nginx/conf.d/chef.opstree.com.conf’ do content IO.read(‘/vagrant/resources/chef.opstree.com.conf’) end”
 
This file resource creates vhost configuration files for opstree.com/blog/ and chef.opstree.com. This time file resource uses a ruby function IO.read to read the content of a sample file and paste them into target file.
 
  • Make entry in /etc/hosts
    • This is possible to make these entry with chef but it is a little complex for this time as we are not so proficient with chef. So we are doing this manually for now, but in our next article we will do this with chef.
$ sudo vim /etc/hosts
127.0.0.1 opstree.com/blog/
127.0.0.1 chef.opstree.com
 
  • Service resource
Service resource is used to manage services. Use this resource to restart the nginx service.
 
    • Restart your nginx server to make changes effective.
 
$ sudo chef-apply  -e “service ‘nginx’ do action [:stop, :start] end”
 
This resource has two defied actions ie. stop and start which run in defined order to stop and then start nginx. You can use restart action of service resource but it is a good habit to use stop and start.
 
You can find all these resources in our git repo under “Chef/centos/chefResources/resources” directory. Just run the all “.sh” files to run our resources.
 
So now you start sensing the power of chef and also aware with the basics of resources. We will next come up with the theory behind the resources  and some new exciting examples in our blog.
“The Good NEWS Is You’re The Pilot. Michael Altshuler”
So stay great as Chef.

Snoopy + ELK : Exhibit sudo commands in Kibana Dashboard

Logging User Commands: Snoopy Logger

About Snoopy Logger

Snoopy logs all the commands that are ran by any user to a log file. This is helpful for auditing and keep an eye on user activities.

Automated Installation

For Automated Installation/Configuration of Snoopy we have created a Puppet module and Ansible Role.

Manual Installation

To install the latest STABLE version of Snoopy, use these commands:
rm -f snoopy-install.sh
wget -O snoopy-install.sh https://github.com/a2o/snoopy/raw/install/doc/install/bin/snoopy-install.sh
chmod 755 snoopy-install.sh
./snoopy-install.sh stable

Output

This is what typical Snoopy output looks like:
2015-02-11T19:05:10+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/cat]: cat /etc/fstab.BAK
2015-02-11T19:05:15+00:00 labrat-1 snoopy[896]: [uid:0 sid:11679 tty:/dev/pts/2 cwd:/root filename:/usr/bin/rm]: rm -f /etc/fstab.BAK
These are default output locations on various Linux distributions:

  • CentOS: /var/log/secure
  • Debian: /var/log/auth.log
  • Ubuntu: /var/log/auth.log
  • others: /var/log/messages (potentially, not necessarily)

For actual output destination check your syslog configuration.
Snoopy provides a configuration file “/etc/snoopy.ini” where you can configure snoopy to generate logs. By default snoopy logs only uid, but doesn’t logs username in logs, so we have to change configuration to get username in logs.You may also specify the log path where you want to generate the snoopy logs.
For getting username in logs edit “/etc/snoopy.ini” and under [snoopy] section add the following line:
message_format = “[username:%{username} uid:%{uid} sid:%{sid} tty:%{tty} cwd:%{cwd} filename:%{filename}]: %{cmdline}”
The output of logs is  shown below:
Feb 25 07:47:27 vagrant-ubuntu-trusty-64 snoopy[3163]: [username:root uid:0 sid:1828 tty:/dev/pts/0 cwd:/root filename:/usr/bin/vim]: vim /etc/snoopy.ini

Enable/Disable Snoopy

To enable snoopy, issue the following command:
snoopy-enable
To disable snoopy, issue the following command:
snoopy-disable

Using ELK to parse logs

Now that we have logs with suitable information we will write a grok pattern in logstash to parse these logs and generate required fields.
A sample grok pattern will be like this:


filter {

 if [type] == “snoopy” {
   grok {
     match => { “message” => “%{SYSLOGTIMESTAMP:date} %{HOSTNAME:hostname} %{WORD:logger}\[%{INT}\]\: \[%{WORD}\:%{USERNAME:username} %{DATA} %{DATA} %{DATA} %{WORD}\:%{DATA:cwd} %{DATA}\]\: %{GREEDYDATA:exe_command}” }
   }
 if “_grokparsefailure” in [tags] {
   drop { }
 }
 }
}

Here we are generating these fields:
date: Timestamp at which log is generated
hostname: Name of host
logger: Name of logger which is generating logs in our case “snoopy”.
username: Name of user issuing the command
cwd: Absolute path of directory from where the command is executed
exe_command: Command that is executed by user with complete options

Place the above grok pattern in filter section of logstash configuration file which is at “/etc/logstash/conf.d/logstash.conf”. Also include logs from “/var/log/auth.log” to be shipped to logstash server from logstash agent at the client.

Creating Dashboard in Kibana

After that you can see these logs in kibana in “Discover” tab as shown in screenshot:

elkdiscover.png

In the left sidebar you can see all the fields via which you can filter including the fields we set in our grok pattern.Now in the search bar you can search according to specific field and its value. For example to search logs for vagrant user and all sudo commands executed by it, you will write the following query in search bar:
username:vagrant AND exe_command:sudo*
Then from the left sidebar add the fields you want to see, for example add “username”, “exe_command” and “cwd”, which will result to a table as shown below:

elktableselectedfields.png

Now save this search from the icon that is just adjacent to left bar with a suitable name. Then go to “Dashboard” menu and click on “plus” icon to add a dashboard. A screen will appear as shown:

adddashboard.png

Click on “Searches” tab and find your saved search and click over it. A resulting screen will appear which will be added to your dashboard as shown below:

dashboardadded.png

Here you can view tabular data for the sudo commands executed by vagrant user. Similarly you can add more searches by clicking on “plus icon” and add it to the same dashboard.Now save this dashboard by clicking on the “save” icon adjacent to search bar with a suitable name.After that you can easily load this dashboard by clicking on “load” icon adjacent to search bar.

Snoopy : Get, Set and log

Recently we got a requirement to log each and every command executed by the users. Upon googling we found a tool that is  exclusively contrived to accomplish this ambition.

This tool was “Snoopy”. This is a open source project whose microscopic documentation is available at their github page:

 

You can find all installation and configuration doc at their github README.

Why snoopy?

We choose snoopy because it is a very lightweight  tool that is just built with the intention  of logging each and every command executed by user. It consists of only a tiny library which does all the logging.

Use cases:

1). Log each executed command by any user with its arguments
2). Configure the pattern of logs according to requirement using a simple configuration file
3). The logs generated by snoopy can be sent to a central logging server and then can be parsed to get useful information. For example: To get all sudo commands executed by a user.

About Snoopy:

Snoopy logger is a majestic utility which makes the admin work more effortless by providing a log of commands with its arguments executed via shell by any user. It comes with a configuration file “/etc/snoopy.ini” where you can configure how the logs are generated and the information in the logs you want to have.

Automating Snoopy                            installation/configuration:

We will make the procedure easy for installation and configuration of snoopy by creating puppet module and ansible role. We’ll soon be launching a puppet module and an ansible role for it.

 

Also we’ll showcase a useful example of snoopy logger with centralized logging using ELK.

Setup of Nginx Vhost

 

Introduction

This is an ancient way of setting up Vhost with nginx. As  we have Chef to automate everything. But before a kickoff with automation using chef, it’s crucial to interpret our problem statement by dealing with it manually.
 
“Choose Older Eggs For Hard Cooking Maria Simmons”

Problem Statement

NGINX is a free, open-source, high-performance HTTP server. Install nginx manually using package manager, and configure virtual host for opstree.com/blog/, chef.opstree.com.

Prerequisites

This exercise considers that you have a basic understanding of Git, and Vagrant. This blog deal with centos7.
 

Install Nginx

Clone our github repository and spin up a bare centos7 vagrant machine.
 
$ git clone git@github.com:OpsTree/Chef.git
 
Go to nginxVhost directory.
 
$ cd  Chef/centos/nginxVhost
 
  • This directory have a Vagrantfile. Which can initiate a centos7 vagrant box with 512mb ram.
 
$ cat Vagrantfile
 
 
This file update and install some basic tools in your vagrant machine using vagrant shell provisioning.
 
  • Launch new vagrant machine and login into it via ssh.
$ vagrant up
 
$ vagrant ssh
 
  • Add nginx repo
As nginx is not available in default list of centos7, we add nginx repo to it.
 
$ sudo yum install -y epel-release
 
  • Install nginx
Install nginx using package manager “yum”.
 
$ sudo yum install -y nginx
 
  • Start nginx
Nginx do not start on its own. Type following to start it.
 
$ sudo service nginx start

Setup Vhost

Let’s go ahead with our problem statement of setting up vhost with nginx. This leads some dull steps to serve our webpages with opstree.com/blog/ and chef.opstree.com.
 
  • Replace nginx.conf file with given nginx.conf file.
 
$ sudo cp /vagrant/nginx.conf /etc/nginx/nginx.conf
 
  • Copy opstree.com/blog/ and chef.opstree.com into the /etc/nginx/conf.d directory
$ sudo cp /vagrant/opstree.com/blog/ /etc/nginx/conf.d/opstree.com/blog/
 
$ sudo cp /vagrant/chef.opstree.com /etc/nginx/conf.d/chef.opstree.com
 
  • Create home directory for vhost.
$ sudo mkdir /usr/share/nginx/blog
$ sudo mkdir /usr/share/nginx/chef
 
  • Create index files.
$ sudo su -c “echo \”Welcome, this is opstree.com/blog/\” > /usr/share/nginx/blog/index.html”
$ sudo su -c “echo \”Welcome, this is chef.opstree.com\” > /usr/share/nginx/chef/index.html”
 
  • Make entry in /etc/hosts
$ sudo vim /etc/hosts
 
127.0.0.1 opstree.com/blog/
127.0.0.1 chef.opstree.com
 
  • Restart nginx server
 
$ sudo service nginx restart
 
  • Access and test your  Vhost
 
$ curl opstree.com/blog/
 
$ curl  chef.opstree.com
You have done all the tiring stuff to set up Nginx Vhost. 
 
“Don’t let a bad day make you feel like you have a bad life.”
 
We understand your hard labor, so in next blog we go ahead to automate all the stuff.   

Chef Start here with ease..

 

 

 

Introduction

 

“Until I discovered cooking, I was never really interested in anything. Julia Child”

Chef, the lead in automation industry has many tickling facet and calibre. Before introducing the potentials of “The Chef”, it’s non negotiable to evade the foresight of its relevance to devops exercises. Chef can take care of server automation, infrastructure environment and continuously deliver your application.

 

Motive behind this array

 

With this blog series, we will familiarize you with the concepts of chef and will try to make you comfortable with our hands on blogs. This series of blog contains 15 blogs in a row which will enhance the knowledge and draw your faith in chef.

 

“Always Pre-Heat The Oven Before Putting The Meat In !!”

 

Prerequisites

 

For all the upcoming blogs we presume that you have basic understanding of Git, Docker,Vagrant and Linux. This blog series is written in consideration with centos as platform, although you can apply them on ubuntu by following some minor changes.

 

 

 

We are going to use our public git repository for all the blogs in this series. We will be using centos7 vagrant box to spin up our testing environment.

 

 

 

We are going to follow a single problem statement in our all blogs to maintain the uniformity and avoid the ambiguity. We are going to install nginx using chef and deploying two virtual host (opstree.com/blog/, chef.opstree.com) with it.

 


Blogs in this series

 

  1. Setup of Nginx Vhost
In this blog we describe Nginx and manually setup the nginx, as per the problem statement and also create two virtual host(opstree.com/blog/, chef.opstree.com).
  1. Chef-Resources Easy as pie..
Here we took some example of resources such as package, git, file and service and put our hands to work with chef-apply. We perform some simple task using chef resources.
  1. Chef-Resources Elementary ingredient..
This blog provides you theoretical concepts about chef resources. In this article  resources and their attributes elaborated.
  1. Chef-Recipes Bake it calmly..
Chef recipes is in consideration for this edition. Create your first recipe and apply it with chef. Complete doctrine behind the recipes of chef with simplified examples.
  1. Chef-Cookbooks Roast it perfectly..
Walls of chef house, the cookbook, written from scratch with step to step explanation. Setup of nginx and proxy implementation with sample cookbook.
  1. Chef-Cookbooks Walls of chef-house..
This blog furnish entire theoretical stuff about cookbooks. This includes command line cookbook generation and handling. One by one description of complete directory structure of a cookbook.
  1. Chef-Kitchen Do it simply..
Installation of chef kitchen. Testing of our nginx cookbook in different environment using docker container. Create, converge, verify and destroy a node with kitchen.
  1. Chef-Kitchen Chefs diagnosis center..
Theory behind the chef kitchen. Complete cycle of kitchen. With in this article elaborated view of .kitchen.yml file, and .kitchen folder provided.
  1. Chef Foodcritic && Chef Rubocop Handle it casually..
Chef lint tools, foodcritic and rubocop requirement. Theory, setup and practice exercises for foodcritic and rubocop.
  1. Chef-Databags Carry all at once..
Introduction to databags and their need. Division of code and data with databags.  Databags implementation with chef-solo. Setup of mysql password with databags.
  1. Chef-Roles Club everybody..
Requirement and implementation of chef roles. Clubbing of multiple nodes with chef roles. Complete web stack (webserver, proxy server and database) setup with roles.
  1. Chef-Environment  Organized wisely..
Chef environments for better management of the need of an organization. A complete organizational view with chef to setup different environment. Handle environments with chef-knife.
  1. Chef Server-Client Setup
Complete setup of chef client-server mode. Use of vagrant provisioning only, to spin up chef-server, chef-client and workstation.
  1. Collaboration of Client Server and Workstations
How chef-server, client and workstations work together to automate a complete infrastructure. Chef-server web interface.
  1. Chef Server-Client Work quietly..
Kickoff working with workstation. Chef-client. Install nginx and setup proxies with nginx cookbook on client node.