Build & Release Challenges : Problems

So here is the consolidated list of the problems that current system have, I’ve categorized all the issues in different categories so that they can be managed properly

  • CI Builds
    • Code stability builds are not in place
    • Code quality builds are not in place
    • Code deployment builds for non-prod environments not in place
    • A lot of manual steps in prod deployments
    • All the projects are performance critical so builds to do profiling of projects
  • Database updates is manual at all the stages of deployment
  • Automated smoke testing of all the applications
  • Integration of bug tracking system with version control system
  • A release server needs to be set-upped so that release can be managed properly
  • No documentation at all 🙂
  • Version Control System : We are using git as version control system
    • Branching strategy has some flaws as a result of which merging takes a lot of time
    • We don’t have a GUI to manage git server

Build & Release Challenges

I was planning to write this blog 2-3 days back in fact not a blog but a blog series & this blog will be the starting of this blog series. This blog will only give you an overview what I’ll be discussing in the coming blogs, first the reason why I’m writing this blog 🙂 well the reason is I’ve changed the job 🙂 and I’ll be working as a Build & Release manager. The challenge that I’ve have is that right now their is very little or no streamlined processes defined for build and release and obvious very less automation so I’ve to work on these things. This means that in the coming blogs I’ll be discussing various problems that I’ll be facing & how to overcome that.

Automated Database Update Or Rollback

One of the important step during release is doing database update and rollback in case something goes wrong, usually people perform this operation manually. In this blog I’ll talk about how we can automate this process by following some convention.

Here I’m taking mysql database as an example we can have same conventions for other databases also

Convention to manage rollback/updates of a release

  • Each project codebase at it’s root will have a folder database_scripts
  • The database_scripts folder will contain folder for each release i.e Release1_1, Release2_0…
  • The database scripts release folder will in turn contains two folders update & rollback which will contain updates & rollbacks scripts for a release.

Automating the rollback/update

  • The update folder will have a source input file FileSequencer.txt. This file will point to all the update scripts in correct order that needs to be executed for the release
  • In the similar manner rollback folder will have a source input file FileSequencer.txt. This file will point to all the rollback scripts in correct order that needs to be executed for the release
  • At last we will have a utility shell script, this script will take db details and execute all the scripts referred in FileSequencer.txt using mysql command

Release Strategy for Java Web based projects

In this post I’ll be discussing about the 2 strategies that  we can follow for releasing a Java based web project.

A project can be primarily released in two ways
    Incremental Release
    Full Release

Incremental Release is done in big projects which has multiple modules & usually few modules gets updated between two releases. It makes sense to include only updated modules in release archive and during deployment only update those modules in application server.

Full release is usually done in small projects where the release archive contains all the components and then this release archive can be deployed to the application server as a whole

Both incremental & full release strategy has their pros & cons, where full release strategy scores in simple release archive generation & deployment incremental release has upper hand in space usage by only having modified components in it, although it brings overhead when doing rollback.

Release Steps in Incremental Release Strategy: If you are following incremental strategy in general you need to perform following steps

1.) Checkout the latest code for the release
2.) Generate the list of components which needs to be deloyed for the release
3.) Generate the release archive based on the list of components
4.) Stop the server(If hot deployment of components is not available)
5.) Take the backup of existing application on application server as we may need to do rollback in case of any issues
6.) Replace the components in application server with the components in the release archive
7.) Start the server(If hot deployment of components is not available)

Release steps in Full Release Strategy: As explained earlier Full release strategy is fairly simple, steps involved are:
1.) Checkout the latest code for the release
3.) Generate the release archive for whole application
4.) Stop the server(If hot deployment of components is not available)
6.) Deploy the release archive to the application server
7.) Start the server(If hot deployment of components is not available)

Git : How to fix issues in a merged branch

Sometime back we faced a peculiar problem in our project where we have to fix some issue introduced due to a merged branch. To explain this problem first I’ve to explain the way we manage branches. As we entered in the maintenance mode of project there was a dire need of fixing issues and working on change requests in isolation. Whenever we need to fix an issue we cut a issue-branch out of main branch fix that issue in isolation and after verifying and making sure the issue is fixed we merge that issue-branch back in main branch. The issue we faced with the approach is that after merging an issue branch with the main branch sometimes we came across some regression bugs due to the issue branch. There were two possible solutions to overcome the problem.

First solution is to fix the issue in issue branch and merge the issue branch again with master. This solution can work but the problem with this approach is that till the time you haven’t fixed the issue introduced due to issue branch and merged it back with master branch you can not create a new issue-branch and much bigger problem is you can’t release.

The second solution is if we can somehow revert the merge of issue branch with main branch and then fix the issue in issue-branch, after that merging the fixed issue branch with the main branch. This approach seems to be straight forward and more logical. Git comes up with a cool command git-revert which can revert existing commits and even revert merge of another branch. I’ll talk about the solution in the next blog 😉