{"id":10137,"date":"2022-03-22T17:34:38","date_gmt":"2022-03-22T12:04:38","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=10137"},"modified":"2022-06-15T15:57:44","modified_gmt":"2022-06-15T10:27:44","slug":"introduction-to-buildkite","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2022\/03\/22\/introduction-to-buildkite\/","title":{"rendered":"Know How to Get Started with Buildkite"},"content":{"rendered":"\n

We’re on the verge where all the small to big organisations are shipping software to customers in minutes. Earlier this was not possible, but thank’s to Continuous Integration (CI) and Continuous Delivery (CD) that made it possible.<\/p>\n\n\n\n

Continuous Integration (CI) is the practice of integrating code changes from multiple contributors into a single project via automation. Continuous Delivery (CD) is the automated process to build, test, configure & deploy software into the production environment. Both methods combined make it possible to build, test & deploy everyday code changes into the production environment.<\/p>\n\n\n\n

There are lots of tools in the market to implement CICD, one among which we are about to discuss today in this post.<\/p>\n\n\n\n

Let me introduce you to Buildkite.<\/p>\n\n\n\n\n\n\n\n

Buildkite<\/h2>\n\n\n\n

Buildkite is a CI and build automation tool that combines the power of your own build infrastructure with the convenience of a managed, centralized web UI. It is a hybrid SaaS platform designed for enterprise scale. You can run as many jobs as you need and buildkite will manage the orchestration and reporting. You\u2019re in charge of your builds, Buildkite is in charge of the platform.<\/p>\n\n\n\n

Buildkite consists of three<\/strong> main components:<\/p>\n\n\n\n

Buildkite Agent:<\/strong><\/h5>\n\n\n\n

The Buildkite agent is a small, reliable, and cross-platform build runner that makes it easy to run automated builds on your own infrastructure. Agents run on several platforms including Ubuntu, Debian, Mac, Windows, Docker, and others. Its main responsibilities are polling buildkite.com for work, running build jobs, reporting back the status code and output log of the job, and uploading the job’s artifacts.<\/p>\n\n\n\n

How does Buildkite Agent Work?
<\/strong>The agent works by polling Buildkite\u2019s agent API over HTTPS. Since HTTPS protocol is used there is no need for port forwarding or firewall access for incoming requests. The agent registers itself with Buildkite & places it into your organization\u2019s agents pool. Because of this scaling of agents is very easy with Buildkite.<\/p>\n\n\n

\n
\"\"<\/figure><\/div>\n\n\n

The agent polls Buildkite for a new build job. Once received and accepted, the agent will execute the command, stream back the build script\u2019s output, and then posts the final exit status. Whilst the job is running, you can use the buildkite-agent meta-data command to set and get build-wide meta-data.
Buildkite provides a rich set of metadata that can be used in post-build actions or post-build notifications.<\/p>\n\n\n\n

Pipelines:<\/strong><\/h5>\n\n\n\n

Pipelines are containers to define and model the build workflows and also to kick off post-build steps.<\/p>\n\n\n\n

A pipeline is a template of the steps or instructions that need to be executed during the build. There are many types of steps that can be included in a single pipeline, some run source code pull, code build, some run scripts, some define conditional logic, and others wait for user input. We can include any number of steps in the pipeline template. When the pipeline is run, a build gets created. Each of the steps in the pipeline ends up as jobs in the build, which then get distributed to available agents.<\/p>\n\n\n\n

Deployments:<\/strong><\/h5>\n\n\n\n

These are steps that run after a build is complete and can be either triggered automatically or manually.
There are many ways to set up both manual and continuous deployment workflows using Buildkite. This covers various ways of architecting deployment pipelines, common workflows, and how to integrate with external deployment systems.<\/p>\n\n\n\n

Single deployment steps:<\/strong>
Adding a deployment step that runs after your tests pass is the simplest way to deploy from a Buildkite pipeline.<\/p>\n\n\n\n

The example pipeline.yml below shows how to set up continuous deployment using a single step that runs after the tests pass.<\/p>\n\n\n\n

pipeline.yml<\/figcaption>
steps<\/span>:<\/span>\n  -<\/span> label<\/span>:<\/span> \"<\/span>\ud83d\udd28\"<\/span>\n    command<\/span>:<\/span> \"<\/span>scripts\/tests\"<\/span>\n\n  -<\/span> wait<\/span>\n\n  -<\/span> label<\/span>:<\/span> \"<\/span>\ud83d\ude80\"<\/span>\n    command<\/span>:<\/span> \"<\/span>scripts\/deploy\"<\/span>\n    if<\/span>:<\/span> build.branch == 'main'<\/span>\n    concurrency<\/span>:<\/span> 1<\/span>\n    concurrency_group<\/span>:<\/span> \"<\/span>my-app-deploy\"<\/span>\n<\/code><\/pre><\/div><\/figure>\n\n\n\n
This pipeline uses a conditional to only run on commits to the main branch and sets a concurrency limit of 1 to ensure that only one deployment happens at a time.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/p>\n\n\n\n
Deployment plugins:<\/strong>
There are Buildkite plugins<\/a> available for various systems and tools. For example, the ECS Deploy plugin<\/a> and the AWS Lambda Deploy plugin<\/a>.<\/p>\n\n\n\n
The following example shows how to use the ECS Deploy plugin to automatically deploy a pre-built Docker image to an AWS ECS<\/code> service:<\/p>\n\n\n\n
pipeline.yml<\/figcaption>
steps<\/span>:<\/span>\n  -<\/span> block<\/span>:<\/span> \"<\/span>Deploy\"<\/span>\n    prompt<\/span>:<\/span> \"<\/span>Deploy<\/span> <\/span>to<\/span> <\/span>production?\"<\/span>\n  -<\/span> label<\/span>:<\/span> \"<\/span>:ecs:<\/span> <\/span>\ud83d\ude80\"<\/span>\n    concurrency<\/span>:<\/span> 1<\/span>\n    concurrency_group<\/span>:<\/span> \"<\/span>rails-app-deploy\"<\/span>\n    plugins<\/span>:<\/span>\n      -<\/span> ecs-deploy#v1.3.0<\/span>:<\/span>\n        cluster<\/span>:<\/span> \"<\/span>production\"<\/span>\n        service<\/span>:<\/span> \"<\/span>app\"<\/span>\n        task-definition<\/span>:<\/span> \"<\/span>production-deploy\/rails-app.json\"<\/span>\n        task-family<\/span>:<\/span> \"<\/span>rails-app\"<\/span>\n        image<\/span>:<\/span> \"<\/span>my.ecr.repo\/rails-app:${BUILDKITE_COMMIT}\"<\/span>\n        task-role-arn<\/span>:<\/span> \"<\/span>deployer\"<\/span>\n        deployment-configuration<\/span>:<\/span> \"<\/span>100\/200\"<\/span>\n<\/code><\/pre><\/div><\/figure>\n\n\n\n
You can set up your pipelines to create a build whenever there is a deployment created in GitHub. You can trigger these builds using a call to GitHub\u2019s Deployments REST API<\/a>, or using the GitHub Slack app<\/a>\u2019s \/github deploy my-org\/my-repo<\/code> command.<\/p>\n\n\n\n
To enable builds to be created from GitHub deployment events, create a pipeline and select ‘Trigger builds on deployment’ in your Buildkite pipeline’s GitHub settings:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
To customize the deployment’s environment name and URL in GitHub, you can set the following two build meta-data<\/a> values in the pipeline that performs the deployment:<\/p>\n\n\n\n
buildkite-agent meta-data set<\/span> \"github_deployment_status:environment\"<\/span> \"staging\"<\/span>\nbuildkite-agent meta-data set<\/span> \"github_deployment_status:environment_url\"<\/span> \"https:\/\/staging.my-app-dev.com\/\"<\/span>\n<\/code><\/pre><\/div>\n\n\n\n
Understanding Buildkite\u2019s Hybrid Infrastructure & Scalablity<\/h5>\n\n\n\n
Scalability<\/strong>
Buildkite is a hybrid SaaS platform designed for enterprise scaling. Since the agents are self-hosted you will have complete control over the parallel scaling of your CI tasks. You can add unlimited agents at no extra cost. Run as many jobs as you need and Buildkite will manage the orchestration and reporting. You\u2019re in charge of your builds, Buildkite is in charge of the platform.<\/p>\n\n\n\n
Extensibility<\/strong>
Buildkite makes it easy to create, find, and implement integrations with third-party technologies, with an extensive library of plugins and deep native support for major source control, containers, and dev tool solutions. It also allows you to write & use your own plugins, this helps customize your pipeline to your specific workflow.<\/p>\n\n\n\n
Security<\/strong>
With the self-hosted Buildkite agent jobs are run in your environments, behind your firewall. Because your source code is never shared with Buildkite, Buildkite provides the control and security of on-prem combined with the convenience of self-hosting.<\/p>\n\n\n\n
Advantages of using Buildkite:<\/h5>\n\n\n\n
GitHub deployments:<\/strong><\/a><\/p>\n\n\n\n