{"id":6257,"date":"2021-04-20T16:32:34","date_gmt":"2021-04-20T11:02:34","guid":{"rendered":"https:\/\/opstree.com\/blog\/\/?p=6257"},"modified":"2021-04-27T16:42:28","modified_gmt":"2021-04-27T11:12:28","slug":"docker-buildkit-faster-builds-mounts-and-features","status":"publish","type":"post","link":"https:\/\/opstree.com\/blog\/2021\/04\/20\/docker-buildkit-faster-builds-mounts-and-features\/","title":{"rendered":"Docker BuildKit : Faster Builds, Mounts and Features"},"content":{"rendered":"\n<p class=\"has-text-align-justify\">It was like any other day working on micro-services project, running on Docker environment. In general, we&#8217;ve had worked on making our Image Builds more efficient, secure, and faster following basic aspects that significantly affect building and working with Docker.  <\/p>\n\n\n\n<ul><li>Understanding Docker layers and structuring the Dockerfile to maximize their efficiency.<\/li><li>Reducing the weight of the Docker image, by being specific about our Base Image Tags which comes up with minimal packages.<\/li><li>Bringing the multi-stage builds concept, etc.<\/li><\/ul>\n\n\n\n<!--more-->\n\n\n\n<p class=\"has-text-align-justify\">But keeping the spirits of being highly productive and improving more, I landed upon <code>Docker BuildKit<\/code>. Docker BuildKit is the next generation container image builder, which helps us to make Docker images more efficient, secure, and faster. It has been lingering in the background of Docker builds for some time. Moreover to enable and unleash some massive performance is to set the&nbsp;<code>DOCKER_BUILDKIT=1<\/code>&nbsp;environment variable when invoking the docker build&nbsp;command, such as:<\/p>\n\n\n\n<pre class=\"wp-block-syntaxhighlighter-code\">$ DOCKER_BUILDKIT=1 docker build .<\/pre>\n\n\n\n<p>In this post, we&#8217;ll walk through with some of its powerful features which I have explored and came up to these results as below : <\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2021\/04\/thanos.gif?w=640\" alt=\"\" class=\"wp-image-6531\" width=\"480\" height=\"270\" \/><\/figure><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Parallelism<\/h3>\n\n\n\n<p class=\"has-text-align-justify\">Considering a Multi-stage Build, when BuildKit comes across a multi-stage build it get concurrency, it analyzes the docker file and create a graph of the dependencies between build steps and uses this to determine which elements of the build can be ignored; which can be executed in parallel; and which need to be executed sequentially. For example, the stages as <code>build 1 and build 2<\/code> can be run in parallel as they don&#8217;t depend on each other however the final build stage depends on the other two so it will be build afterwards when both the other two stages are completed. <\/p>\n\n\n\n<pre class=\"wp-block-syntaxhighlighter-code\">FROM alpine AS build1\nRUN touch \/tmp\/dee.txt\nRUN sleep 10\n\nFROM alpine AS build2\nRUN touch \/tmp\/sandy.txt\nRUN sleep 10\n\nFROM alpine AS final\nCOPY --from=build1 \/tmp\/dee.txt \/tmp\/\nCOPY --from=build2 \/tmp\/sandy.txt \/tmp\/\n<\/pre>\n\n\n\n<p class=\"has-text-align-justify\">And for more checks I&#8217;ve added a delay of <code>10s<\/code> to the <code>build 1 and build 2<\/code> stages. When using the legacy build engine it executes top to bottom so it will execute two separate sleep commands of 10s which accumulates to a wait time of <code>20s<\/code>, nevertheless using BuildKit it will execute both the sleep commands at the same time and therefore only accumulate a sleep time of <code>10s<\/code>. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2021\/03\/buldkit1.png?w=817\" alt=\"\" class=\"wp-image-6409\" width=\"613\" height=\"502\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2021\/03\/buildkit3.png?w=1024\" alt=\"\" class=\"wp-image-6414\" width=\"613\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-justify\">Hence, the standard Docker comes without concurrency in which build command performs builds on Dockerfile sequentially, which means it reads and builds each line or layer of the Dockerfile at a time and as a result it took <code>49.135s<\/code>. Whereas BuildKit, allows for parallel build processing resulting in better performance and faster build times thus it only took <code>27.2s<\/code> to build it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Build Secrets Volumes<\/h3>\n\n\n\n<p class=\"has-text-align-justify\">Sometimes we need some secret keys or password to run our build, just like credentials to access AWS S3 repository. In classic Docker builds there is no good way to do this; the obvious methods are insecure, and the workarounds are hacky. Therefore, BuildKit adds support for securely passing build secrets,&nbsp;which allows build container to access secure files such as secret access keys without baking them into the image.<\/p>\n\n\n\n<pre class=\"wp-block-syntaxhighlighter-code\"># syntax = docker\/dockerfile:1.2  \n\nFROM python:3\n\nRUN pip install awscli\n\nRUN --mount=type=secret,id=aws,target=\/root\/.aws\/credentials aws s3 cp s3:\/\/walletprodtest\/terraform .\/ --recursive<\/pre>\n\n\n\n<p>To build the image, <\/p>\n\n\n\n<pre class=\"wp-block-syntaxhighlighter-code\">$ DOCKER_BUILDKIT=1 docker build --secret id=aws,src=\/root\/.aws\/credentials .<\/pre>\n\n\n\n<p class=\"has-text-align-justify\">The way BuildKit secrets work is that a file with the secret gets mounted to a temporary location during the&nbsp;<code>RUN<\/code>&nbsp;command, e.g.&nbsp;<code>\/root\/.aws\/credentials<\/code>. Since, it\u2019s only mounted during a particular&nbsp;<code>RUN<\/code>&nbsp;command, it doesn\u2019t end up embedded in the final image.<\/p>\n\n\n\n<p class=\"has-text-align-justify\">BuildKit mount types doesn&#8217;t end only with <code>secret<\/code>, we have few more :<\/p>\n\n\n\n<ul><li><strong>Cache Mount<\/strong> <strong>:<\/strong>  Sick of re-downloading all external dependencies every time when there\u2019s a change to only one of them, the&nbsp;cache mount&nbsp;can help us save time in the future. Inside of our Dockerfile, add a mount flag, specifying which directories should be cached during the step.<\/li><\/ul>\n\n\n\n<p><code>RUN --mount=type=cache,target=\/var\/lib\/apt\/lists \u2026<\/code><\/p>\n\n\n\n<ul><li><strong>SSH Mount :<\/strong> This mount type allows the build container to access SSH keys via SSH agents, with support for passphrases.<\/li><\/ul>\n\n\n\n<p><code>RUN --mount=type=ssh \u2026<\/code> etc.<\/p>\n\n\n\n<p class=\"has-text-align-justify\">Using these mount types features, makes it easier for us to pass build-time secrets, handle SSH credentials, and cache directories in between builds even if the layer needs to be rebuilt.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Remote Cache <\/h3>\n\n\n\n<p class=\"has-text-align-justify\">This feature is an upgraded version from the <code>--cache-from<\/code> which was having problems such as images need to be pre-pulled, no support for multi-stage builds, etc. With BuildKit, you don&#8217;t need to pull the remote images before building since it caches each build layer in your image registry. Then, when you build the image, each layer is downloaded as needed during the build.<\/p>\n\n\n\n<p class=\"has-text-align-justify\">For example,  keeping the Dockerfile same as used in Build Secrets Mount, we build the image using, <\/p>\n\n\n\n<pre class=\"wp-block-syntaxhighlighter-code\">$ DOCKER_BUILDKIT=1 docker build -t dgupta9068\/demo --secret id=aws,src=\/root\/.aws\/credentials --build-arg BUILDKIT_INLINE_CACHE=1 .\n\n$ docker push dgupta9068\/demo\n<\/pre>\n\n\n\n<p class=\"has-text-align-justify\">Now will make a small change in the Dockerfile by adding one more <code>RUN<\/code> command which will create a <code>test<\/code> directory. After this, prune the Docker system and try to rebuild the image using <\/p>\n\n\n\n<pre class=\"wp-block-syntaxhighlighter-code\">$ DOCKER_BUILDKIT=1 docker build --cache-from dgupta9068\/demo .<\/pre>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/opstree.com\/blog\/\/wp-content\/uploads\/2021\/04\/8.png?w=1024\" alt=\"\" class=\"wp-image-6545\" width=\"613\" height=\"400\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-justify\">To use an image as a cache source, cache metadata needs to be written into the image on creation which was done by setting&nbsp;<code>--build-arg BUILDKIT_INLINE_CACHE=1<\/code>&nbsp;when building the image. <\/p>\n\n\n\n<p class=\"has-text-align-justify\">After that, for the second fresh image build, we used <code>dgupta9068\/demo<\/code> image as cache source by which BuildKit automatically pulled up all the cached layers and just executed the last <code>RUN<\/code> layer of creating a directory. And hence, ended in a faster build of the Docker Image.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">In Conclusion<\/h3>\n\n\n\n<p class=\"has-text-align-justify\">I hope this article will give a kickstart to use and knowing more about BuildKit and its features\u00a0which will help you to simplify and speed up your Docker build workflows.\u00a0If you want to learn more about <a href=\"https:\/\/opstree.com\/blog\/\/2020\/05\/12\/fasten-docker-build\/\">fasten-docker-build<\/a> , besides BuildKit &#8211; do checkout !<\/p>\n\n\n\n<p>Till then keep building \ud83d\ude42<\/p>\n\n\n\n<p>Reference &#8211; <a rel=\"noreferrer noopener\" href=\"https:\/\/media.tenor.com\/images\/e3b5e6bca89c67b06c182c70164bc68c\/tenor.gif\" target=\"_blank\">GIF<\/a><\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><strong>Blog Pundit:&nbsp;<\/strong>&nbsp;<a href=\"https:\/\/opstree.com\/blog\/\/author\/abhishekbhardwaj510\/\"><strong>Abhishek Dubey<\/strong><\/a><\/p>\n\n\n\n<p>Opstree is an End to End DevOps solution provider<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link\" href=\"https:\/\/www.opstree.com\/contact-us\" target=\"_blank\" rel=\"noreferrer noopener\">CONTACT US<\/a><\/div>\n<\/div>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>It was like any other day working on micro-services project, running on Docker environment. In general, we&#8217;ve had worked on making our Image Builds more efficient, secure, and faster following basic aspects that significantly affect building and working with Docker. Understanding Docker layers and structuring the Dockerfile to maximize their efficiency. Reducing the weight of &hellip; <a href=\"https:\/\/opstree.com\/blog\/2021\/04\/20\/docker-buildkit-faster-builds-mounts-and-features\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Docker BuildKit : Faster Builds, Mounts and Features&#8221;<\/span><\/a><\/p>\n","protected":false},"author":198919067,"featured_media":29900,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[28070474],"tags":[346454121,768739305,717704292,292631285,257677096,717704299],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/opstree.com\/blog\/wp-content\/uploads\/2025\/11\/DevSecOps-1.jpg","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pfDBOm-1CV","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/6257"}],"collection":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/users\/198919067"}],"replies":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/comments?post=6257"}],"version-history":[{"count":24,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/6257\/revisions"}],"predecessor-version":[{"id":6616,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/posts\/6257\/revisions\/6616"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media\/29900"}],"wp:attachment":[{"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/media?parent=6257"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/categories?post=6257"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/opstree.com\/blog\/wp-json\/wp\/v2\/tags?post=6257"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}