%% Use bold for emphasises and italics for quotes or phrases %% ## Sections %% ## What problem does it solve %% ### Docker on Clouds Docker is a **set of software** components, which integrate on different levels with the underlying OS, it is *typically* the name of the **Docker Engine** which consists of the CLI (**docker**) used for communication with the docker daemon (**dockerd**). Alternatively, Windows users utilise a slightly different **Docker Desktop** software, as Docker Engine is integrated natively with Linux only and needs extra steps to work on Windows. [[#^1|(1)]] Cloud Providers often offer popular FOSS in the Software-as-a-Service (SaaS) model, this may include e.g. Docker, Kubernetes, computational solutions - such as Spark, Hadoop. Azure offers an Infrastructure-as-a-Service component (IaaS) called Azure Container Registry, which incorporates the basic infrastructure and the basic container image building and storage support. While it is true that Azure Container Registry on paper offers both the building and storage capabilities of Docker images, the issue as always lies in the details. ### The limitations of Azure Container Registry Azure Container Registry may be operated using **Azure CLI**, which means that tools normally available on the system level with their own corresponding CLIs (such as Docker) are put behind another CLI and a remote host, both of which may impose limitations on the interface. [[#^2|(2)]] The **ACR tasks** allow to work around some of these limitations in a smart way. Azure Container Registry comes with ACR tasks, which make up an abstraction layer on top of ACR agents running the ACR software (system, dependencies) and the Docker itself. [[#^3|(3)]] >[!info] Info >The ACR tasks are also **requested** through a CLI. However, the task are executed behind the CLI, meaning that some extra stuff can be "injected" into the agent! <center><img src="https://media.tenor.com/Hyi9fxFJyNkAAAAC/bender-futurama.gif"></center> Some of the limitations, which can be worked around this way are: - image cache support (exception being the "local" and "inline" cache arguments, which are supported out of the box) - custom container drivers - multi-platform builds - generation of Software Bill of Materials (SBOM) %% ## How does it solve it %% ### Overcoming the limitations with Buildx Buildx is a Docker CLI plugin, which allows to extend the default build tool with capabilities of BuildKit. Full features can be found at the official documentation. [[#^4|(4)]] Some of the most useful Buildx commands are: - `docker buildx create` - e.g. `buildx create --name penguin_attack --driver docker-container` - `docker buildx inspect` - e.g. `docker buildx inspect penguin_attack --bootstrap` - `docker buildx build` - e.g. `buildx build . --file <file-path> --builder=penguin_attack --load --tag <image-name:version>` The first command allows to create custom **builders** with different types of **drivers**: - docker (default) - the default built-in Docker driver - docker-container - the BuildKit driver, which enables extra options, e.g. exporting cache beyond locally and inline. - kubernetes - remote The second command allows to inspect the builders. The `--bootstrap` option ensures that the builder is operational before inspection. The last command is used to build images with a custom builder. >[!info] Info >Using the extended CLI of Buildx with BuildKit, extra features become unlocked, even the preview ones. For instance, Buildx and BuildKit [[#^5|(5)]] allow to unlock some of the extended or experimental cache backends [[#^6|(6)]]: - registry (extended) - allows to embed the image layers cache into a separate image then push it into the registry - gha (beta) - allows for uploading image layers cache into GitHub Actions cache - s3 (unreleased) - allows for uploading image layers cache into AWS S3 buckets - azblob (unreleased) - allows for uploading image layers cache into Azure Blob Storage %% ## How to use it %% ### Enabling BuildKit-enhanced Docker builder Some of the BuildKit features may be enabled in **Azure Pipelines** by providing the environmental variable called **DOCKER_BUILDKIT**. Azure Pipelines run on Azure DevOps agents (either self-hosted or Microsoft-hosted), it is worth noticing that according to the documentation the BuildKit support is not available on Windows agents. >[!quote] Documentation [[#^7|(7)]] > "How do I set the BuildKit variable for my Docker builds? > > [BuildKit](https://github.com/moby/buildkit) introduces build improvements around performance, storage management, feature functionality, and security. BuildKit currently isn't supported on Windows hosts." The variable may be provided in many different ways, using the `env` block, the `variables` block, anything that produces an environmental variable entry in the Shell. ```yml ^c1 - task: AzureCLI@2 name: acrBuild displayName: "ACR BuildKit Build" inputs: azureSubscription: <service-connection-name> scriptType: bash scriptLocation: inlineScript inlineScript: | cd "<dockerfile-folder-path>" az acr build . --tag <image-name:version> --registry $ACR_NAME \ --build-arg <docker-argument-name>=$DOCKER_ARGUMENT_NAME env: DOCKER_BUILDKIT: 1 ACR_NAME: <value> DOCKER_ARGUMENT_NAME: <value> ``` >[!info] Example of BuildKit ACR build in Azure Pipelines YAML flavour task [[#^8|(8)]] ### Combining BuildKit, Buildx and ACR tasks In order to use the ACR task, the Azure Pipeline `.yml` task must look slightly different ```yml - task: AzureCLI@2 name: acrBuildx displayName: "ACR Buildx Build" inputs: azureSubscription: <service-connection-name> scriptType: bash scriptLocation: inlineScript inlineScript: | cd "<acr-task-folder-path>" az acr run . --file <acr-task-name>.yml --registry $ACR_NAME \ --set ACR_ARGUMENT_NAME=$ACR_ARGUMENT_NAME env: ACR_NAME: <value> ACR_ARGUMENT_NAME: <value> ``` >[!info] Example of ACR task run in Azure Pipelines YAML flavour task [[#^8|(8)]] The differences are slight, yet noticeable: - using a file with a nested YAML template for the ACR task - a different argument passing method (ACR task arguments) Subsequently, the ACR task definition must also be defined, and this is the point where the magic happens. Just two remarks before further explanation - the `>-` symbol stands for multi-line support in YAML and the `&&` Shell operator allows to step out of the default `docker run`/`docker` command alias. This is why certain commands re-add the `docker` prefix after the `&&` operator. ```yml version: v1.1.0 steps: - id: buildx cmd: >- # Multiline support docker build https://github.com/docker/buildx.git#v0.19 --platform=local --output=. --build-arg BUILDKIT_INLINE_CACHE=1 && mkdir -p ~/.docker/cli-plugins && cp -r buildx ~/.docker/cli-plugins/docker-buildx env: ["DOCKER_BUILDKIT=1"] when: ["-"] - id: builder cmd: >- # Multiline support buildx create --name penguin_attack --driver docker-container && docker buildx inspect penguin_attack --bootstrap when: ["buildx"] timeout: 600 # 10m env: ["DOCKER_BUILDKIT=1"] - id: image cmd: >- # Multiline support buildx build . --file <dockerfile-name>.dockerfile --builder=penguin_attack --load --tag <image-name:version> --cache-to type=azblob,name=<image-name>,account_url=https://<storage-account-name>.blob.core.windows.net/,secret_access_key=<storage-account-primary-access-key>,mode=max --cache-from type=azblob,name=<image-name>,account_url=https://<storage-account-name>.blob.core.windows.net/,secret_access_key=<storage-account-primary-access-key> --build-arg ACR_ARGUMENT_NAME={{.Values.ACR_ARGUMENT_NAME} when: ["builder"] timeout: 1800 # 30m env: ["DOCKER_BUILDKIT=1"] - push: ["$Registry/<image-name:version>"] when: ["image"] ``` >[!info] Example of ACR task content [[#^8|(8)]] ^cb1 The first task [[#^cb1|(buildx)]] builds a **binary** from the GitHub repository of **buildx**. The binary is built for the local host architecture and using BuildKit. The task then proceeds to export the binary using the `--output` flag. >[!quote] Documentation [[#^9|(9)]] >"[Export binaries from a build](https://docs.docker.com/build/building/export/#export-binaries-from-a-build) > >If you specify a filepath to the `docker build --output` flag, Docker exports the contents of the build container at the end of the build to the specified location on your host's filesystem. This uses the `local` [exporter](https://docs.docker.com/build/exporters/local-tar/)." The binary is then installed to the Docker CLI plugins folder, as described in the official buildx repository. [[#^4|(4)]] Notice that the buildx version can be easily specified and Docker will handle the building process directly from a GIT repository, which is a really convenient feature. The second task [[#^cb1|(builder)]] builds the **builder** using a custom **driver**. Since buildx is already installed on the ACR agent, the extended CLI is available by this step. Buildx can be used to create a new **docker-container** driver, which supports the extended features, such as multi-platform builds and cache exports. [[#^10|(10)]] The third task [[#^cb1|(image)]] can now fully utilise the extended CLI and order Docker to build the image using the extended or even experimental features. In this case, it will build a custom image from a Dockerfile, then store away the layer cache inside Azure Storage Account in an automatically generated blob container. The last task [[#^cb1|(push)]] will take the image and upload it into the registry under the appropriate tag. >[!summary] TL;DR By installing Buildx manually, the Docker CLI can be 'unlocked' to elevate capabilities of BuildKit on an ACR agent, even if it not supported natively. ## References ### Section 1 - Docker on Clouds - https://docs.docker.com/reference/glossary/ ^1 ### Section 2 - The limitations of Azure Container Registry - https://learn.microsoft.com/en-us/cli/azure/acr?view=azure-cli-latest ^2 - https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tasks-overview ^3 - https://learn.microsoft.com/en-us/azure/container-registry/tasks-agent-pools (extra) - https://github.com/Azure/acr/issues/721#issuecomment-3129939863 (extra) - https://github.com/microsoft/azure-pipelines-tasks-common-packages/issues/273 (extra) ### Section 3 - Overcoming the limitations with Buildx - https://github.com/docker/buildx ^4 - https://github.com/moby/buildkit ^5 - https://docs.docker.com/build/cache/backends/ ^6 ### Section 4 - Enabling BuildKit-enhanced Docker builder - https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/containers/build-image?view=azure-devops#buildkit ^7 - https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tasks-reference-yaml ^8 - https://docs.docker.com/build/building/export/ ^9 - https://docs.docker.com/reference/cli/docker/buildx/create/#docker-container-driver ^10 ## Metadata Date of creation: 2025.01.07 Date of revision: 2025.12.12