## Sections ### How Docker works Docker is a **set of software** components, which integrate on different levels with the underlying OS, it is *typically* the name of the **Docker Engine** which consists of the CLI (**docker**) used for communication with the docker daemon (**dockerd**). Alternatively, Windows users utilise a slightly different **Docker Desktop** software, as Docker Engine is integrated natively with Linux only and needs extra steps to work on Windows. Cloud Providers often offer popular FOSS in the Software-as-a-Service (SaaS) model, this may include e.g. Docker, Kubernetes, computational solutions - such as Spark, Hadoop. Azure offers an Infrastructure-as-a-Service component (IaaS) called Azure Container Registry, which incorporates the basic infrastructure and the basic container image building and storage support. > *So what is the problem? Azure offers ACR without full Docker support?* While it is true that Azure Container Registry offers both the building and storage capabilities of Docker images, the issue as always lies in the technical details. ### The limitations of Azure Container Registry Azure Container Registry may be operated using **Azure CLI**, which means that tools normally available on the system level with their own corresponding CLIs (such as Docker) are put behind another CLI and a remote host, both of which may impose limitations on what the tools can do. Azure CLI on its own is slightly restricting the capabilities of ACR through the lens of its delegated interface. The **ACR tasks** allow to workaround some of these limitations in a smart way. In case of Docker, Azure Container Registry comes with ACR tasks, which make up an abstraction layer on top of ACR agents running the ACR software (system, dependencies) and the Docker itself. > *The ACR tasks are also requested through the CLI, but the task definitions are executed behind the CLI, meaning that some extra stuff can be "injected" into the agent!* <center><img src="https://media.tenor.com/Hyi9fxFJyNkAAAAC/bender-futurama.gif"></center> Some of the limitations, which can be worked around this way are: - image cache support (exception being the "local" and "inline" cache arguments, which are supported out of the box) - custom container drivers - multi-platform builds - generation of Software Bill of Materials (SBOM) ### Overcoming the limitations with Buildx Buildx is a Docker CLI plugin, which allows to extend the default build tool with capabilities of BuildKit. Full features can be found at the official documentation page at: https://docs.docker.com/reference/cli/docker/buildx Some of the most useful Buildx commands are: - `docker buildx create` - e.g. `buildx create --name penguin_attack --driver docker-container` - `docker buildx inspect` - e.g. `docker buildx inspect penguin_attack --bootstrap` - `docker buildx build` - e.g. `buildx build . --file <file-path> --builder=penguin_attack --load --tag <image-name:version>` The first command allows to create custom **builders** with different types of **drivers**: - docker (default) - the default built-in Docker driver - docker-container - the BuildKit driver, which enables extra options, e.g. exporting cache beyond locally and inline. - kubernetes - remote The second command allows to inspect the builders. The `--bootstrap` option ensures that the builder is operational before inspection. The last command is used to build images with a custom builder. > *Using the extended CLI of Buildx with BuildKit, the building capabilities obtain extra features, even preview ones!* For instance, Buildx and BuildKit allow to unlock some of the extended or experimental cache backends: - registry (extended) - allows to embed the image layers cache into a separate image then push it into the registry - gha (beta) - allows for uploading image layers cache into GitHub Actions cache - s3 (unreleased) - allows for uploading image layers cache into AWS S3 buckets - azblob (unreleased) - allows for uploading image layers cache into Azure Blob Storage ### Enabling BuildKit-enhanced Docker builder Some of the BuildKit features may be enabled in **Azure Pipelines** by providing the environmental variable called **DOCKER_BUILDKIT**. Azure Pipelines run on Azure DevOps agents (either self-hosted or Microsoft-hosted), it is worth noticing that according to the documentation the BuildKit support is not available on Windows agents. Quote: `BuildKit introduces build improvements around performance, storage management, feature functionality, and security. BuildKit currently isn't supported on Windows hosts.` The variable may be provided in many different ways, using the `env` block, the `variables` block, anything that produces an environmental variable entry in the Shell. ```yml - task: AzureCLI@2 name: acrBuild displayName: "ACR BuildKit Build" inputs: azureSubscription: <service-connection-name> scriptType: bash scriptLocation: inlineScript inlineScript: | cd "<dockerfile-folder-path>" az acr build . --tag <image-name:version> --registry $ACR_NAME \ --build-arg <docker-argument-name>=$DOCKER_ARGUMENT_NAME env: DOCKER_BUILDKIT: 1 ACR_NAME: <value> DOCKER_ARGUMENT_NAME: <value> ``` Example of BuildKit ACR build in Azure Pipelines YAML flavour task ### Combining BuildKit, Buildx and ACR tasks In order to use the ACR task, the Azure Pipeline `.yml` task must look slightly different ```yml - task: AzureCLI@2 name: acrBuildx displayName: "ACR Buildx Build" inputs: azureSubscription: <service-connection-name> scriptType: bash scriptLocation: inlineScript inlineScript: | cd "<acr-task-folder-path>" az acr run . --file <acr-task-name>.yml --registry $ACR_NAME \ --set ACR_ARGUMENT_NAME=$ACR_ARGUMENT_NAME env: ACR_NAME: <value> ACR_ARGUMENT_NAME: <value> ``` Example of ACR task run in Azure Pipelines YAML flavour task The differences are slight, yet noticeable: - using a file with a nested YAML template for the ACR task - a different argument passing method (ACR task arguments) Subsequently, the ACR task definition must also be defined, and this is the point where the magic happens. Just two remarks before further explanation - the `>-` symbol stands for multi-line support in YAML and the `&&` Shell operator allows to step out of the default `docker run`/`docker` command alias. This is why certain commands re-add the `docker` prefix after the `&&` operator. ```yml version: v1.1.0 steps: - id: buildx cmd: >- # Multiline support docker build https://github.com/docker/buildx.git#v0.19 --platform=local --output=. --build-arg BUILDKIT_INLINE_CACHE=1 && mkdir -p ~/.docker/cli-plugins && cp -r buildx ~/.docker/cli-plugins/docker-buildx env: ["DOCKER_BUILDKIT=1"] when: ["-"] - id: builder cmd: >- # Multiline support buildx create --name penguin_attack --driver docker-container && docker buildx inspect penguin_attack --bootstrap when: ["buildx"] timeout: 600 # 10m env: ["DOCKER_BUILDKIT=1"] - id: image cmd: >- # Multiline support buildx build . --file <dockerfile-name>.dockerfile --builder=penguin_attack --load --tag <image-name:version> --cache-to type=azblob,name=<image-name>,account_url=https://<storage-account-name>.blob.core.windows.net/,secret_access_key=<storage-account-primary-access-key>,mode=max --cache-from type=azblob,name=<image-name>,account_url=https://<storage-account-name>.blob.core.windows.net/,secret_access_key=<storage-account-primary-access-key> --build-arg ACR_ARGUMENT_NAME={{.Values.ACR_ARGUMENT_NAME} when: ["builder"] timeout: 1800 # 30m env: ["DOCKER_BUILDKIT=1"] - push: ["$Registry/<image-name:version>"] when: ["image"] ``` Example of ACR task content The first task builds a **binary** from the GitHub repository of **buildx**. The binary is built for the local host architecture and using BuildKit. The task then proceeds to export the binary using the `--output` flag. Quoting the official documentation: `If you specify a filepath to the docker build --output flag, Docker exports the contents of the build container at the end of the build to the specified location on your host's filesystem.` The binary is then installed to the Docker CLI plugins folder, as described in the official buildx repository at https://github.com/docker/buildx?tab=readme-ov-file#linux-packages. Notice that the buildx version can be easily specified and Docker will handle the building process directly from a GIT repository, which is a really convenient feature. The second task builds the **builder** using a custom **driver**. Since buildx is already installed on the ACR agent, the extended CLI is available by this step. Buildx can be used to create a new **docker-container** driver, which supports the extended features, such as multi-platform builds and cache exports. More information can be found at: https://docs.docker.com/reference/cli/docker/buildx/create/#docker-container-driver The third task can now fully utilise the extended CLI and order Docker to build the image using the extended or even experimental features. In this case, it will build a custom image from a Dockerfile, then store away the layer cache inside Azure Storage Account in an automatically generated blob container. The last task will take the image and upload it into the registry under the appropriate tag. This is **it**. By installing Buildx, the Docker CLI can be unlocked to elevate capabilities of BuildKit on an ACR agent, even if it not supported natively. ## References ### Section 1 - How Docker works - https://docs.docker.com/reference/glossary/ ### Section 2 - The limitations of Azure Container Registry - https://learn.microsoft.com/en-us/cli/azure/acr?view=azure-cli-latest - https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tasks-overview - https://learn.microsoft.com/en-us/azure/container-registry/tasks-agent-pools ### Section 3 - Overcoming the limitations with Buildx - https://github.com/docker/buildx - https://github.com/moby/buildkit - https://docs.docker.com/build/cache/backends/ ### Section 4 - Enabling BuildKit-enhanced Docker builder - https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/containers/build-image?view=azure-devops#buildkit - https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tasks-reference-yaml ## Metadata Date of creation: 2025.01.07 Date of revision: 2025.06.15