How Docker works

Docker is a set of software components, which integrate on different levels with the underlying OS, it is typically the name of the Docker Engine which consists of the CLI (docker) used for communication with the docker daemon (dockerd). Alternatively, Windows users utilise a slightly different Docker Desktop software, as Docker Engine is integrated natively with Linux only and needs extra steps to work on Windows.

Cloud Providers often offer popular FOSS in the Software-as-a-Service (SaaS) model, this may include e.g. Docker, Kubernetes, computational solutions - such as Spark, Hadoop.

Azure offers an Infrastructure-as-a-Service component (IaaS) called Azure Container Registry, which incorporates the basic infrastructure and the basic container image building and storage support.

So what is the problem? Azure offers ACR without full Docker support?

While it is true that Azure Container Registry offers both the building and storage capabilities of Docker images, the issue as always lies in the technical details.

The limitations of Azure Container Registry

Azure Container Registry may be operated using Azure CLI, which means that tools normally available on the system level with their own corresponding CLIs (such as Docker) are put behind another CLI and a remote host, both of which may impose limitations on what the tools can do.

Azure CLI on its own is slightly restricting the capabilities of ACR through the lens of its delegated interface. The ACR tasks allow to workaround some of these limitations in a smart way.

In case of Docker, Azure Container Registry comes with ACR tasks, which make up an abstraction layer on top of ACR agents running the ACR software (system, dependencies) and the Docker itself.

The ACR tasks are also requested through the CLI, but the task definitions are executed behind the CLI, meaning that some extra stuff can be "injected" into the agent!

Some of the limitations, which can be worked around this way are:

• image cache support (exception being the "local" and "inline" cache arguments, which are supported out of the box)
• custom container drivers
• multi-platform builds
• generation of Software Bill of Materials (SBOM)

Overcoming the limitations with Buildx

Buildx is a Docker CLI plugin, which allows to extend the default build tool with capabilities of BuildKit. Full features can be found at the official documentation page at: https://docs.docker.com/reference/cli/docker/buildx

Some of the most useful Buildx commands are:

• docker buildx create
  • e.g. buildx create --name penguin_attack --driver docker-container
• docker buildx inspect
  • e.g. docker buildx inspect penguin_attack --bootstrap
• docker buildx build
  • e.g. buildx build . --file <file-path> --builder=penguin_attack --load --tag <image-name:version>

The first command allows to create custom builders with different types of drivers:

• docker (default)
  • the default built-in Docker driver
• docker-container
  • the BuildKit driver, which enables extra options, e.g. exporting cache beyond locally and inline.
• kubernetes
• remote

The second command allows to inspect the builders. The --bootstrap option ensures that the builder is operational before inspection. The last command is used to build images with a custom builder.

Using the extended CLI of Buildx with BuildKit, the building capabilities obtain extra features, even preview ones!

For instance, Buildx and BuildKit allow to unlock some of the extended or experimental cache backends:

• registry (extended)
  • allows to embed the image layers cache into a separate image then push it into the registry
• gha (beta)
  • allows for uploading image layers cache into GitHub Actions cache
• s3 (unreleased)
  • allows for uploading image layers cache into AWS S3 buckets
• azblob (unreleased)
  • allows for uploading image layers cache into Azure Blob Storage

Enabling BuildKit-enhanced Docker builder

Some of the BuildKit features may be enabled in Azure Pipelines by providing the environmental variable called DOCKER_BUILDKIT. Azure Pipelines run on Azure DevOps agents (either self-hosted or Microsoft-hosted), it is worth noticing that according to the documentation the BuildKit support is not available on Windows agents.

Quote: BuildKit introduces build improvements around performance, storage management, feature functionality, and security. BuildKit currently isn't supported on Windows hosts.

The variable may be provided in many different ways, using the env block, the variables block, anything that produces an environmental variable entry in the Shell.

- task: AzureCLI@2
  name: acrBuild
  displayName: "ACR BuildKit Build"
  inputs:
	azureSubscription: <service-connection-name>
	scriptType: bash
	scriptLocation: inlineScript
	inlineScript: |
	  cd "<dockerfile-folder-path>"
	  az acr build . --tag <image-name:version> --registry $ACR_NAME \
	  --build-arg <docker-argument-name>=$DOCKER_ARGUMENT_NAME
  env:
	DOCKER_BUILDKIT: 1
	ACR_NAME: <value>
	DOCKER_ARGUMENT_NAME: <value>

Combining BuildKit, Buildx and ACR tasks

In order to use the ACR task, the Azure Pipeline .yml task must look slightly different

- task: AzureCLI@2
  name: acrBuildx
  displayName: "ACR Buildx Build"
  inputs:
	azureSubscription: <service-connection-name>
	scriptType: bash
	scriptLocation: inlineScript
	inlineScript: |
	  cd "<acr-task-folder-path>"
	  az acr run . --file <acr-task-name>.yml --registry $ACR_NAME \
	  --set ACR_ARGUMENT_NAME=$ACR_ARGUMENT_NAME
  env:
	ACR_NAME: <value>
	ACR_ARGUMENT_NAME: <value>

The differences are slight, yet noticeable:

• using a file with a nested YAML template for the ACR task
• a different argument passing method (ACR task arguments)

Subsequently, the ACR task definition must also be defined, and this is the point where the magic happens. Just two remarks before further explanation - the >- symbol stands for multi-line support in YAML and the && Shell operator allows to step out of the default docker run/docker command alias. This is why certain commands re-add the docker prefix after the && operator.

version: v1.1.0
steps:
  - id: buildx
    cmd: >- # Multiline support
      docker build https://github.com/docker/buildx.git#v0.19 --platform=local --output=.
      --build-arg BUILDKIT_INLINE_CACHE=1 &&
      mkdir -p ~/.docker/cli-plugins && cp -r buildx ~/.docker/cli-plugins/docker-buildx
    env: ["DOCKER_BUILDKIT=1"]
    when: ["-"]
  - id: builder
    cmd: >- # Multiline support
      buildx create --name penguin_attack --driver docker-container &&
      docker buildx inspect penguin_attack --bootstrap
    when: ["buildx"]
    timeout: 600 # 10m
    env: ["DOCKER_BUILDKIT=1"]
  - id: image
    cmd: >- # Multiline support
      buildx build . --file <dockerfile-name>.dockerfile --builder=penguin_attack --load --tag <image-name:version>
      --cache-to type=azblob,name=<image-name>,account_url=https://<storage-account-name>.blob.core.windows.net/,secret_access_key=<storage-account-primary-access-key>,mode=max
      --cache-from type=azblob,name=<image-name>,account_url=https://<storage-account-name>.blob.core.windows.net/,secret_access_key=<storage-account-primary-access-key>
      --build-arg ACR_ARGUMENT_NAME={{.Values.ACR_ARGUMENT_NAME}
    when: ["builder"]
    timeout: 1800 # 30m
    env: ["DOCKER_BUILDKIT=1"]
  - push: ["$Registry/<image-name:version>"]
    when: ["image"]

The first task builds a binary from the GitHub repository of buildx. The binary is built for the local host architecture and using BuildKit. The task then proceeds to export the binary using the --output flag. Quoting the official documentation:

If you specify a filepath to the docker build --output flag, Docker exports the contents of the build container at the end of the build to the specified location on your host's filesystem.

The binary is then installed to the Docker CLI plugins folder, as described in the official buildx repository at https://github.com/docker/buildx?tab=readme-ov-file#linux-packages.
Notice that the buildx version can be easily specified and Docker will handle the building process directly from a GIT repository, which is a really convenient feature.

The second task builds the builder using a custom driver. Since buildx is already installed on the ACR agent, the extended CLI is available by this step. Buildx can be used to create a new docker-container driver, which supports the extended features, such as multi-platform builds and cache exports. More information can be found at: https://docs.docker.com/reference/cli/docker/buildx/create/#docker-container-driver

The third task can now fully utilise the extended CLI and order Docker to build the image using the extended or even experimental features. In this case, it will build a custom image from a Dockerfile, then store away the layer cache inside Azure Storage Account in an automatically generated blob container.

The last task will take the image and upload it into the registry under the appropriate tag.

This is it. By installing Buildx, the Docker CLI can be unlocked to elevate capabilities of BuildKit on an ACR agent, even if it not supported natively.

References

Section 1 - How Docker works

• https://docs.docker.com/reference/glossary/

Section 2 - The limitations of Azure Container Registry

• https://learn.microsoft.com/en-us/cli/azure/acr?view=azure-cli-latest
• https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tasks-overview
• https://learn.microsoft.com/en-us/azure/container-registry/tasks-agent-pools

Section 3 - Overcoming the limitations with Buildx

• https://github.com/docker/buildx
• https://github.com/moby/buildkit
• https://docs.docker.com/build/cache/backends/

Section 4 - Enabling BuildKit-enhanced Docker builder

• https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/containers/build-image?view=azure-devops#buildkit
• https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tasks-reference-yaml