docker build platform on macos apple silicon - macos docker image - docker apple silicon

How to build docker images on Apple M-series

Or any other architecture which is different than your deployment target

  |  

0 Comments k8s Kubernetes k8s-dev

How to Build a Docker Container on an Apple Silicon M-series Machine

TL;DR version (slow way) - example:

Impatient? Here:

Method 1: Use docker buildx build QEMU way

docker buildx build --platform linux/amd64 --tag my-container:latest --load .
Better ways…
Method 2 and Method 3 below - are better ways to do a docker build on Apple Silicon / Mac M1 or later machine

Details:

  • buildx build tell Docker to build using BuildKit. buildx appears in Docker version >= 19.03.
  • --platform linux/amd64 - a buildx option to set the target architecture of our build
  • --tag my-container:latest - The normal tag image option. Replace with your own value
  • --load - the buildx option will not automatically load the image into the cache (unlike normal build) - so if you plan on using the image you need this
  • . The directory context for docker. We are just using the current directory - replace as needed.

Example with extra options:

docker buildx build -f Dockerfile --platform linux/amd64 --tag my-container:latest --load .

Tag & Push?

Works the same.

docker tag my-container:latest myregistry.com/somewhere/my-container:latest
docker push

But’s It’s So Slow!

Yes it is. This is because buildx command uses QEMU. The buildx Docker build system will start up QEMU, emulating a linux/amd64 (aka x86_64) architecture. In this emulated mode it will run the entire build process. So running toolchains such as the go compiler will be slow.

Here is how to build images faster:

  • Obviously - but we will say it anyway. Just build on the native architecture. Your CI / CD flow could (should) do this.
  • Use a cross-build toolchain. This way the compiler runs native - without QEMU. This still uses buildkit

Let’s talk about the later - for the sake of building on your laptop or workstation - but much faster.

Using --platform to do fast native cross-compile builds

Method 2: Roll your own multi-arch Dockerfile - Golang example:

As you may know, Go’s compiler supports many architectures out of the box. For projects with straightforward build procedures, we can make some simple modifications to our Dockerfile and get a much faster build.

Let’s say your Golang project’s Dockerfile looked like this:

FROM golang:1.20
LABEL org.opencontainers.image.source=https://github.com/IzumaNetworks/trivial-golang-k8s-deployment
LABEL org.opencontainers.image.description="Trivial Go K8s example"
LABEL org.opencontainers.image.licenses=Apache-2.0

ADD app /app
ENV CGO_ENABLED=0
RUN cd /app && go build -o app .
WORKDIR /app
ENTRYPOINT [ "/app/app" ]

To make Docker tell Go to target a different arch (use it’s cross compiler toolchain):

FROM --platform=$BUILDPLATFORM golang:1.20 AS build
LABEL org.opencontainers.image.source=https://github.com/IzumaNetworks/trivial-golang-k8s-deployment
LABEL org.opencontainers.image.description="Trivial Go K8s example"
LABEL org.opencontainers.image.licenses=Apache-2.0

ADD app /app
# Automatic platform ARG variables produced by Docker
ARG TARGETOS TARGETARCH
# No CGO!
ENV CGO_ENABLED=0
WORKDIR /app
# Use the args to make Go build for the target arch
RUN GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o app .
ENTRYPOINT [ "/app/app" ]

(Example code is on Github.)

Now run the same build command:

docker buildx build -f Dockerfile --platform linux/amd64 --tag my-container:latest --load .

Orders of magnitude faster.

But what about Cgo?

In the above example, commenting out

# ENV CGO_ENABLED=0

Will result in an error such as:

#0 1.665 # runtime/cgo
#0 1.665 gcc: error: unrecognized command-line option '-m64'
------
Dockerfile-cross:9
--------------------
   7 |     ENV CGO_ENABLED=1
   8 |     WORKDIR /app
   9 | >>> RUN GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o app .
  10 |     ENTRYPOINT [ "/app/app" ]
  11 |
--------------------
error: failed to solve: process "/bin/sh -c GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o app ." did not complete successfully: exit code: 1
NOTE
Buildx has cutesy output - where you see a nice neat little list of build stages. When debugging this can be annoying. To get all the output use the switch --progress=plain with Docker

Why? Well because the version of gcc included in the golang image that the Dockerfile is using for the build, lacks a gcc cross-compiler targeting x86-64.

Using a Cross Toolchain

We need to install the right cross-compile version of gcc and then tell Go to use it:

FROM --platform=$BUILDPLATFORM golang:1.20 AS build
LABEL org.opencontainers.image.source=https://github.com/IzumaNetworks/trivial-golang-k8s-deployment
LABEL org.opencontainers.image.description="Trivial Go K8s example"
LABEL org.opencontainers.image.licenses=Apache-2.0
# get gcc cross toolchain
RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install g++-x86-64-linux-gnu libc6-dev-amd64-cross
RUN DEBIAN_FRONTEND=noninteractive apt-get -y clean

ADD app /app
ARG TARGETOS TARGETARCH
ENV CGO_ENABLED=1
WORKDIR /app
# RUN gcc -dumpmachine with --progress=plain here to see what gcc is used by default
RUN CC=x86_64-linux-gnu-gcc GOOS=$TARGETOS GOARCH=$TARGETARCH go build -o app .
ENTRYPOINT [ "/app/app" ]

Method 3: Dockerfile for multi-arch using xx (Best way)

But… Our Dockerfile above is now specific to x86-64. This kind of sucks - for instance let’s say you want to build and run locally on your Apple Silicon Mac, and then you decide to run it on an Arm 64-bit instance on AWS. Or you just build it on an x86-64 machine where you don’t need the cross-compiler. You don’t want to maintain multiple Dockerfiles. What then?

Use the tonistiigi/xx tools. These are written by the same author who built much of the buildx subsystem and who works at Docker. Docker’s own documentation suggests using them.

Using tonistiigi/xx:

FROM --platform=$BUILDPLATFORM tonistiigi/xx AS xx
FROM --platform=$BUILDPLATFORM golang:1.20
LABEL org.opencontainers.image.source=https://github.com/IzumaNetworks/trivial-golang-k8s-deployment
LABEL org.opencontainers.image.description="Trivial Go K8s example"
LABEL org.opencontainers.image.licenses=Apache-2.0

RUN DEBIAN_FRONTEND=noninteractive apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get -y install build-essential
RUN DEBIAN_FRONTEND=noninteractive apt-get -y clean

COPY --from=xx / /
ADD app /app
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev gcc
ENV CGO_ENABLED=1
WORKDIR /app
RUN xx-go --wrap
RUN go build -o app . && xx-verify app
ENTRYPOINT [ "/app/app" ]

Our Dockerfile is no longer glued to a specific architecture.

Once again, the same build command…

docker buildx build -f Dockerfile --platform linux/amd64 --tag my-container:latest --load .

Also, for reference, here is the same build but using Alpine as the underlying image:

FROM --platform=$BUILDPLATFORM tonistiigi/xx AS xx
FROM --platform=$BUILDPLATFORM golang:1.20-alpine
LABEL org.opencontainers.image.source=https://github.com/IzumaNetworks/trivial-golang-k8s-deployment
LABEL org.opencontainers.image.description="Trivial Go K8s example"
LABEL org.opencontainers.image.licenses=Apache-2.0

# get xx tools to build image
RUN apk add clang lld
COPY --from=xx / /
ADD app /app
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
ENV CGO_ENABLED=1
WORKDIR /app
# RUN gcc -dumpmachine
RUN xx-go --wrap
RUN go build -o app . && xx-verify app
ENTRYPOINT [ "/app/app" ]

What is going on with the xx- tools?

The tools in the xx image wrap common build tools, but with an understanding of Docker’s built-in BUILDPLATFORM and other variables.

So when you run docker build buildx --platform=linux/amd64 the linux/amd64 value is understood by xx-go, xx-gcc and xx-apk etc. These will then call the appropriate platform-specific command for your target architecture.

Conclusion: Use xx When Possible

Troubleshooting

What if the buildx command is missing?

Docker Desktop

By default, the buildx docker command is included with Docker Desktop. Any version past 19.03 should have it. If you have some other Docker installation or are not using MacOS you might need to install the buildx plugin:

docker plugin install buildx

Installation on Linux:

Download the latest docker buildx plugin from here:

https://github.com/docker/buildx/releases

And place it in ~/.docker/cli-plugins

Then verify the install.

Example:

# Download buildx

curl -L "https://github.com/docker/buildx/releases/download/v0.6.3/buildx-v0.6.3.linux-amd64" -o ~/.docker/cli-plugins/docker-buildx

# Make it executable

chmod a+x ~/.docker/cli-plugins/docker-buildx

# Verify installation

docker buildx version