Docker build failing for buildkit with error `authorization denied by plugin pipelines`

[Divyang Patel]

Using bitbucket runners

Docker builds suddenly started failing from yesterday evening.

Pipeline error

#1 [internal] booting buildkit

#1 pulling image moby/buildkit:buildx-stable-1

#1 pulling image moby/buildkit:buildx-stable-1 6.7s done

#1 creating container buildx_buildkit_default 0.0s done

#1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

------

 > [internal] booting buildkit:

-----

ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

[Theodora Boudale]

Hi everyone,

Just to give you some context, when a docker service is defined in a pipelines step, pipelines will automatically mount the docker client executable under the /usr/bin directory. This removes the requirement of the build image needing to have docker installed. Currently, the docker client in Pipelines uses docker CLI v20.10.23~20.10.24 (we are progressively rolling our 20.10.24 as per https://confluence.atlassian.com/bbkb/infrastructure-changes-in-bitbucket-pipelines-1142431176.html).

Certain docker images, that you may use as build containers, already come with the docker client installed on them.

We noticed that some docker images were recently updated to include the docker client v23 or v24 installed under the path /usr/local/bin.

If /usr/local/bin comes before /usr/bin in the image's $PATH variable, the system will use the docker client it finds under /usr/local/bin.

Since docker client v23, Docker has set Buildx and BuildKit as the default builder on the Linux environment. With that version, when you execute a docker build command, this is actually an alias to docker buildx build.

The buildx command tries to use the --privileged flag when building the image, and this flag is restricted in Bitbucket pipelines for security reasons (the privileged flag means that docker would allow access to all other builds on the machine), causing the pipeline to fail with the error message :

ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

To fix this issue, you can use one of the following options that were already mentioned:

(1) Export the DOCKER_BUILDKIT=0 environment variable

You can either add this command in the script of your yml file or define it as a workspace variable so you won't have to modify your yml file.

(2) Change the $PATH variable during the build time so the /usr/bin directory comes before /usr/local/bin and the pipelines docker client will be used.

export PATH=/usr/bin:$PATH

You can add this command at the beginning of your script in your yml file.

Kind regards,
Theodora

[Patrick Nelson]

Hi Theodora. Would it be possible to update the standard documentation for this error with the additional details you included in your response above (particularly  about the $PATH)? https://confluence.atlassian.com/bbkb/docker-buildkit-error-in-pipelines-when-deploying-to-google-cloud-1235859715.html

Your detailed explanation makes it much easier to understand exactly what's happening here. The KB article on the other hand still seems a big vague in comparison. This is a problem since simply disabling BuildKit might not work long term, given Docker warns that it could be removed in a future version.

Unfortunately, I had already burdened support with a ticket asking them what the true long term fix here would be before I saw your explanation, so I think better documenting this there will help a bunch of other users, too!

[Theodora Boudale]

Hi Patrick,

Thank you for the feedback. We have updated the knowledge base article and it now includes all the additional info I shared in my post.

Kind regards,
Theodora

[Christian Sciberras]

Hey,

I started implementing some stuff in a second project and (re)used most of the existing code, including the pipeline definition. Both of them use `docker:latest` image to bootstrap the rest of the project (we use `docker-compose` so our devs can bootstrap the env locally as well).

The old project is still working fine, but in the new project, I had the error mentioned above - which was successfully solved with the DOCKER_BUILDKIT suggestion.

So what confuses me, (after reading the explanation), is, why would this problem happen when both are using the same builder image? Is it possible it's also actually broken for the old project, but it's currently working through some caching?

[Theodora Boudale]

Hi Christian,

We would need to investigate to figure this out, which means looking at the build logs and yml file of the failed build (possibly also of the successful build in the old project, in order to answer why it works on the old project).

I would suggest creating a new question via https://community.atlassian.com/t5/forums/postpage/board-id/bitbucket-questions and providing that info (after sanitizing any sensitive/private data) and we can look into this. It's best to have a separate question as a certain question can become difficult to follow if we try to troubleshoot and investigate multiple users' issues with potentially different root causes.

If the workspace where these repos belong is on a paid billing plan, you can also create a ticket with the support team via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product.

Kind regards,
Theodora

[Lucas Pickersgill Teixeira]

I'm having the same problem. Only with the docker build kit turned off does it work. If anyone has any progress please share.

[Bryant Foresman]

If Buildkit must be disabled, why does the official docker in BB Pipelines documentation imply you can use it?   Is there in fact some way to enable it on BB hosted infrastructure, or can it only be enabled in self hosted runners?  From https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/

Docker BuildKit caching limitations

If Docker BuildKit is enabled and the build layers need to be cached, we recommend using the Docker Build --cache-from option. This allows one or more tagged images stored in an external image registry to be used as a cache source. This methods also avoids the 1GB size limit of the predefined docker cache.

[Bryant Foresman]

I just tried enabling Buildkit on a self-hosted runner and that fails with the same OP error; can someone explain why this limitation would be enforced on my own hardware?  It makes no sense to me.

[Theodora Boudale]

Hi Bryant,

The issue reported in this question and the answer I posted in 2023 have to do with one specific scenario. I explained it in this excerpt:

Certain docker images, that you may use as build containers, already come with the docker client installed on them.

We noticed that some docker images were recently updated to include the docker client v23 or v24 installed under the path /usr/local/bin.

If /usr/local/bin comes before /usr/bin in the image's $PATH variable, the system will use the docker client it finds under /usr/local/bin.

Outside of this very specific scenario, the current state is:

• The Docker Engine is at version 25.0.5 and BuildKit is enabled as the default builder. It can still be disabled by running export DOCKER_BUILDKIT=0 during the build.
• Docker compose v2 won’t work with BuildKit because it requires privileged mode, so BuildKit needs to be disabled for Pipelines builds running on Atlassian's infrastructure if you want to use Docker compose v2.
• The Docker client is at version 20.10.24 and we don’t support docker clients greater than that.

This and some additional info are mentioned in this community article:

• https://community.atlassian.com/t5/Bitbucket-articles/Recent-docker-service-upgrade-in-Bitbucket-Pipelines/ba-p/2603987

With regards to self-hosted runners, for Linux Docker runners you'll need to use a custom dind image for the Docker service so that you won't have the same restrictions as in Cloud runners. There is an example in the following documentation:

• https://support.atlassian.com/bitbucket-cloud/docs/configure-your-runner-in-bitbucket-pipelines-yml/#Custom-docker-in-docker-image

The other type of runners (Linux Shell, MacOS, and Windows) run builds directly on the host machine and not in Docker containers, so they don't need a Docker service. You can install Docker on the runner's host machine in order to use it during your Pipelines builds.

Kind regards,
Theodora

[Heikki Kalliokoski]

I was having same problem since 2. June. Contacted Atlassian today and they pointed out that probably my pipeline image had been upgraded (with docker client v23) and that caused the problem.

They suggested two ways to solve this:

• First one was to add DOCKER_BUILDKIT=0 to repository variables to force disable BuildKit. 
• The second one, which I chose, was to add "/usr/bin" in front of the the $PATH variable (export PATH=/usr/bin:$PATH) before running the docker build -command.  This would ensure that the used docker-cli command would be the one that the pipeline automatically mounts.

[Lucas Pickersgill Teixeira]

Thanks.
That second option worked for me!

One question, how does this change solve the problem? Why add this export?!
Did support explain why?

[Charlie Davies]

Second thanks.

Still regard the second option as a workaround as I will need to go through each pipeline (my org has a lot!) changing the configuration to get around this.

Have Atlassian indicated that they intend to fix this their side so that it will work as it has before (it has been working seamlessly for ages)?

There are two docker-cli inside the containers in the pipeline Atlassian mounted one and the container one. By changing the path you change the order of which docker-cli is automatically ran by the container. 

I believe this solves the issue because it means that the docker-cli you are running inside your pipeline is the one Atlassian expose and this has the relevant permissions.

[Patrick Nelson]

@Lucas Pickersgill Teixeira @Charlie Davies 

I, too, was confused by this. Thankfully Theodora provided a great detailed explanation above about why this works (hopefully official documentation will catch up). That's why I'm also recommending folks use the updated $PATH fix instead of simply disabling BuildKit in case the legacy builder ever gets completely removed in later versions.

[Igor Makowski]

Self hosted runner with dind image can still do buildkit. 
But seems that self hosted runners are broken too, no matter how much instances we have in our infrastructure, all jobs are getting queued to the first runner.

[Wouter Horré]

We see the same failure in all our pipelines since yesterday. The workaround works for us. 

Since legacy build is deprecated in docker, I wonder how to fix `docker build` without disabling BuildKit? 

[Charlie Davies]

We had the same issue where one minute our pipelines were working and the next they were not. Since nothing has changed on our side I assumed that it is related to the restrictions which are imposed on the cloud pipelines and an update to docker buildkit has broken something. I think it is an issue for Atlassian to fix?

I have never reported a bug/issue to Atlassian before so hoped by adding to this thread that is would draw attention to the issue.

Glad the workaround works, it saved us yesterday when needing to get a release out in a hurry.

Echo your question that disabling docker buildkit is not an ideal solution and hopefully this issue can be fixed so buildkit can be re-enabled.

[Charlie Davies]

Can confirm that we are also seeing this issue in our Bitbucket pipelines with docker builds.

The workaround for the time being was to set DOCKER_BUILDKIT=0 as a repository variable.