Node 22 in Bitbucket pipeline slow

[Pieter Wigboldus]

I use Node 22 in my Bitbucket pipeline, and run tests with the native Node test runner, and collect coverage with the native Node coverage reporter.

In Node 22.11 it takes around 6 a 7 minutes, but Node 22.12 till 22.14 takes more than 20 minutes, so around 3x slower.

Locally it doesnt matter a lot which Node version I use.

Do you have any idea why Node 22.12+ is much slower than 22.11 and earlier?

[Suhas Sundararaju · Accepted solution]

Hi Everyone, 

Our engineering team has investigated the issue and identified that it is related to the libuv version 1.49.1, which is used in Node.js 22.12 and later.

Our Pipelines infrastructure is based on Kubernetes and runs with cgroups v2. Starting from Node.js 22.12, there is a change in behavior regarding how parallelism is calculated due to an update in libuv version 1.49.1. When executing the following command:

node -e 'console.log(require("os").availableParallelism())'

• For Node.js 22.11 and below, it correctly returns the number of available CPU cores based on the step size.

• For Node.js 22.12 and above, it incorrectly returns only 1 core for all step sizes, causing build slowness in our pipelines.

Bug Report:

We have logged a bug with the libuv team: 🔗 Libuv Issue #4740

Workaround:

Until this bug is resolved, we recommend you to use Node.js 22.11 or below as a temporary workaround. Additionally, please review the bug report and subscribe for updates.

Another potential workaround is to run your tests using a Pipelines Docker service as a Node.js container, instead of using the pipelines default build container.

definitions:
  services:
    docker:
      memory: 4096

pipelines:
  branches:
    master:
      - step:
          size: 2x
          services:
            - docker
          script:
            # Print available CPUs inside the Node.js container
            # Eg: For step size 2x and docker memory 4096, it should show 4
            - docker run --rm node:22.14 bash -c "node -e 'console.log(require(\"os\").availableParallelism())'"

            # Run tests inside a Docker container
            - export CONTAINER_NAME=node_test
            - docker run -d -t --name $CONTAINER_NAME -v $(pwd):/app node:22.14
            - docker exec -w /app $CONTAINER_NAME bash -c "npm test"

If your tests have dependencies like DBs containers and others, you can also explore Docker Compose to run the tests along with other services. 

This approach can help work around the libuv bug that prevents Node.js from properly identifying available CPU cores in the default Pipelines build container.

[Marius-B]

Big thanks for you and everyone else who helped with moving forward with solving this issue!!

[Theodora Boudale]

Hi Pieter,

My first question is the same as Gajesh's below, does this step run on Atlassian's infrastructure or with a self-hosted runner?

If it runs with a self-hosted runner, which type of runner is it (Linux Docker, Linux Shell, MacOS or Windows runner)?

If it runs on Atlassian's infrastructure, do you use a different Docker image as a build container for Node 22.11 and another one for 22.12+? Or is it the same Docker image, and you install Node during the build?

Some things that come to mind as possible reasons are:

• If we're talking about different Docker images as a build environment, the slowness may have something to do with the different Docker image.
  
  
• The newer versions of Node, 22.12+, may be more resource-intensive. Each Pipelines step has a certain amount of memory and CPU allocated to it and if Node 22.12+ is more resource intensive, this could lead to slowness.
  
  It is possible to allocate more resources to a step, this is something you could try to see if it makes a difference. You can refer to our documentation here:
  
  https://support.atlassian.com/bitbucket-cloud/docs/step-options/#Size
  
  Please keep in mind that larger size steps have billing implications, this is explained in the documentation I shared.

Assuming again that you run this step on Atlassian's infrastructure, I would recommend debugging the step locally with Docker using the following guide and then comparing performance:

• https://confluence.atlassian.com/bbkb/troubleshoot-failed-bitbucket-pipelines-locally-with-docker-1318882094.html

This way, you reproduce in an environment as close as possible to the Pipelines environment and that can help us narrow down whether the issue is with Pipelines or not. I would suggest running the step a few times locally as per the guide above, so that you get an average of the time it needs to run. Please feel free to let me know what these local tests show.

Kind regards,
Theodora

[Pieter Wigboldus]

I run on Atlassian's infrastructure.

I use the `image: node:22`

I have add multiple in parallel with the same scripts, running on different Node versions, like Node 20, 22.10, 22.11, 22.12, 22.13 and 22.14
I see that 22.12 and later are 3x slower than earlier 22.x and 20.x

So I don' t install Node, I use the official Node docker image.

And I scale it up to a size of 4x, for all Node versions in parallel, what should be more than enough.

[Theodora Boudale]

Hi Pieter and @Jakob Schröter,

Troubleshooting the slow step locally with Docker as per the guide I shared above is the way to narrow down if this seems to be a Pipelines-specific issue or not.

If you are using larger size steps, you can adjust the command on Step 4 of the guide to use the same memory and cpus as your Pipelines step. The command on Step 4 uses 4 GB of memory and 4 CPUs which applies to 1x steps, you can can adjust it accordingly depending on the size of your step (see here for reference: https://support.atlassian.com/bitbucket-cloud/docs/step-options/#Size).

When you reach Step 5 of the guide, you can add the command time in front of every command of your step to monitor how much time it takes to run. If you run it a couple of times locally and you observe faster times on average than the respective Pipelines step, you can create a ticket with the support team, share the results and the URL of a step that runs longer in Pipelines, for further investigation.

You can create a ticket via https://support.atlassian.com/contact/#/, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. When you are asked to provide the workspace URL, please make sure you enter the URL of the workspace that is on a paid billing plan to proceed with ticket creation.

Kind regards,
Theodora

[Jakob Schröter]

We're having the same issue. Using Atlassian cloud runners, standard `node:22.14.0` image and Vitest instead of the native test runner. See https://github.com/vitest-dev/vitest/issues/7302#issuecomment-2671813675 for more details and links to public repo.

[Gajesh Bhat]

Are you using the Atlassian Provided Cloud Runner or running your self-hosted runners? 

[Pieter Wigboldus]

Atlassian Provided Cloud Runner