Random error while uploading data to media

[Lou Andrew Navarro]

Hi, we have some builds that are running in self-hosted runners (macOS & Windows env). Once in a while they both fail and gets this error:

[6ddd3b85-1, L:/0.0.0.0:56246 - R:api.media.atlassian.com/104.192.137.17:443] The connection observed an error

io.netty.handler.timeout.ReadTimeoutException: null

 An error occurred whilst uploading data to media.

io.netty.handler.timeout.ReadTimeoutException: null

Wrapped by: org.springframework.web.reactive.function.client.WebClientRequestException: nested exception is io.netty.handler.timeout.ReadTimeoutException

at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)

Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 

Error has been observed at the following site(s):

*__checkpoint ⇢ Request to PUT https://api.media.atlassian.com/upload/


Removed some of the sensitive data so some part of the logs may not make sense. After a few days the builds would pass but its becoming an inconvenience when we need the builds asap and this timeout error happens. 

I can also confirm that the builds are not getting blocked by the firewall. 

[ben_estrabaud]

The same issue has started happening to us too, but strangely, only after updating the bitbucket self hosted docker runner from version 3.1.0 to 3.6.0.

As you can see, the timeout occurs within 10 seconds:

[2024-11-19 15:58:15,347] Successfully got total chunks FileChunksInfo{dataSize=131178422B, totalChunks=3}.
[2024-11-19 15:58:15,350] Uploading 3 chunks to s3

[2024-11-19 15:58:15,357] Getting s3 upload urls for artifact.
[2024-11-19 15:58:26,103] [e6e8fa4f-1, L:/172.17.0.2:42468 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.182.101.105:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-19 15:58:26,103] [adb5d308-1, L:/172.17.0.2:42466 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.182.101.105:443] The connection observed an error
io.netty.handler.timeout.ReadTimeoutException: null
[2024-11-19 15:58:26,103] [1b69a855-1, L:/172.17.0.2:42474 - R:micros--prod-east--bitbucketci-file-service--files.s3.amazonaws.com/16.182.101.105:443] The connection observed an error

That artifact is pretty small, 125MB, and would take about 10 seconds to upload normally. We don't have an outbound firewall here that would prevent the connection. The fact that it seems to happen after 10 seconds (when the upload should be done) seems to point out to a different issue inside bitbucket. Would you be able to help?

[Syahrul]

G'day, @Lou Andrew Navarro 

There are a few reasons why you might have received this error. Typically, it occurs due to the following issues:

1. Large artifacts or cache files causing a timeout during upload.
2. Firewall or antivirus software performing packet filtering on the file during upload, which results in a timeout.

If the cache and artifacts is small then I suggest discussing these issues with your network team. Check if there is any packet filtering or capturing happening on the runner. If so, you may need to allow the URL https://api.media.atlassian.com/ to resolve the issue.

Let me know how it goes

Regards,
Syahrul

[Lou Andrew Navarro]

Hi @Syahrul thanks for the respose. This happens occasionally, and then after a while (maybe a few days), it fixes itself. We didn't apply any changes to our network as well. What we're aiming to do is not get instances like this anymore because the builds running on the runners are crucial, and we can't afford any delays in the future. 

[Syahrul]

Hey @Lou Andrew Navarro 

I noticed that you are on a paid account. Can you please raise a support ticket with us in our Support portal? We can troubleshoot the issue further. Please provide the whole runner logs attached to the ticket.

Regards,
Syahrul

[Lou Andrew Navarro]

Alright. Thanks!