Webhooks for Codebuild appear to randomly not trigger when code is committed to Bitbucket.

[Ryan Tandy]

I am trying to trigger 50 test builds to rebuild simultaneously in AWS Codebuild when I change the code in a directory and push it. However, each time I make a change and push to bitbucket, only a portion of the 50 start building. The tests are identical apart from a unique id in the name and they all point to the same code in the same directory. The tests that start building appear to be selected at random, and the total number of tests that successfully start to build appears to be random (22 start 1st commit, 31 start 2nd commit, 27 3rd commit, etc..). 

In bitbucket next to the commits, the builds tick mark will say 22/22 passed, 31/31 passed, or 27/27 passed etc... so it seems like an issue with Bitbucket webhooks? But I cannot figure out why.

[Patrik S · Accepted solution]

Hello @Ryan Tandy  and welcome to the community!

My understanding is that you have your workspace connected to AWS's CodeBuild using webhooks, so each event that happens in your repository is reported to AWS's Codebuild via a webhook request.

In this case, as you mentioned you have 50 builds to trigger on the AWS side, does each of the build corresponds to a webhook entry on your repository, meaning you would have 50 webhooks, or do you have just one webhook for CodeBuild in your repository? I ask this because in Bitbucket Cloud there's a limit of 50 webhooks in a repository as documented in Manage webhooks.

Also, in order to confirm if the webhook is correctly sent to AWS CodeBuild, you can try enabling the webhook history on the repository in question by navigating to Repository > Repository settings > Webhooks > View requests > Enable History (see Troubleshoot Webhooks)

Once that option is enabled, during the period of 1 week, all the subsequent webhooks will be logged and you should have access to detailed information about the request sent by Bitbucket and the response from the destination server. This can help to troubleshoot if Bitbucket is correctly sending the request and what AWS is returning as a response code.

Thank you, @Ryan Tandy !

Patrik S

[Ryan Tandy]

Hi Patrik, thanks for your reply!

Each build does correspond to a webhook so there are 50 in total. The webhook limit was already taken off. After investigating the webhook histories, it appears the webhooks were triggered and the failed ones were retried 3 times. AWS was throwing a "ThrottlingException" with the response body = {"message": "Rate exceeded"}. so it looks like the issue is coming from AWS. 

I'll update this reply if I find out a way around it.

Update:

Received the following reply on AWS:

https://repost.aws/questions/QUa5ruxsa3ThyKzalgoJ9kaA/bitbucket-webhooks-for-aws-codebuild-throttled-rate-limited

"The issue you're encountering is related to the rate at which AWS CodeBuild accepts incoming build requests. While you might not be exceeding the concurrent build limit, AWS CodeBuild imposes a limit on the number of build requests that can be made per second.

ThrottlingException errors typically occur when the number of requests exceeds the AWS CodeBuild API request rate limits.

AWS does not publicly disclose the exact rate limit, but it's important to know that it exists and is separate from the limit on concurrently running builds. This rate limit applies to the number of requests made to start builds, not the number of builds that are running.

To resolve this issue, you can implement a few strategies:

1. Retry logic: Implement retry logic in your webhook handler to handle throttling exceptions. After receiving a ThrottlingException, wait for a few seconds and then retry the request.

2. Queueing system: Use a queueing system like Amazon SQS to handle bursts of build requests. Your webhook handler can push messages to an SQS queue, and a separate worker can pull messages from the queue and start builds at a controlled rate.

3. Request limit increase: You can also contact AWS Support to request an increase in your API rate limit.

Remember, these strategies can be used in combination. For example, you could use an SQS queue to handle bursts of requests, and implement retry logic in your worker to handle any throttling exceptions that still occur.

I hope this helps!"