Bamboo weekly performance troubles 4.3.3

[Mike Rocheleau]

We have a large Bamboo instance that since upgrading to 4.3.3 sees massive user-experienced lag on the web interface after a week or so of being up. To the point that I can't get in and check system info or agents.

Looking at the bamboo.log while the slowness occurs we see any task in there that normally finishes in under 100ms is taking 40,000+ms to complete. I see nothing before this occurs in the logs that might cause this though, normal usage.

Host resources are totally fine across the board, Apache front-end restarted and behaving normally.

Of interesting note I looked at the last week's worth of CPU performance on this host and saw a steady ramp up in the Mhz used. It was a definite ramp over the entire week. Presuming the plan/build load hasn't changed in this time, and nothing else was added to the box, curious that the cpu metric would be constantly increasing until critical failures hit.

Any ideas would be great to check! Post restart we're again back to sub 100ms responses to all tasks in the logs, and this is the third time it's occured (each one week apart) since we upgraded to 4.3.3.

Cheers!

[Mike Rocheleau · Accepted solution]

We made the following changes to our environment and have seen no issues since:

- Took out application link between Confluence and Bamboo

- Cleaned up failed builds, we had a number of nightly recurring builds hitting the build queue that were specced to only run on a single agent which hadn't been online in weeks. So 3 builds per night were queuing up with "no agent can perform this build".

- Removed consistently failing builds

- Lowered logging on access logging to "warn" from "debug" or "info" or whatever it was.

Since this we've had no issues and Bamboo has been much faster. It's had no performance troubles in 2+ weeks now and no crashes. Good deal!

[Przemek Bruski]

One option is to upgrade to 4.4.x. There are performance/stability fixes inside that would help. If you can't do it or it does not help, please contact support and provide them with thread dumps taken when the slowdown occurrs. Also, what's the load on that machine? Is the CPU taken up by Bamboo's java process or is there something else?

[Mike Rocheleau]

Thanks, our upgrade path will have us move up to 4.4 (or 4.5 if it's out) at the end of the summer, we only just moved to 4.3.3 from 4.2.1 (4.1.2?). The load on the machine is very low, it's heavily over provisioned on memory, cpu and fast disk. It's isolated to Bamboo only so the cpu is only being used by base system processes and Java.

I have a support ticket open but wanted to see if others in the community had similar experiences at all on this release. We weren't seeing this sort of trouble on the old release, but we are getting more Agents online now so perhaps that contributed.