heartbeat/broker timeout, communication issue between server and agents

[Jesse Zhang]

We currently experiencing a problem on my Bamboo Server 6.9 for heart beat issues show on following remote agent logs, and after it reaches 10 mins timeout, the bamboo server just show 'Agent 'remote-bamboo-agent-01' went offline while building xxxxx'. The results of that build will not be available.'

INFO | jvm 5 | 2020/12/26 02:14:29 | 2020-12-26 02:14:29,652 INFO [scheduler_Worker-4] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 3 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:15:29 | 2020-12-26 02:15:29,651 INFO [scheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 4 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:16:29 | 2020-12-26 02:16:29,652 INFO [scheduler_Worker-9] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 5 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:17:29 | 2020-12-26 02:17:29,652 INFO [scheduler_Worker-2] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 6 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:18:29 | 2020-12-26 02:18:29,653 INFO [scheduler_Worker-3] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 7 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:19:29 | 2020-12-26 02:19:29,652 INFO [scheduler_Worker-5] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 8 minutes ago, dropping the current heartbeat...
INFO | jvm 5 | 2020/12/26 02:20:29 | 2020-12-26 02:20:29,652 INFO [scheduler_Worker-7] [AgentHeartBeatJob
..............................
INFO | jvm 5 | 2020/12/26 02:23:29 | 2020-12-26 02:23:29,660 FATAL [scheduler_Worker-10] [AgentContext] Agent encountered a fatal error: 'The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent'. Exiting.
INFO | jvm 5 | 2020/12/26 02:23:29 | java.util.concurrent.TimeoutException: The last successful transmission was 10 minutes ago, the heartbeat timeout is 10 minutes, terminating agent

I have tried the following measures as mentioned on bamboo server documentations:
1. Increase bamboo server heap size to 16GB
2. Delete all build logs in <BAMBOO_HOME>/xml-data/builds
3. Sync bamboo servers and remote agent to same NTP servers

All remote agents are solo instance on 16 cores 32GB machines, and the Atlassian suite is also hosted on an 64 cores machines.

In addition, I have also tried to increase heartbeat timeout intervals in
bamboo server <BAMBOO_HOME>/bin/setenv.sh as following "${JVM_SUPPORT_RECOMMENDED_ARGS:="bamboo.agent.heartbeatTimeoutSeconds=1200"}". And also added 'wrapper.successful_invocation_time=3, wrapper.max_failed_invocations=720
' to remote agent's wrapper.conf .

However, the change of heartbeat timeout settings on bamboo server and remote agents doesn't apply, and the timeout time is till 10mins(600s) by default values.

After several tries to catch the network stats, the TCP send Q both accumulated to around 6000 on both bamboo server and remote agent before previous condition happens. And 8085 http port is not reachable.

We are using bamboos to build 65 mins duration C++ builds with bamboo server + 5 remote agents.

Appreciated if anyone can give me some insight or help