Confluence node is not joining the cluster

[Robert Tone]

Hello,

Since 3rd of October the service is not working properly on node1 out of 4 of Confluence cluster.

Current errors from catalina.out logs are:

14-Oct-2024 02:55:06.079 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors

14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.296 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

14-Oct-2024 02:55:06.308 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]

14-Oct-2024 02:55:06.322 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [62851] milliseconds

Following these errors the service in systemctl appears running but in reality the node joins the cluster only at service start and then disconnects.

Looking back in the logs I see that the service was running just fine until September 19 after which there is a big interruption in the logs until 3rd of October when the issues started to appear:

19-Sep-2024 03:23:27.538 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8090"]

19-Sep-2024 03:23:27.542 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8090"]

NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/latest/logs/gc-2024-10-03_20-50-40.log instead.

03-Oct-2024 20:50:41.394 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]

03-Oct-2024 20:50:41.928 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]

03-Oct-2024 20:50:41.995 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]

On the 3rd of October we start to see errors:

2024-10-03 20:50:52,486 ERROR [Catalina-utility-1] [com.hazelcast.instance.AddressPicker] log [LOCAL] [Dell CPG Confluence] [3.12.14-atlassian-5] Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.

com.hazelcast.core.HazelcastException: Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.

03-Oct-2024 20:50:52.659 SEVERE [main] org.apache.catalina.core.StandardServer.await Failed to create server shutdown socket on address [localhost] and port [8000] (base port [8000] and offset [0])

       java.net.BindException: Address already in use (Bind failed)

2024-10-03 20:50:52,809 FATAL [Catalina-utility-2] [atlassian.spring.container.SpringContainerContext] getComponent Spring Application context has not been set

2024-10-03 21:05:38,479 WARN [Catalina-utility-2] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextDestroyed Failed to shut down plugin system during ServletContext shutdown: Spring Application context has not been set

03-Oct-2024 21:07:42.558 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]

       org.apache.catalina.LifecycleException: Protocol handler initialization failed

I have excluded node1 from load balancer so no traffic is forwarded towards it.

I have tried to gracefully power off and on the service using the confluence scripts (stop-confluence.sh & start-confluence.sh ), but no improvement.

A service restart using systemctl did not work also, neither a node restart.

I have checked file permissions and made sure that the service is running with confluence user.

I checked the database configuration from confluence.cfg.xml and everything is fine.

I compared the node with the other 3 which work just fine in regards of firewall rules and all is the same. 

May I please have support on this issue?

Sorry I couldn't provide a support zip with the logs & config because the node does not stay connected long enough to download it.

[Pavel Junek · Accepted solution]

Hi Robert,

Welcome to Atlassian community!

To resolve your issue, I recommend creating a ticket directly to Atlassian Support (Request assistance button on top), where you can describe the situation and include a log of the damaged Cluster in the attachment. Atlassian support will be able to help you more than "regular Atlassian users".

Pavel

[Robert Tone]

You're right, thanks.