Hello,
Since 3rd of October the service is not working properly on node1 out of 4 of Confluence cluster.
Current errors from catalina.out logs are:
14-Oct-2024 02:55:06.079 SEVERE [Catalina-utility-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [net.sourceforge.jtds.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.295 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.microsoft.sqlserver.jdbc.SQLServerDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.296 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [com.github.gquintana.metrics.sql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
14-Oct-2024 02:55:06.308 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8090"]
14-Oct-2024 02:55:06.322 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [62851] milliseconds
Following these errors the service in systemctl appears running but in reality the node joins the cluster only at service start and then disconnects.
Looking back in the logs I see that the service was running just fine until September 19 after which there is a big interruption in the logs until 3rd of October when the issues started to appear:
19-Sep-2024 03:23:27.538 INFO [main] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8090"]
19-Sep-2024 03:23:27.542 INFO [main] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8090"]
NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
[0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/opt/atlassian/confluence/latest/logs/gc-2024-10-03_20-50-40.log instead.
03-Oct-2024 20:50:41.394 WARNING [main] org.apache.tomcat.util.digester.SetPropertiesRule.begin Match [Server/Service/Connector] failed to set property [debug] to [0]
03-Oct-2024 20:50:41.928 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8090"]
03-Oct-2024 20:50:41.995 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]
On the 3rd of October we start to see errors:
2024-10-03 20:50:52,486 ERROR [Catalina-utility-1] [com.hazelcast.instance.AddressPicker] log [LOCAL] [Dell CPG Confluence] [3.12.14-atlassian-5] Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.
com.hazelcast.core.HazelcastException: Cannot bind to a given address: /10.237.49.206. Hazelcast cannot start. Port [5801] is already in use and auto-increment is disabled.
03-Oct-2024 20:50:52.659 SEVERE [main] org.apache.catalina.core.StandardServer.await Failed to create server shutdown socket on address [localhost] and port [8000] (base port [8000] and offset [0])
java.net.BindException: Address already in use (Bind failed)
2024-10-03 20:50:52,809 FATAL [Catalina-utility-2] [atlassian.spring.container.SpringContainerContext] getComponent Spring Application context has not been set
2024-10-03 21:05:38,479 WARN [Catalina-utility-2] [atlassian.confluence.plugin.PluginFrameworkContextListener] contextDestroyed Failed to shut down plugin system during ServletContext shutdown: Spring Application context has not been set
03-Oct-2024 21:07:42.558 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector["http-nio-8090"]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
I have excluded node1 from load balancer so no traffic is forwarded towards it.
I have tried to gracefully power off and on the service using the confluence scripts (stop-confluence.sh & start-confluence.sh ), but no improvement.
A service restart using systemctl did not work also, neither a node restart.
I have checked file permissions and made sure that the service is running with confluence user.
I checked the database configuration from confluence.cfg.xml and everything is fine.
I compared the node with the other 3 which work just fine in regards of firewall rules and all is the same.
May I please have support on this issue?
Sorry I couldn't provide a support zip with the logs & config because the node does not stay connected long enough to download it.
Hi Robert,
Welcome to Atlassian community!
To resolve your issue, I recommend creating a ticket directly to Atlassian Support (Request assistance button on top), where you can describe the situation and include a log of the damaged Cluster in the attachment. Atlassian support will be able to help you more than "regular Atlassian users".
Pavel
You're right, thanks.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Stand out as a skilled Confluence Cloud administrator. The Confluence Administration for Cloud certification proves you can configure permissions, manage site and space settings, and monitor activity—ensuring secure, effective collaboration for your team.
Start here
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.