Confluence had problems starting up

[DG]

Hi together,

We have Confluence running in a Cluster.

On the first node the Confluence runs without error after installation but on the second node we get this message when I try the url direct from the server

Has anyone any ideas what we can check:

2023-02-16 08:20:12,469 ERROR [Catalina-utility-1] [atlassian.config.db.DatabaseHelper] closeConnectionOrLog Connection was null. We could not successfully connect to the specified database
2023-02-16 08:20:12,469 ERROR [Catalina-utility-1] [atlassian.config.lifecycle.LifecycleManager] panicAndShutdown Unable to start up Confluence. Fatal error during startup sequence: confluence.lifecycle.core:dblowercheck (Check to ensure the database correctly supports lower() for non-ASCII characters)
org.hibernate.HibernateException: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: query_wait_timeout
at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:63)
at com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider.configure(DelegatingHikariConnectionProvider.java:53)
at net.sf.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:84)
at com.atlassian.config.db.DatabaseHelper.getConnection(DatabaseHelper.java:111)
at com.atlassian.config.db.DatabaseHelper.setDatabaseLowerProperty(DatabaseHelper.java:45)
at com.atlassian.config.db.DatabaseLowerPropertyCheck.startup(DatabaseLowerPropertyCheck.java:31)
at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.startUp(DefaultLifecycleManager.java:52)
at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextInitialized(LifecycleServletContextListener.java:15)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4766)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: FATAL: query_wait_timeout
at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:596)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:582)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81)
at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:59)
... 18 more
Caused by: org.postgresql.util.PSQLException: FATAL: query_wait_timeout
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:316)
at org.postgresql.core.SetupQueryRunner.run(SetupQueryRunner.java:55)
at org.postgresql.core.v3.ConnectionFactoryImpl.runInitialQueries(ConnectionFactoryImpl.java:884)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:304)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247)
at org.postgresql.Driver.makeConnection(Driver.java:434)
at org.postgresql.Driver.connect(Driver.java:291)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
... 21 more
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:383)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:316)
at org.postgresql.core.SetupQueryRunner.run(SetupQueryRunner.java:55)
at org.postgresql.core.v3.ConnectionFactoryImpl.runInitialQueries(ConnectionFactoryImpl.java:884)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:304)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247)
at org.postgresql.Driver.makeConnection(Driver.java:434)
at org.postgresql.Driver.connect(Driver.java:291)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81)
at org.hibernate.hikaricp.internal.HikariCPConnectionProvider.configure(HikariCPConnectionProvider.java:59)
at com.atlassian.confluence.impl.hibernate.DelegatingHikariConnectionProvider.configure(DelegatingHikariConnectionProvider.java:53)
at net.sf.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:84)
at com.atlassian.config.db.DatabaseHelper.getConnection(DatabaseHelper.java:111)
at com.atlassian.config.db.DatabaseHelper.setDatabaseLowerProperty(DatabaseHelper.java:45)
at com.atlassian.config.db.DatabaseLowerPropertyCheck.startup(DatabaseLowerPropertyCheck.java:31)
at com.atlassian.confluence.impl.setup.DefaultLifecycleManager.startUp(DefaultLifecycleManager.java:52)
at com.atlassian.config.lifecycle.LifecycleServletContextListener.contextInitialized(LifecycleServletContextListener.java:15)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4766)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1396)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1386)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:455)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
... 34 more
2023-02-16 08:20:12,489 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] destroy Shutting down long running task service
2023-02-16 08:20:12,919 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

[Bhanu KN]

Hi @DG , 

Hope you are doing well :)

If the Database is set up on a different server, can you check if it's reachable from the second node? Make sure two-way communication is established and try again?

I hope the hardware requirements are met (I faced issues with 2nd node because of insufficient heap and CPUs)

Cheers,

Bhanu

[DG]

Hi Bhanu,
Thank you for your kindly reply.

The first one, I had checked was the network and port security.
And I would say, this is not the cause, the settings for network are still the same, as from the other node.

Telnet and netstat looks well.

We used for the 2nd node 16 gb and 4 cpus. This is more than the minimal requirements.

Do you have another idea? 

Kind regards
Dennis

[Nic Brough -Adaptavist-]

The error you are getting is in the conversation happening over the network, which should be:

• Confluence:  Hi, Database. Can you do this for me?  Here are my credentials; x/y/z
• Database: Hi, yes, here's the result (assuming credentials ok)
• Confluence: <goes on to next task>

What's actually happening is:

• Confluence:  Hi, Database. Can you do this for me?  Here are my credentials; x/y/z
• Database: <says nothing for ages>
• Confluence:  Database isn't talking to me, so I can't do anything else

There are three possible causes for this

Confluence has exhausted the pool of possible connections and cannot create a new one.  You can check this by looking at how far through the log after startup the error occurs.  Was it pretty much the first thing Confluence was doing with the database or did it get a long way through startup before failing?

The database is not responding to the connection.  Check the database config (including the security - sometimes databases time out when incorrect security credentials are supplied or they're configured not to talk to the requesting system)

The database is not reachable - that's the networking error to check.  The simplest one to check here is try to connect to the database from the database's command-line tool (temporarily) installed on the Confluence server.