Thread / Load error. Java process consuming over 75% CPU as of 2AM.

[Jonathan Smith]

On 6/20 we hit our thread max which caused the environment to go down. An Atlassian support rep recommended increasing the max threads to 120. This seemed to be a band-aid because we are approaching max again.

Error in logs:

20-Jun-2018 15:46:30.024 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-8090-exec-29" (id=23022) has been active for 185,298 milliseconds (since 6/20/18 3:43 PM) to serve the same request for https://confluence.ulinedm.com/json/startheartbeatactivity.action and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 7 thread(s) in total that are monitored by this Valve and may be stuck. java.lang.Throwable
	at java.lang.Object.wait(Native Method)
	at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1465)
	at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
	at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)

The applied fix: 

1) Open the Confluence.cfg.xml file in the data directory and change the following;

<property name="hibernate.c3p0.max_size">120</property>

2) Open the server.xml file in the installation directory/conf and change the following:

maxThreads="100"

This morning, a Confluence process is currently consuming over 75% of the overall CPU. I am not sure if this is related. We plan to restart the Confluence service at 8PM tonight as long as the CPU stays around 80%. 

Anyone have any other suggestions to resolve high CPU/load/thread issues? If not, I will open another case.

[Jonathan Smith · Accepted solution]

Restarting the primary confluence service brought the levels back to normal.

[Brian]

Restarting the confluence service is a little like restarting the windows server to fix any'n all issue.
But OK, it gets Confluence running again. just doesn't explain anything ;) 

[Franko Ivasic]

Another good point is adding more memory to Java in /opt/atlassian/confluence/bin/setenv.sh

CATALINA_OPTS="-Xms4g -Xmx4g -XX:+UseG1GC ${CATALINA_OPTS}"

(here is 4g (4GB), add according to the resources.)

[DKB Service GmbH]

hi, we have the same issue and allready have set the values

<property name="hibernate.c3p0.max_size">200</property>

<Connector SSLEnabled="true" acceptCount="100" port="443" clientAuth="false" 
connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false" keyAlias="1" 
... 
maxHttpHeaderSize="8192" maxSpareThreads="75" maxThreads="150" minSpareThreads="25" 
protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" secure="true" 
sslProtocol="TLSv1.2" useBodyEncodingForURI="true" sslEnabledProtocols="TLSv1.2"
xpoweredby="false" 

 but the cpu load still climb to 99% of the java process...

do you have a idea?

[Damien Davis]

What version of Confluence are you using?  How many users do you have?

[Jonathan Smith]

Cores are stressed as well: