Jira usage very slow and cpu-usage very high, after a couple of days

[Knut_Luebke]

Hi there,

if this problem occures, we try to restart the jira-service first. But its never successful. The jira-service goes into the state "stopping service" but it could/will not be stopped within ~20-30min (very long time).

It`s necessary to terminate the corresponding tomcat-process manual.

Afterwards the jira-service is stoppend and the cpu-usage is in the normal range again ~10%. 

The problem repeats again after a couple of days.

We are running jira v8.2.4 under WindowsServer2012 R2 with an oracle database 12.1.0.2.0.

Our jira-instance is only accessible from our intranet. The instance is not reachable from the internet.

We created a thread dump from the last occurrence.

https://www.dropbox.com/s/bw1glftxe1pcwha/jira_threaddump.txt?dl=0

Can anybody help us what might cause this problem?

Thanks in advance for any help.

Regards 

-Knut

[Knut_Luebke]

Hi there,

the investigation of the structure support team reached out that the structure-plugin doesn’t seem to be the culprit in the performance problem. 

Here is the link for the coressponding structure-support-case:

https://support.almworks.com/servicedesk/customer/portal/8/ALM-15022

This link is for the last generated threaddumps: https://www.dropbox.com/s/4dc2ig93puvtpfg/jira-threaddumps.zip?dl=0

And here are the corresponding jira support logs: https://www.dropbox.com/s/vh7cbrs9ue4l6yf/Jira_support_2020-02-13-09-40-04.zip?dl=0

I`m looking forward for any help.

Best regards,

-Knut

[Mario O.G]

Hi @Knut_Luebke 

As mentioned in other post, here's the summary of our latest investigation.

1. Judging by the Performance Audit Log, there are no structures with misconfigurations, and they are all presenting a fast generation time.

2. From the thread dumps, we see a few threads blocking other threads (This might be the reason for the slowness).

org.apache.log4j.RollingFileAppender is blocking 8 threads.

java.lang.Object is blocking 5 threads.

java.util.concurrent.locks.ReentrantLock$NonfairSync is blocking 2 threads.

java.util.WeakHashMap is blocking 1 thread.

oracle.jdbc.driver.T4CConnection is blocking 1 thread.

These threads might be an indication of the following bug:

https://jira.atlassian.com/browse/JRASERVER-62958

3. We were also able to observe some database errors taking place.

2020-03-17 08:47:58,779 http-nio-8090-exec-16 ERROR knl 527x165x1 vcpnc2 10.160.1.55 /secure/AjaxIssueEditAction!default.jspa [c.d.j.p.impl.field.FieldServiceImpl] There was a SQL exception thrown by the Active Objects library:
Database:
- name:Oracle
- version:Oracle Database 12c Standard Edition Release 12.1.0.2.0 - 64bit Production
- minor version:1
- major version:12
Driver:
- name:Oracle JDBC driver
- version:12.1.0.1.0 java.sql.SQLException: Protocol violation: [ 19, 227, ]
com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
Database:
- name:Oracle
- version:Oracle Database 12c Standard Edition Release 12.1.0.2.0 - 64bit Production
- minor version:1
- major version:12
Driver:
- name:Oracle JDBC driver
- version:12.1.0.1.0java.sql.SQLException: Protocol violation: [ 19, 227, ]

There is a KB from Confluence mentioning this error:

[java.sql.SQLException: Protocol violation caught while accessing a page and Oracle DB is used|https://confluence.atlassian.com/confkb/java-sql-sqlexception-protocol-violation-caught-while-accessing-a-page-and-oracle-db-is-used-781393258.html]

Even though the KB relates to Confluence, the root cause appears to be the database driver, so it could be the same issue.

Conclusion and next steps:

I'd suggest looking into the mentioned bug and also investigate the database errors.

Judging by all the files, Structure is not playing any role in the experienced issue.

Let me know if you have any questions.

Mario O.G
[ALM Works]

[Egor Tasa [ALM Works]]

Hi Knut,

As Andy suggested, please, contact us at our support portal or by email support@almworks.com and we'll be happy to help you. The first thing to check from our side would be Performance Audit Log (found under Administration>Structure>Support), we'll let you know in the ticket what other logs and checks we'll need.

Regards,
Egor Tasa

ALM Works

[Knut_Luebke]

this was the wrong place for my answer, thats the cause i want to remove this reply, but i can`t :/

[Andy Heinzer]

Hi Knut,

I understand that your Jira instance appears to be getting into an unresponsive state after a few days and you are looking for more information about why.  Thanks for including the thread dump.  I used the samurai.jar utility to take a closer look at this thread dump.  I found that there were at least 2 threads blocked, and another 2 that were actively blocking.

Having blocked threads in that file are a clear indicator of a performance problem here, but by themselves they typically are not the source of the problem.  For that, we typically need to look at the blocking threads instead, which is where samurai is helpful.  The two threads that appear to be causing this blocking were titled:

Structure-Jobs4cb20abe System-Thread#2
JIRA-INFORM-Thread-0

The first thread is clearly related to the Structure plugin.  It might help to try to reach out to the Structure support team for help with troubleshooting that plugin to Jira.   I would try to focus on that thread first.

The second thread appears to be related to the Jira's native event listeners.  These can be used for triggering notifications on changes, or possibly utilized by third party plugin in Jira for triggering some other actions to happen.

I suspect that structure is trying to perform some huge amount of work and getting really bogged down the in the process, but it's unclear to as to the details of how or why. 

I hope this helps,

Andy