Jira Software Thread get stucked

[Shiroukoji]

Hey,

I have actually a Problem with my self-hosted Jira Software. I'm using a fresh install of the docker container atlassian/jira-software:8. The installation works well, but after the installation I can use it for a few minutes and then it freezes. I also can't login anymore, because it freezes. 

It tried to access Jira directly or behind an reverse proxy with https, each of them have the same issues. 

After taking a look at the logs, every time there are these warning:

[http-nio-8080-exec-11] com.sun.jersey.spi.container.servlet.WebComponent.filterFormParameters A servlet request, to the URI https://_myURL_/rest/gadget/1.0/login, contains form parameters in the request body but the request body has been consumed by the servlet or a servlet filter accessing the request parameters. Only resource methods using @FormParam will work as expected. Resource methods consuming the request body by other means will not work as expected.

and 

[ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8080-exec-10] (id=[29]) has been active for [123,347] milliseconds (since [8/4/19 3:49 PM]) to serve the same request for [https://_myURL_/secure/] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

I set up the Jira JVM to get 4096MB (it uses ~1200MB) memory and it has no limitations for cpu.

Have you any idea how I can fix it? 

Greetings

[Andy Heinzer]

Hi,

I see that you are noticing a stuck thread error in the logs and that this message appears to be related to a time when Jira is unresponsive.  I would like to learn more about this unresponsiveness and how you get around this problem in the past.  For example, when this happens does Jira eventually recover?  Or are you having to stop the process and restart Jira here?

If Jira does not log anything else in the $JIRAHOME/log/atlassian-jira.log or the $JIRAINSTALL/logs/catalina.out, then there are a couple of things that could be happening.  Either

1. The java process running Jira has been terminated.  This is known to happen on Linux environments when Jira's memory allocation is set so high that the system runs out of system memory needed for other basic operating system processes, and then the Linux kernel can terminate the process using the most resources.  When this happens Jira's logs can't record anything relating to this event, and you have to look at some system logs to determine if this is the case.  Details on how to do this are in How to Resolve JIRA Process being Terminated Unexpectedly in Linux.
2. Jira could be processing a large garbage collection of its memory.  If this is a stop-the-world collection, the application becomes completely unresponsive until the Java garbage collection process can complete.  I would recommend walking through Using Garbage Collection Logs to Analyze JIRA Application Performance.   Personally, I like using gcviewer to review these logs files to see what kinds of stoppage time a Java application like Jira is recording, such as how often and for how long

Between those two possibilities that should cover the most common reasons Jira would appear unresponsive here.   #1 is a condition in which you can only start Jira again to recover.  Whereas #2 will eventually finish, it just can take some time depending on system performance.   I'd also recommend that if you set the Xmx for Jira to be 4096m that you also set the Xms to be the same value.   The reason for this is also covered in Why is it important to have xms and xmx value as same?.  The short answer is, it tends to help overall performance to eliminate Java from having to do any kind of resizing on the heap itself (increases or decreases), as both event types cause a stop-the-world type event for an application like Jira.

If you have covered these steps above and are still seeing this same stuck thread problem, then I would want to recommend trying to Generating a thread dump.  Or rather a series of thread dumps just before and during this the time when this happens to Jira.  Sometimes we can learn more from the threads when the logs do not contain clear errors or warning. 

Regards,

Andy

[Shiroukoji]

Thanks for your answer!

I don't have to stop Jira, it recovers after a little time. 

I tried the recommendations you've linked, but it didnt't helped.

I did some thread-dumps but they don't give me any information I can work with.

Maybe you can help me. I uploaded them on my OneDrive, because I don't found an solution to attach a file to a post. 

https://1drv.ms/u/s!AvTWLwQkyNe5hjgXEQ7kBAiMi8x7?e=ONMzAT

[Andy Heinzer]

Unfortunately, the thread dumps don't show any blocking, blocked, or deadlocked threads here that would be clear indicators of a problem bit of code.  At least not in those pair of threads.  Would it be possible for us to take a closer look at a support zip form this instance?  Perhaps we can start looking there instead to see if there are other clues that might help us here.

Andy

[vusxi1]

27-May-2020 09:03:27.941 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [ajp-nio-8009

-exec-133] (id=[31725]) has been active for [120,702] milliseconds (since [5/27/20 9:01 AM]) to serve the same request for [https://jira.vcapps.org/] and may be stuck (configured threshold

for this StuckThreadDetectionValve is [120] seconds). There is/are [74] thread(s) in total that are monitored by this Valve and may be stuck.

 java.lang.Throwable

Getting this logs and jira server went in hang mode,any one can help on this

[Andy Heinzer]

Hi @vusxi1,

Sorry to hear of this problem with Jira hanging.  However this message by itself is not enough for us to know why just yet.  Ideally we would want to see a thread dump taken at the time Jira becomes unresponsive.  Steps to do this are in Generating a thread dump.   Thread dumps taken after the problem are not useful in determining what is causing Jira to hang at that time.

It might also help if we could take a closer look at a support zip from this Jira site taken after Jira has recovered or has been restarted.  This might contain clues about your environment that tell us more about why this happened. 

Is this hanging something you can recreate within Jira?  Or was this a one time concurrence?

Andy

[Earl McCutcheon]

Hello Shiroukoji,

The First warning you noted relating to the "filterFormParameters" is just excess logging and just a false positive, check out the following feature request for some details on the:

• https://jira.atlassian.com/browse/JRASERVER-59898

The second warning as well is just warnings, the stuck thread message is to bring our attention to threads that may be causing performance issues, so they help when we are troubleshooting a specific issue.   As long as you are not seeing any performance hiccups in tandem with hanging threads they are mostly benign, however in your case you can clearly see the instance is suffering from a freeze up when this is happening so defiantly something performance related.  You'll want to look at the logging to see if there are any additional errors that can help narrow this one down.

As a follow up I would recomend that you collect a thread dump while the error is occuring to get additional details on the hanging thread that was stuck, but overall I would recomend focusing on the rest of the logging for additional errors as the first step to get additional context on the failure point, and let me know what you find.

Regards,
Earl

[Benjamin Yu]

We have the same problem of stuck valve. We found when it reach 200 stuck threads, jira is not responsive, basically just crashes. 

Here is the error:

may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.

it normally take 3 hours from There is/are [1] thread(s) to There is/are [200] thread(s), then jira crashes. Our linux server has 64 gb memory, 8 CPUs.

We need help on this ASAP.

Thank you.

[Luong La]

I have same issue when ran Integrity checker
Error: "../secure/admin/IntegrityChecker.jspa] and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds).

jira v9.5.0 #950000-sha1:bfd830f
with
OS: Ubuntu 20.04
Java Version 11.0.17
Java VM OpenJDK 64-Bit Server VM
Apache Tomcat: 9.0.68
Database type and version: postgres 14.4
Database driver PostgreSQL JDBC Driver 42.5.1

[Benjamin Yu]

Add more memory and CPU to my web server fixed this problem. You should try it.

[Luong La]

Thanks, but cpu/mem is not the issue here, our jira is m4.2xlarge (8 vCPU, 32 GiB mem), with JVM opt  -Xms12048m -Xmx26702m 

RDS is db.m5.large (2 vCPU, 8 GiB mem)

Java VM Memory Statistics
- Total Memory 26704 MB
- Free Memory 20714 MB
- Used Memory 5990 MB

[Boris T]

@Luong La Does your instance become unresponsive and you are forced to wait or restart when you run integrity checker? 

[Luong La]

No, everyting seems working fine except the integrity checker did not work!

[Boris T]

We are in the same boat, when running integrity checker - we get stuck thread warning and overall system performance not effected. 

From reading Earl McCutcheon post, I believe you can ignore stuck thread warning produced by integrity checker when overall performance of system is not effected.

Integrity checker not working is an issue there is no argument about that, and this discussion is about stuck threads that lead to unresponsive system. 

[Benjamin Yu]

we have 8 cpu and 64 gb memory on Linux. We don't see any error any more. Integrity check is not work at all. Just don't use it.