Is a regular JIRA restart important?

[Jose M.]

Do you restart JIRA e.g. once a week? Is it recomended in order to speed it up? Some users using JIRA very intensively are requesting this, realising that the system is slowing down after a certain time.

Thanks for your help!!

[no_longer_in_sudoers_file · Accepted solution]

Jose,

I am the Systems Engineer responsible for support.atlassian.com (SAC), a JIRA instance here at Atlassian.  While I do have weekly maintenance windows which sometimes require restarts, I try to avoid restarting SAC at all just to keep my up-time numbers high and meet the SLA my managers have set.

Can you provide some context for what you refer to as "very intense use"?  I would say the support.atlassian.com site is pretty intense, but I know we have customers who have even more intense implementations out there.

[Jose M.]

Hi Sam, If there are around 1000 http requests or even much more shown for an user in the user session gui, I suposed this to be an intensive use. This happens especially for those running Scrum and Kanban boards with an high number of Projects and issues (but not 10.000,00).

[Jose M.]

A few days ago JIRA was having a 30 to 60s response time. Instead of restarting the system, I locked and re-indexed it (I was in doubt, if a re-index in the background would bring the same result). After that, the system was responding in up to 10s. But I am neither interested in restart the system very often, nor lock the system for a re-index.

[Jose M.]

Thanks for all the good answers! Since only one can be accepted, I am going to accept Sam's answer.

[Nic Brough -Adaptavist-]

Regular restarts are not the answer!  That's just "plastering over the cracks".

 You need to work out why it's slowing down and deal with the cause of the issue.  I've worked with a number of "intensively used" JIRA systems and not needed (or wanted) to restart them for months (and maintenance is for patches, hardware change or upgrades, never because they're slow).  Since version 4, I've been able to nail the "it's running too slowly" symptom to faults in plugins or users doing bad things (like running remote filters for "everything" over REST, or trying to have Agile boards with 10,000 issues for example)

[Jose M.]

Hi Nic, you have been able to nail the issues causing the system to slow down. You mentioned faults in plugins, as a possible source for such problems. But how to identify them correctly? E.g., is the scriptRunner such a candidate? After using it for a while, my account was terrible affected (but it was already slower as normal).

[Nic Brough -Adaptavist-]

Script-runnner itself is not a culprit, but oh heck yes, the scripts you can run with it are. It's the reason I don't ever recommend the use of script runner unless you have admins who know Jira well enough to be able to avoid writing scripts that can break it. There are plenty of other plugins that can cause problems, although I would say that most of them are caused by edge-cases, where normal usage of the plugin is fine, but when you try to do something odd, or "clever", then you run into issues which the developers didn't predict. The problem is that there is no easy answer to what the cause is. You need to monitor it, look for patterns, look for the type of slowdown - Something eaten all the memory? Disk IO thrashing? CPU loaded? Run out of threads? Database being thrashed? etc.

[Sergey Svishchev]

Surely there's method to this madness by now? :) Atlassian's own support team must know it, I hope...

[Harry Chan]

What do you mean by method? The instance needs to be well managed and tuned. You need someone to look at it from time to time to tell. Each instance is different. You need to find where the bottleneck is.

[Sergey Svishchev]

The method by which someone who's not an JIRA developer or member of support team can identify the fault in reasonable timeframe and/or tune the instance. Something like https://blogs.atlassian.com/2015/05/webinar-recap-five-secrets-jira-performance-scale/ maybe?

[xinan liu]

I am facing the same situation with Jose.

If I edit a workflow in a long running JIRA, I need to use a long time to entering the workflow editing page just like condition postfunction. Then it will become very fast after I restart the JIRA.

Some times I will use the "Force garbage collection" function in "System - System Info". It can speed up the JIRA but not effective then restart.

Is that the reason was due to JAVA? Unlike c or c++, the programe running on JVM always have some performence problem, restart the machine may be a way to solve it.

And there is not a hadware problem in my situation because my server has 48GB memroy and I give JIRA 8GB, the cpu is enough for using too (Xeon E5 * 2).

[Harry Chan]

It could be the way it's configured and settings as well. Also the whole stack, e.g. version of Java, garbage collection algorithms - there's a lot to it. Have configured JIRA instances to run for a long time without any slowdown.

[Nic Brough -Adaptavist-]

Just throwing resource at a Java system is often not the answer. It is when you're running out, but doing it without *knowing* that low resources are the problem can make things worse. Restarting the machine is the wrong answer too - you need to monitor and investigate what is really happening before you can fix it properly.

[Jose M.]

I have just learned today, from Atlassian Support, that the matter with Using Garbage Collection Logs to Analyze JIRA Performance is worth to be aware of. 

[Daniel Wester]

You should take a look at access or even use something like New Relic (do a trial - and it's free for a bit  ). It will help you see what's going on. I've seen slow down across the system due to caching not really being configured correctly. Another issue has been a third party system overloading the system with requests (iframe on their side etc) which "disappeared" when the restart of my system occurred (and after an hour or so they were back).

Def. look in the lhttp ogs and get some type of live profiling going on that will let you dig into the java classes and see what's the most active.

[Sergey Svishchev]

It is possible (saw this more than a few times) for the app to accumulate long-running requests that tie up resources (CPU etc.). App itself never terminates them, even if the client is long gone.

[Harry Chan]

Have you checked if there's enough memory? How has CPU usage been? It's often the case that after prolonged use a lot of things get cached in and out and the lack of memory is actually causing a lot more swapping to occur.

[Jeanette Hamel]

I realize this is an older thread, but certainly a relevant issue for us.

We also perform a scheduled weekly stop and restart of Jira, then a lock and re-index. I'm not sure why, and no one could tell me, but I believe it might have been a cache issue. We have over 400k issues in our Jira instance, and the lock and re-index is taking us an average of 45 minutes to an hour. Having an outage of 1 hour/week is not ideal, as our NOC is 24/7 and it interrupts their ability to actively work issues. I anticipate that this will only take longer, as our Jira instance is increasingly growing. 

A few questions:

• Is an automated stop and restart of Jira still not recommended?
• Is there any other way to clear our cache?
• Is there any way to schedule a background re-index instead of a lock and re-index?
• We are currently on Jira Server 7.3, but will upgrading to a more recent version of Jira Server help with this?

Thanks for any help anyone can provide!

[Vickey Palzor Lepcha]

• I do an automated stop and start every night - it is scheduled as per my database stop and start. I JIRA shuts down first - then my DB shuts down . After a while, by DB starts and then my JIRA starts. They are all automated.
• Cache - not quite sure which cache you are hinting at.
• I run an automated - background re-indexing every night after my JIRA is up.
• These are all more related to System Administration - new releases have more to do to with JIRA administration.

You may have to deal more on linux/windows scripting for your issues here.

[Nic Brough -Adaptavist-]

• Is an automated stop and restart of Jira still not recommended?

Yes.  You should not need to restart and stop it.  If you are having problems that a restart fixes temporarily, you should be looking at why those problems are occurring, not restarting and ignoring them.  There is every chance that they will corrupt your data sooner or later.

• Is there any other way to clear our cache?

Partially, but a restart is the cleanest way to clear the cache.  But, as above, you should not need to.

• Is there any way to schedule a background re-index instead of a lock and re-index?

There are some scripts that can do it, but a background index fixes fewer problems than a full index.  But, as above, if you're re-indexing because of problems in the index, cache or other faults, you need to be fixing the causes of these, not plastering over the cracks.

• We are currently on Jira Server 7.3, but will upgrading to a more recent version of Jira Server help with this?

Depends on what the root cause of the problem is.  There are bug fixes in later versions, but unless you know what the problem really is, we don't know if they might fix what you're running into.

[Nic Brough -Adaptavist-]

Taking the database out like that is a good reason for a restart, but yes, it could easily do damage to your system unless you do it correctly. Stop JIRA *properly* (not a "kill") Stop and restart the database Restart JIRA That should prevent problems, as a clean shutdown shouldn't break anything.

[Vickey Lepcha]

Just to add - I have regular restart ( once a week ) automated . Would that cause a problem in the long run ? I believe any application requires a restart time to time . My reason to schedule a restart is because the Database my Jira communicates to gets restarted once a week - and that seems to be creating some errors in my log , so to avoid that . I have also scheduled a jira restart along with the DB .

[Nic Brough -Adaptavist-]

Sorry, I didn't mean to say "wrong answer" really. It's the "wrong thing to do" - you should not ever automate this. While you're working on the correct thing to do (find and fix the cause), yes, you may need to restart. But if you automate it, you'll be tempted to ignore it because you're seeing less problems on the surface (although probably little improvement because the problem might not even be a leak, and if it's not, restarting at the wrong time won't help you at all). And that's how your house falls down.

[Jose M.]

Hi Nic, That was not an answer!, but my comment to my own question. I am grateful for all the help and I am trying to follow them and fix the wrong things. But the system was already very slow since 5 days and the situation inacceptable.

[Nic Brough -Adaptavist-]

No, that is the wrong answer - all the answers you've already had are about finding the *Cause* of the slow down, not patching over a flaw. Don't waste your time filling a crack in a wall with a bit of plaster every time it cracks again, when the crack could be telling you "your house is about to fall down"

[Jose M.]

System became unusable and crashed. After a restart it is now working again as stable and fast as expected. If we don't know yet why the system slows down after a certain time, we should look for a way to restart the system periodically and automatically.