How to increase performance regarding search results according to updated issues?

[Günter Halmans]

Hi Jira Support Team,

we face a performance problem in the following situation:

Background: In our Software Development process we use Release Planning Events (RPE) to discuss and plan features for a specific release. Features can be impacted by several teams. Each team documents its planning result concerning the commitment (committed, uncommitted, out of scope) in a subtask of the feature (one subtask for each impacted team). We use subtasks because there is a high variation of impacted teams over all features and using the subtasks is our current solution to implement this 1:n relation.

Challenge: At the end of the RPE the teams document their votes in the corresponding subtasks. We have created a dashboard where we can see how much features are "voted" and the consolidated vote  of the features (by defining a rule according to the linked subtasks of the specific feature). We face the problem, that the documentation of the teams in the subtasks takes a significant amount of time to be propagated to the dashboard - in a worst case it took hours.

Tests: To get an idea about the reason of this performance issue, I created a script (by using ScriptRunner) which changes the field entry of all relevant subtasks. In my tests I have used 350 features and overall 715 subtasks of these features. The number of subtasks of the feature varied from 1 to 10. For example the script changes the "vote" from "committed" to "uncommitted" in all subtasks.

Performance values of the tests: The duration of the script takes round about 2 minutes. After these  2 minutes the value of the field "vote" has been changed in all subtasks. When I search (filter) for the subtasks with the new value in the "issue search" dialogue, it takes about 7 minutes to get all subtasks with the new value in the search result table.

Questions: Is this behavior "normal"? Is there a way to speed up this behavior? Currently, we face the problem that we cannot report in real time the status of the voting because of this performance issue.

I'm looking forward to your feedback,

best regards,

Günter

[Nic Brough -Adaptavist- · Accepted solution]

Ok, there's a couple of things that worry me here, but I need to ask some more questions, with a little bit of context to them.  

Jira's search engine relies on a physical index system on the disk - it's a copy of most of the data in the database, but organised for searching (Jira's database is not, it's, at best, just a data store).

Whenever you change an issue, the index for that issue is updated with your new data.  The dashboard runs mostly off searches, and hence the index, so if you make a change to an issue that is being reported on in your dashboard, then refresh the dashboard, you should see an immediate update.

So, my questions are:

1. How exactly are you doing this voting on sub-tasks?  Is it a field you need everyone to edit (adding their names to it for example?), a counter, an action in the workflow?
2. If it is some form of change to an issue, then it should reflect in dashboards immediately (on a page refresh).  If it is not, then it starts to make me worry that something is badly wrong with your indexing.  have you modified or tinkered with it?
3. In your tests, you say "When I search (filter) for the subtasks with the new value in the "issue search" dialogue, it takes about 7 minutes to get all subtasks with the new value in the search result table" - what is this 7 minute "get"?  Does it take 7 minutes to render the issue navigator when you ask for a list of the issues, or do you mean it takes 7 minutes before the search reliably returns all the issues that were changed?

As for your main question at the end, the short answer is "no, this does not sound normal".  But I'm not sure it's a performance issue...

[Günter Halmans]

Hi Nic,

thanks a lot for your quick feedback!!

Here are my answers to your questions:

1. It is a custom field (radio button, single choice) with three options ("committed", "uncommitted", "out of scope")
2. No, as far as I know we have not changed anything according to the indexing (to be honest: I exactly know it for my own activities, but we are a team of admins for Jira, so I will ask my colleagues too - but I assume, that they also have nothing changed)
   We conduct a re-indexing once a night. Therefore, I expect that the indexes would be save every morning ;-)
3. It takes 7 minutes before the search reliably returns all the issues that were changed. I repeated refreshing the search and after 7 minutes the number of selected issues are as I would expect it.

Maybe some other system values are helpful:

In our Jira System we currently have round about...

- 300 Projects
- 300,000 Issues
- 180 issuetypes;
- 900 Customfields, 38 of them are scripted fields (using ScriptRunner)
- 1,400 activ users
- 60 permission-schemes
- 350 workflows

...that is somehow reassuring. Thus, we "only" have to find the reason behind the behavior ;-)

Additional questions regarding the indexing:

• in which way impacts the context of a custom field the indexing process, e.g. is it better to have a global context or a specific one (e.g. single project and single issue type)?
• are there any other parameters that we could use to influence positively our indexing process (e.g. reduce the time of the process in total)
• Are there any values to "read" in our system parameters which could give us hints regarding "bad" indexing situation (or other system characteristics)?

Once more: thanks a lot for your support!

Best regards,

Günter

[Nic Brough -Adaptavist-]

Ok, the vote being a single field doesn't really make much sense in terms of it being a vote (it's just a flag field), but it does contribute to the "this is a really simple action", and tells me it is just a symptom, not the cause of your problems.

A single change, to an issue, such as setting your voting field, should be indexed before you have a time to swap tabs/windows back to a dashboard or search and click "refresh"

If it's taking 7 minutes for a single change (or even a batch of a couple of hundred), your indexing is broken, there's something horribly wrong with it.  

You really need to get your admins to take a close look at the configuration, disk io, and errors so they can find and fix whatever it is that's wrong.  There's one extra clue - "we re-index every night" - NO.  That's "plastering over the cracks".  You've got a bad setup and broken process to try to pretend something isn't wrong.  This all needs fixing.  A properly set up Jira of your size should not even consider automatic indexing runs, re-indexing when fields need it should be enough (and those could be months apart)

So, on to the follow up questions

• Yes, reducing the context of fields down to where they are needed used to make a big difference until Jira 8 arrived.  Jira 7 and below indexed (but didn't use) "fields that have context, but will be empty because they're not on screen or hidden by field config".  8 changed that, and although it's still a help to limit the context of fields, it's unlikely to be worth the bother now.
• There are no other parameters that really affect indexing, other than whatever it is that is breaking your current indexing
• The indexing is not directly tune-able like that.  You need to be looking at logs, hardware, and the threads on issue change - what is it doing for 6 minutes 59 seconds left when the indexing and issue update should be finished in under a second?

[Günter Halmans]

Ok, it seems to be that I was not able to describe our use case properly, but that doesn't matter. Independent from the use case I also assume that such a change (reasonable or not) should not lead to such a system behavior.

I will ask our admins concerning the configuration, disk-io, and errors. Hopefully, they are able to see some hints for the underlying problems in our setup.

I wonder that re-indexing "could be months apart" because Jira asks for "re-indexing" every time whenever a new custom field is introduced.

Thanks for your assessment.

[Nic Brough -Adaptavist-]

No, I think it was me not quite grasping it, your description was good, and your patient explanation of it for me was excellent.

But yes, everything is pointing towards some problem with your index.  There's absolutely nothing wrong with your voting, there's something wrong with your Jira that your voting system is exposing or highlighting.

[Günter Halmans]

One more question: Due to the fact that I'm not part of the admin team (my job is to configure in Jira, e.g. workflows, screens etc.) which has installed Jira Software and Jira Service Desk in our system landscape - is there a documentation available -  beneath the "official" installation page(s) on confluence - which describes how a suitable system setup should look like?

[Nic Brough -Adaptavist-]

See https://confluence.atlassian.com/adminjiraserver/jira-applications-installation-requirements-938846826.html

[Günter Halmans]

OK, thanks!