Lucene Index Corruption

[Premiere Creative]

I’m writing about a new problem we started experiencing yesterday (February 6, 2023). In the early morning, our Jira system was working perfectly without a hitch. But soon, the Jira system started acting up, failing to display many of our open tickets. If you look at the screenshot titled “SystemDashboard,” you’ll see how our various filtered views appear incorrectly.

 When we attempt to search for various tickets, we received error notices (see screenshots “SearchError1” and “SearchError2”).

 I then navigated to the System support tab, clicked on Troubleshooting and Support Tools, then performed a health check on the system. Two errors were found.

• The first error showed that Jira version 7.9.x has reached End of Life and that we should upgrade (see screenshot titled “HealthCheck_EndOfLife”)

• The second error displayed an Indexing issue related to Lucene. If you look at the screenshot titled “HealthCheck_Lucene,” you can see that that Lucene search terminated abnormally.

Searching for a way to resolve these issues, I looking into Lucene Search indexes and found this article. The one piece of information that stuck out was the Lucene search terminated abnormally. If you look at the screenshot titled “LuceneSearch,” I’m instructed to have this issue assessed by Atlassian. Heavy load in the Jira instance may instigate this problem.

Basically, we’re looking for a streamlined way to resolve the search errors that keep triggering. I'm including some additional information about our Jira version so that I can receive a better diagnosis. 

Jira Software (Server): version 7.9.1

Database Type: mysql

Database Version: 5.5.5-10.3.27-MariaDB

The Add-Ons we have include:

• Time Reports for Jira (version 2.3.9)
• Atlassian Troubleshooting and Support Tools (version 1.8.11)
• Atlassian Universal Plugin Manager Plugin (version 2.22.4)

[Fazila Ashraf]

Hi @Premiere Creative 

Have you tried performing a reindex https://confluence.atlassian.com/adminjiraserver/search-indexing-938847710.html ?

<Note: The deployment type and tags to this question of yours are wrongly set to cloud>

[Premiere Creative]

Hi Fazila, 

Could we hire you to fix the issue? 

[Fazila Ashraf]

Hi @Premiere Creative 

Sorry my current contract doesnt permit working for another employer. 

You might find many freelance JIRA experts in linkedin and also try posting in https://community.atlassian.com/t5/Jobs-Careers/gh-p/JobsCareers 

However, i strongly believe that a full reindex following instructions in https://confluence.atlassian.com/adminjiraserver/search-indexing-938847710.html will fix the problem you see at this time.

[Jennifer Kelley]

I agree, the first thing you need to do is complete a full reindex.  That will likely solve the problem.  It should be the first step before moving onto anything else.

[Nic Brough -Adaptavist-]

This error can be caused by several things, but there is one that jumps out at me from your screenshots.

• This can happen when the server is out of disk space (either literal space or inodes - check both with "df -kh" and "df -kih)
• The server has run out of memory while running the indexing
• And the one I suspect it is - Mariadb is not supported; it can give Jira corrupt data which it can't then write to the index

I would

• Migrate to a supported database
• Stop Jira after getting the new database working
• Go into <jira home>/caches and move all the files and directories to a safe space outside Jira home
• Restart Jira 
• Do a full locking re-index

[Kelly Arrey]

(As an aside, this question is tagged incorrectly - if you're on Jira Software (Server) 7.9.1, then this is definitely not a `cloud` / `jira-software-cloud` issue.)