Jira Software Full Re-Index is stuck on 0%

[O M]

1. We connected production Oracle DB to our Test server

2. Jira process is restarted , but I can not create issues. Also Jira runs slow

3. Decided to re-index, but it only allows me to do a full re-index (incremental is not be chosen)

4. Started a full re-index, but after 24h, it is still at 0% complete.

catalina.out:

catalina.out:78145:2024-01-10 16:53:49,012-0600 Caesium-1-1 INFO      [c.a.jira.upgrade.UpgradeIndexManager] There are no reindex requests of type [IMMEDIATE, DELAYED] so none will be run

catalina.out:78151:2024-01-10 16:53:49,325-0600 Caesium-1-1 INFO      [c.a.jira.upgrade.UpgradeIndexManager] Reindexing is not allowed after this upgrade and there is no immediate reindex requests

catalina.out:78153:2024-01-10 16:53:49,448-0600 Caesium-1-4 INFO      [c.a.jira.startup.IndexRecoveryLauncher] Running start-index-recovery

catalina.out:78155:2024-01-10 16:53:49,929-0600 Caesium-1-4 ERROR      [c.a.jira.startup.IndexRecoveryLauncher] [INDEX-FIXER] Issue index is out of date with the database (issue_version) by more than 48 hours. Automatic recovery will not be attempted

then  com.atlassian.jira.util.index.CompositeIndexLifecycleManager.shutdown(CompositeIndexLifecycleManager.java:147)

Any ideas are appreciated!

[Nic Brough -Adaptavist-]

Welcome to the Atlassian Community!

It worries me that you "connected production database to test server".  I am really hoping you mean "we copied the production database to a test database and connected that to a test server"?

There's one obvious starting question - is the test server the exact same version of Jira as the production system?

If it is, then I want to ask how it was created.  Did you install an empty Jira, get it working to the point where your admin can log in, and then stop it and point it at the test copy of the database?  Or some other method?

I might also want to try nipping into the test system and disabling the brassband git plugin, to see if that is the blocker.

[O M]

Nic

 Thank you for your response.

 Sorry for my poor description of what has been done. You are correct on "we copied the production database to a test database and connected that to a test server"

The test server has exactly the same version as the prod. The test server was fully functional before we copied the PROD DB to a TEST DB, and also after we changed the DB.  The only problem we are having after the DB was copied is that Jira became very sluggish (even I can log in), and I was not able to create new issues (note I am a JIRA admin on this site).

When I tried to re-index, the only option that I saw was a "full reindex."  After I backed up the .tar.gz index archives, I started a "full reindex."  Checked after 24 hours, and the progress was still 0%.  Then I restored one of my most recent index files and went under Administration -> Services -> Indexing and "Recover Search Index from file" option.  Note that my 'recent' index file is not very recent, but I hoped that this will trigger an index restore and then possibly a full re-index.  After that, I have been observing the same screen for about 8hours that, "Jira index is being recovered. Depending on how large the database is, this may take a few minutes. Jira will automatically become available as soon as this task is complete."  I am not seeing any status bar during this operation, and I could not find any logs that would tell me how long this process will take.  I have not tried to disable the BigBrass plugin to see if this speeds up the indexing process.  I am willing to wait longer, but I am concerned that this reindexing process is stuck again and is not moving.  Any help is appreciated. 

[Nic Brough -Adaptavist-]

Ok, that's good and bad.  Good because you've done the right thing, and in almost exactly the way I would.   But bad because it rules out the one thing that would definitely be a problem, so we do not have a quick fix.

The only bit left on that is that I am assuming that the test system is the sluggish one, not production, from here on.

Having Jira admin just means you can administrate the system, it does not mean "can do everything", you may need to grant yourself more project permissions, but it's also a useful test in itself:

• Are your admin rights in Production the same?   Have they been inherited by the database copy, in other words.  More importantly, are your project rights in Production the same (at least for any project you want to use for investigating)?  What errors or symptoms are you getting when you try to create an issue?
• What bits of Jira are sluggish for you?  Obviously, I assume it's stuff like "create / transition / edit issues", but there's a set of problems that have a symptom of "most, if not everything, in a project is slow", but the Jira and System administrators find that the admin screens all run at a decent speed.

While either your full, locking, re-index, or the index recovery, are running, could you take a look at what the server is doing?  Start with a look at the memory, CPU and disk input/output look like, and whether the Jira process is a huge load on any of them.

[O M]

Thank you for your answer. I have also confirmed that a power user (not me), who is capable of creating issues is not able to create an issue (so it is not just my problem). When we try to create an issue, JIRA is just  'spinning' after we click on the "submit issue" button.  I am pretty sure I was able to create issues on this system before.

Yes, I am currently trying to fix the test environment. 

Jira GUI is sluggish because the buttons on the top of the screen (such as Projects, Issues, etc) render very slow, and eventually appear, but it takes a couple of minutes for them to do so.  A restart of a JIRA process seems to help that issue, but this sluggishness comes back after a few hours.

I will check the CPU, disk input/output. Thank you for your help!

[O M]

Also when the DB was switched, I got those messages:

catalina.fail:  at com.atlassian.jira.issue.index.DefaultIssueIndexer.shutdown(DefaultIssueIndexer.java:485)

catalina.fail:28-Dec-2023 19:56:00.265 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [bigbrassband-gitplugin-RevisionIndexerImpl:thread - 1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

catalina.fail:  at com.atlassian.jira.issue.index.DefaultIssueIndexer.shutdown(DefaultIssueIndexer.java:485)

Thank you!

[Olga Mill]

This log entry was appearing before we upgraded the device. this log entry is not relevant any more