Jira Startup Failed - Update from 7.1.4 to 7.2.15 - Unable to either create or replace clustered job

[Henry Auffahrt]

Hello,

I found a very similar topic here, but the solution there didn´t solve our problem.

We have a very old Jira instance running on AWS infrastructure (EC2-Instance with Amazon Linux). We want to update it to the newest version, therefore we want small update steps, to avoid big update problems.

Update from 7.1.4 to 7.2.15 - After running the installer and startup the instance the instance failed to start up with this error message:

Here an excerpt from the "logs/catalina.out" Logfile:

[..]

2018-07-23 12:11:46,571 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] 
 
***********************************************************************************
JIRA 7.2.15 build: 72018 started. You can now access JIRA through your web browser.
***********************************************************************************
 
2018-07-23 12:11:46,588 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system lateStartup begun
2018-07-23 12:11:46,591 JIRA-Bootstrap INFO [c.a.plugin.manager.DefaultPluginManager] Plugin system lateStartup ended
2018-07-23 12:11:46,594 JIRA-Bootstrap DEBUG [c.a.activeobjects.osgi.ActiveObjectsServiceFactory] getService bundle [com.atlassian.plugins.atlassian-whitelist-api-plugin]
2018-07-23 12:11:46,625 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] 
 
___ Plugin System Started _________________
 
2018-07-23 12:11:46,625 JIRA-Bootstrap ERROR [c.a.jira.upgrade.UpgradeLauncher] Skipping, JIRA is locked.
2018-07-23 12:11:46,626 JIRA-Bootstrap INFO [c.a.jira.scheduler.JiraSchedulerLauncher] JIRA Scheduler not started: JIRA startup checklist failed.
2018-07-23 12:11:46,678 JIRA-Bootstrap ERROR [c.a.jira.startup.AnalyticsLauncher] Start analytics not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=03def26b-b3ec-4a6d-afc7-a7b0e102d241,jobRunnerKey=com.atlassian.jira.startup.AnalyticsLauncher.Start,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Mon Jul 23 12:14:46 CEST 2018,intervalInMillis=0]],nextRunTime=Mon Jul 23 12:14:46 CEST 2018,version=1,rawParameters=(null)]
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
at com.atlassian.jira.startup.AnalyticsLauncher.runStartAnalyticsJob(AnalyticsLauncher.java:72)
at com.atlassian.jira.startup.AnalyticsLauncher.start(AnalyticsLauncher.java:40)
at com.atlassian.jira.startup.ActiveServicesLauncher.start(ActiveServicesLauncher.java:56)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:178)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:171)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:159)
at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:156)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:145)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:101)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:99)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
at java.lang.Thread.run(Thread.java:745)
2018-07-23 12:11:46,745 JIRA-Bootstrap ERROR [c.a.jira.startup.IndexRecoveryLauncher] Index recovery not scheduled
com.atlassian.scheduler.SchedulerServiceException: Unable to either create or replace clustered job: ImmutableClusteredJob[jobId=260b755d-0820-42db-9726-3e1d91c91d50,jobRunnerKey=com.atlassian.jira.DefaultJiraLauncher.IndexRecovery,schedule=Schedule[type=INTERVAL,intervalScheduleInfo=IntervalScheduleInfo[firstRunTime=Mon Jul 23 12:11:46 CEST 2018,intervalInMillis=0]],nextRunTime=Mon Jul 23 12:11:46 CEST 2018,version=1,rawParameters=(null)]
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.createOrReplaceWithRetry(CaesiumSchedulerService.java:202)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleClusteredJob(CaesiumSchedulerService.java:189)
at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.scheduleJob(CaesiumSchedulerService.java:137)
at com.atlassian.scheduler.core.AbstractSchedulerService.scheduleJobWithGeneratedId(AbstractSchedulerService.java:114)
at com.atlassian.scheduler.core.DelegatingSchedulerService.scheduleJobWithGeneratedId(DelegatingSchedulerService.java:77)
at com.atlassian.jira.startup.IndexRecoveryLauncher.runIndexRecoveryJob(IndexRecoveryLauncher.java:102)
at com.atlassian.jira.startup.IndexRecoveryLauncher.start(IndexRecoveryLauncher.java:49)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postTenantArrived$4(DefaultJiraLauncher.java:181)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager$StartupTask.run(DefaultInstantUpgradeManager.java:139)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.runTask(DefaultInstantUpgradeManager.java:55)
at com.atlassian.jira.startup.DefaultInstantUpgradeManager.doNowOrWhenInstanceBecomesActive(DefaultInstantUpgradeManager.java:42)
at com.atlassian.jira.startup.DefaultJiraLauncher.postTenantArrived(DefaultJiraLauncher.java:171)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDBActivated$3(DefaultJiraLauncher.java:159)
at com.atlassian.jira.tenancy.DefaultTenantManager.doNowOrWhenTenantArrives(DefaultTenantManager.java:55)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:156)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:145)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:298)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:194)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:135)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:101)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:99)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:149)
at java.lang.Thread.run(Thread.java:745)
2018-07-23 12:11:46,755 JIRA-Bootstrap INFO [c.a.jira.startup.DefaultInstantUpgradeManager] Late startup launchers took 0s
2018-07-23 12:11:46,760 JIRA-Bootstrap ERROR [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.IllegalStateException: Abnormal system startup detected
at com.atlassian.jira.startup.LauncherContextListener.initDone(LauncherContextListener.java:180)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:162)
at java.lang.Thread.run(Thread.java:745)
2018-07-23 12:11:46,775 JIRA-Bootstrap INFO [c.a.jira.startup.LauncherContextListener] Memory Usage:
---------------------------------------------------------------------------------
Heap memory : Used: 418 MiB. Committed: 1602 MiB. Max: 1946 MiB
Non-heap memory : Used: 224 MiB. Committed: 244 MiB. Max: 1264 MiB
---------------------------------------------------------------------------------
TOTAL : Used: 642 MiB. Committed: 1846 MiB. Max: 3210 MiB
---------------------------------------------------------------------------------
2018-07-23 12:11:48,826 Modification Check:thread-1 INFO [c.a.jira.startup.JiraStartupLogger] 
 
___ Modifications ___________________________
 
Modified Files : jira-application.properties
Removed Files : None

To solve this I tried the following things (found in this three articles)
- https://community.atlassian.com/t5/Jira-Core-questions/JIRA-Startup-Failed-Unable-to-either-create-or-replace-clustered/qaq-p/686280

- https://community.atlassian.com/t5/Jira-Software-questions/Unable-to-start-JIRA/qaq-p/726633

-https://confluence.atlassian.com/jirakb/troubleshooting-jira-startup-failed-error-394464512.html

I run these to commands on our database:

jiradb=# reindex database "jiradb";
jiradb=# vacuum analyze public.clusteredjob

This did not work, then I delete the "clustereddjob" table complete, also this didn't work. (There were no duplicate rows in this table)

Also, I tried:

- Increasing JIRA Memory.
- Start Jira in Safe Mode (all plugins disabled)
- Plugins require more time to startup (add 30sec wait)
     -Datlassian.plugins.enable.wait=300

 Can you help me with this problem?

[Earl McCutcheon · Accepted solution]

Hey There Henry,

It looks like you have already covered a lot of ground on this one, and thanks for all the details.

As mentioned in the alternate thread you mentioned with similar errors the next steps recommended before the alternate workaround was noted which you tried without success, was to set up a new empty Database, point the install to the new DB and migrate the data using a XML:

It might be helpful here to try to create a new empty database, and a new sql user.   You could then shutdown Jira, edit the dbconfig.xml file to tell Jira to use this new empty database, and from there you could then start up Jira.   When Jira starts up connected to an empty database, it launches the setup wizard.  In this setup wizard there is an option to import from a backup.  By default Jira is still creating these xml backup zip files in the $JIRAHOME/export/ directory.   Provided you have a backup zip file here, you could then just copy this zip file to your $JIRAHOME/import/ directory and then in the setup wizard you can import this backup as a means to copy your Jira data into this new empty database.   If the old database or user has in some way become corrupted, these steps should allow you to still maintain your data and get around that problem.

I fully agree and believe this would be the best approach moving forward.

Regards,
Earl

[Henry Auffahrt]

Hi Earl,

thanks for your answer. You are right. I solved the issue with this step. But one day after I posted it here. Sorry that I forget to inform you guys here in the community.

I´m happy that I read this post again and it helped me :)

Thanks a lot.

Henry