Jira 7 upgrade Startup Failed

[alextechadventures]

Greetings,

I tried to upgrade to JIRA 7 Software today from 6.4.6 + agile. In the end got error

The following plugins are required by JIRA, but have not been started:

• Gadget Directory Plugin (com.atlassian.gadgets.directory)
• Atlassian JIRA - Plugins - Gadgets Plugin (com.atlassian.jira.gadgets)
• Atlassian JIRA - Plugins - Issue Search (com.atlassian.jira.jira-issue-nav-plugin)
• Atlassian JIRA - Plugins - Quick Edit Plugin (com.atlassian.jira.jira-quick-edit-plugin)
• Atlassian JIRA - Plugins - REST Plugin (com.atlassian.jira.rest)

Things checked:

• java version compatibility
• all plugins are up to date prior to upgrade. Only third party I have is JIRA Client from almworks.
• all steps in migration hub.
• License for JIRA 6 valid until January 2016

Things tried:

• increase -Datlassian.plugins.enable.wait=300 from original 60. Longer timeout wait did not help
• increase memory limit to 1024 from original 768
• deleted plugin caches .bundled-plugins and .osgi-plugins
• with and without apache proxy

Log is huge so I hosted it at http://pastebin.com/im3tAWA1

Anything I missed in the upgrade process? Thanks.

[alextechadventures · Accepted solution]

Got answer from Atlassian support. The problem is there are some dependencies that must have been optional in 6.4 that are now required in 7. If these dependencies have been disabled due to some temporary load issue in the past, the installer keeps their state as disabled but then cannot turn on after upgrade.  To force turn them all back on execute

update pluginstate set pluginenabled='true';

in your database. Don't worry if there are incompatible plugins. JIRA 7 will disable them on startup and you can update them or disable again in control panel after it turns on.

[anon]

Thank you! This worked for me after days of pulling my hair out and restoring from backups several times!

[Tim Smith]

This worked for me too. Thank you, saved me a lot of time.

[Dave Coleman]

You are a lifesaver. Thanks.

[Henrik Poulsen]

This worked for me too. Thanks final got upgrade to 7.0.5

[moffet]

Yes, this solved that problem for me too. Excellent, thank you!

[Hussein]

Successfully upgraded from 6.4 to 7.1 with this hint. 

[Stanislav Karamyshev]

Thanks a lot!

[Luvo]

where can I find this on my instance 

update pluginstate set pluginenabled='true'

[Mike Curwen NL]

This is just a metoo, but I was slightly more picky about what I enabled: 

update pluginstate set pluginenabled='true' where pluginkey like '%atlassian%'

The only thing I had disabled from atlas was the importer core and a couple things like github and redmine, and the welcome gadget.   enabling %atlassian% worked for me. 

[Юрий Цапков]

Thanks! It's truly works fine

[Trupti Kamble]

Hello,

I am facing same issue while upgrading from 7.3.6 to 7.10.2. 

System plugins failed to start: We haven't been able to start all the required system plugins

I tried all above solutions:

1. Removed .osgi and .bundled plugins

1. Increased memory by 1024 in setenv.sh

2. Added -Datlassian.plugins.enable.wait=300

3. Removed "Plugins" folder and did restart

4. Added 

update pluginstate set pluginenabled='true'

Though it's not working.

Can somebody please save me?

Logs: 2019-03-05 15:21:37,098 JIRA-Bootstrap ERROR [c.a.plugin.manager.PluginEnabler] Unable to start the following plugins due to timeout while waiting for plugin to enable: com.atlassian.scala.plugins.scala-2.11-provider-plugin,com.atlassian.jira.plugins.jira-importers-github-plugin,com.atlassian.feedback.jira-feedback-plugin,com.atlassian.mywork.mywork-jira-provider-plugin,com.atlassian.jira.rest,com.atlassian.jirawallboard.atlassian-wallboard-plugin,com.atlassian.plugins.atlassian-awareness-capability,com.atlassian.jira.plugins.jira-transition-triggers-plugin,com.atlassian.plugins.jira-html5-attach-images,com.atlassian.gadgets.opensocial,com.atlassian.oauth.atlassian-oauth-service-provider-spi,com.atlassian.jira.jira-fileviewer-plugin,com.atlassian.troubleshooting.plugin-jira,tac.jira core.languages.ro_RO,com.atlassian.jira.plugins.jira-quicksearch-plugin,com.atlassian.plugins.atlassian-nps-plugin,tac.jira core.languages.ru_RU,com.atlassian.applinks.applinks-plugin,tac.jira core.languages.pt_BR,com.atlassian.labs.hipchat.hipchat-for-jira-plugin,tac.jira core.languages.da_DK,com.atlassian.oauth.admin,com.atlassian.jira.plugins.inline-create.jira-inline-issue-create-plugin,com.atlassian.upm.upm-application-plugin,com.atlassian.jira.jira-

[Mahantesh Chanapannavar]

Whenever there is a reboot of JIRA server, the issue comes up - JIRA Startu Failed.

Majority times it will be with the statement "The following plugins are required by JIRA, but have not been started:" and I do the following things to make JIRA ON.

1. Stop the JIRA

2. Delete the "Pluggins" folder

3. Restart the JIRA and it works

Is there any permanent solution where in future, even if there is a restart/reboot of server pc, such errors or issues won't come up? 

[Nic Brough -Adaptavist-]

When a computer starts up, it is thrashed to death by every service and program trying to start itself at the same time.

JIRA is now very heavy on starting itself, and the problem you're getting is almost certainly that it can't get all the resources it needs for a successful startup, because everything else on your server needs them too while booting.

Your starting options are

• Get a more powerful server
• Give JIRA more resources
• Stagger the JIRA startup so that it only tries to start after the other things have mostly finished
• Give JIRA more time to get started (change the "plugins wait" time as above)

I'd also do some housekeeping - check if your system really does need all the add-ons you have

[Mahantesh Chanapannavar]

Thank you Nic for the quick reply and the options to set.

I have set the wait time and will keep JIRA under observation. Lets see if it solves the issue or not.

[Deleted user]

I got the same Issue on our Testsystem.
We have Lizenz for JIRA and the JIRA Agile Plugin.

I tried many ways to update the system but every try I got the error that JIRA has been locked and could not start.

This Time I deinstalled the JIRA Agile Plugin and run the update.
Now I got JIRA Software installed

[Julian Radünz]

Nothing but this did help:

• Set up a completely new machine "Sandbox"
• Install empty JIRA 7. It starts successfully.
• Copy DB from Prod JIRA to "Sandbox" JIRA
• Point "Sandbox" JIRA to copied Prod DB
• Upgrade task will be completed successfully
• Shutdown, copy missing plugins to jira_home
• Startup - everything works fine

[Neil Taylor]

I had this issue; I tried the upgrade again and this time I disabled all plugins prior to upgrading and it worked like a charm - maybe try this is you haven't and you have the available backups to do this

[alextechadventures]

I want to try this. Of all the plugins listed in startup logs only a few are controllable through admin panel. Is it sufficient to disable just those or do I have to go deeper prior to upgrade?

[Neil Taylor]

I disabled just the user installed plugins - I didn't go into system plugins - basically just disabled what would get disabled if you put it into "Safe Mode" - good luck!

[alextechadventures]

No, didn't help. Disabled the 2 user installed plugins I had - Agile and Client - and put it to safe mode but those 5 built in plugins still do not want to turn on.

[Neil Taylor]

Hmm. Interesting - I had the exact list of plugins that didn't turn on. Though I had significantly more user installed plugins -- 10+ I'm running Java min at 2GB, max at 8GB with these options: -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -Dcatalina.home=C:\NGP\JIRA -Dcatalina.base=C:\NGP\JIRA -Djava.endorsed.dirs=C:\NGP\JIRA\endorsed -Djava.io.tmpdir=C:\NGP\JIRA\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\NGP\JIRA\conf\logging.properties -XX:MaxPermSize=1024m -Datlassian.plugins.enable.wait=300 -Djira.jelly.on=true -Dfile.encoding=UTF8

[alextechadventures]

Interesting that in my case I had to force turn everything on, instead of off.

[Julian Radünz]

I have exactly the same problem and have been trying the same things to solve it.

No success yet.

Neither increasing memory significantly nor atlassian.plugins.enable.wait increase helps.
Restoring the backup jira_server directory leads to a working JIRA 6.

[Nic Brough -Adaptavist-]

Well, the main error in the original question was actually that the disk permissions have been messed up outside JIRA. But, what are you trying to run here? Just an empty JIRA 7? Or an upgrade? If it's an upgrade, what does the log say about the upgrade process? If it's not, then what does the log say about the startup, how much memory have you got and what's the CPU like?

[Julian Radünz]

Hi Nic, it is an upgrade from JIRA 6.4. The JIRA log doesn't say anything about the upgrade process - I guess the upgrade part (database migration,...) starts after successful JIRA startup. CPU: 2 core RAM: 6 Gb JIRA Logfile: https://s3-eu-west-1.amazonaws.com/comoqm/catalina.out Installation logfile: https://s3-eu-west-1.amazonaws.com/comoqm/installation.log The Installation Log is showing an error at the end which seems to be related to the problem, but not sure what it means.

[Nic Brough -Adaptavist-]

I can't open those files, so I'm a bit stuck. On the memory, how much are you allocating to JIRA?

[Julian Radünz]

--------------------------------------------------------------------------------- Heap memory : Used: 296 MiB. Committed: 856 MiB. Max: 1470 MiB Non-heap memory : Used: 191 MiB. Committed: 209 MiB. Max: 1264 MiB --------------------------------------------------------------------------------- TOTAL : Used: 487 MiB. Committed: 1065 MiB. Max: 2734 MiB ---------------------------------------------------------------------------------

[Julian Radünz]

This is the part from the installation.log where the first error begins: [INFO] com.install4j.runtime.beans.actions.files.CopyFileAction [ID 65555]: Execute action Property shared: false Property uninstallMode: If created Property mode: 644 Property overwriteMode: Always ask except for update Property destinationFile: /home/jira/jira_server/bin/msvcr71.dll Property delay: false Property triggerReboot: true Property recursive: true Property directoryMode: 755 Property showProgress: false Property showFileNames: true Property files: [/home/jira/jira_server/jre/bin/msvcr71.dll] Property fileFilter: null Property directoryFilter: null Property rollbackSupported: true The file "/home/jira/jira_server/jre/bin/msvcr71.dll " does not exist [ERROR] com.install4j.runtime.beans.actions.files.CopyFileAction [ID 65555]: Execute action not successful [INFO] com.install4j.runtime.beans.actions.files.SetModeAction [ID 66248]: Execute action Property mode: u+x Property recursive: true Property showProgress: false Property showFileNames: true Property files: [bin] Property fileFilter: I4jScript_Internal_357 Property directoryFilter: null Property rollbackSupported: true Setting mode for /home/jira/jira_server/bin Setting mode for /home/jira/jira_server/bin/shutdown.sh Setting mode for /home/jira/jira_server/bin/setclasspath.sh Setting mode for /home/jira/jira_server/bin/daemon.sh Setting mode for /home/jira/jira_server/bin/startup.sh Setting mode for /home/jira/jira_server/bin/check-java.sh Setting mode for /home/jira/jira_server/bin/digest.sh Setting mode for /home/jira/jira_server/bin/configtest.sh Setting mode for /home/jira/jira_server/bin/tool-wrapper.sh Setting mode for /home/jira/jira_server/bin/setup_user.sh Setting mode for /home/jira/jira_server/bin/config.sh Setting mode for /home/jira/jira_server/bin/setenv.sh Setting mode for /home/jira/jira_server/bin/user.sh Setting mode for /home/jira/jira_server/bin/catalina.sh Setting mode for /home/jira/jira_server/bin/install_linux_service.sh Setting mode for /home/jira/jira_server/bin/apr Setting mode for /home/jira/jira_server/bin/stop-jira.sh Setting mode for /home/jira/jira_server/bin/version.sh Setting mode for /home/jira/jira_server/bin/start-jira.sh Execute action successful [INFO] com.install4j.runtime.beans.actions.text.ModifyTextFileAction [ID 66786]: Execute action Property searchValue: # !INSTALLER SET JAVA_HOME Property replaceValue: JAVA_HOME="/home/jira/jira_server/jre/"; export JAVA_HOME Property logReplacement: true Property failIfNoReplacement: false Property escapeForPropertyFile: false Property encoding: null Property files: [bin/setenv.sh] Property rollbackSupported: true Backing up /home/jira/jira_server/bin/setenv.sh Execute action successful [INFO] com.install4j.runtime.beans.actions.text.ModifyTextFileAction [ID 59970]: Execute action Property searchValue: <Connector port="8080" Property replaceValue: <Connector port="8080" Property logReplacement: true Property failIfNoReplacement: false Property escapeForPropertyFile: false Property encoding: null Property files: [conf/server.xml] Property rollbackSupported: true Backing up /home/jira/jira_server/conf/server.xml Execute action successful [INFO] com.install4j.runtime.beans.actions.text.ModifyTextFileAction [ID 59971]: Execute action Property searchValue: <Server port="8005" Property replaceValue: <Server port="8005" Property logReplacement: true Property failIfNoReplacement: false Property escapeForPropertyFile: false Property encoding: null Property files: [conf/server.xml] Property rollbackSupported: true Backing up /home/jira/jira_server/conf/server.xml Execute action successful [INFO] com.install4j.runtime.beans.actions.text.RegexTextFileAction [ID 69086]: Execute action Property replacement: jira.home = /home/jira/jira_home Property logReplacement: true Property regex: (?m)jira.home\s*=.*?$ Property replaceAll: true Property quoteVariables: true Property failIfNoReplacement: true Property escapeForPropertyFile: true Property encoding: null Property files: [atlassian-jira/WEB-INF/classes/jira-application.properties] Property rollbackSupported: true Backing up /home/jira/jira_server/atlassian-jira/WEB-INF/classes/jira-application.properties Execute action successful [INFO] com.install4j.runtime.beans.actions.text.RegexTextFileAction [ID 65935]: Execute action Property replacement: jira.home = Property logReplacement: true Property regex: jira.home \\= Property replaceAll: true Property quoteVariables: false Property failIfNoReplacement: false Property escapeForPropertyFile: false Property encoding: Property files: [atlassian-jira/WEB-INF/classes/jira-application.properties] Property rollbackSupported: true Backing up /home/jira/jira_server/atlassian-jira/WEB-INF/classes/jira-application.properties Execute action successful [INFO] com.install4j.runtime.beans.actions.files.DeleteFileAction [ID 60020]: Execute action Property rollbackSupported: true Property backupForRollback: true Property recursive: false Property showProgress: false Property showFileNames: true Property files: [.install4j/installation.id] Property fileFilter: null Property directoryFilter: null The file "/home/jira/jira_server/.install4j/installation.id " does not exist Execute action successful [INFO] com.install4j.runtime.beans.actions.text.WriteTextFileAction [ID 60019]: Execute action Property text: JIRA181111112259 Property logText: true Property append: true Property encoding: Property escaped: true Property file: .install4j/installation.id Property rollbackSupported: true Execute action successful [INFO] com.install4j.runtime.beans.actions.files.CopyFileAction [ID 67717]: Execute action Property shared: false Property uninstallMode: If created Property mode: 644 Property overwriteMode: Always ask except for update Property destinationFile: bin/setenv.sh.orig Property delay: false Property triggerReboot: true Property recursive: true Property directoryMode: 755 Property showProgress: false Property showFileNames: true Property files: [bin/setenv.sh] Property fileFilter: null Property directoryFilter: null Property rollbackSupported: true Copy /home/jira/jira_server/bin/setenv.sh to /home/jira/jira_server/bin/setenv.sh.orig Install file: /home/jira/jira_server/bin/setenv.sh.orig; size: 3749 bytes; exists: false File options set to: [fileTime: Mon Oct 12 11:45:54 CEST 2015, mode: 644, overwriteMode: Always ask except for update, shared: false, uninstallMode: If created, delayIfNecessary: false] Execute action successful [INFO] com.install4j.runtime.beans.actions.control.RunScriptAction [ID 67715]: Execute action Property rollbackSupported: false Property rollbackScript: null Property script: I4jScript_Internal_391 Execute action successful [INFO] com.install4j.runtime.beans.actions.files.ExtractZipFileAction [ID 70546]: Execute action Property mode: 644 Property destinationDirectory: /home/jira/jira_server/.. Property fileFilter: I4jScript_Internal_394 Property zipFile: /home/jira/jira_server-6.4.7-back.zip Property showProgress: true Property dirMode: 755 Property rollbackSupported: true Execute action successful [INFO] com.install4j.runtime.beans.screens.InstallationScreen [ID 5]: Rollback barrier reached command: move 1 screens, executing actions, checking condition [INFO] com.install4j.runtime.beans.screens.CustomizableProgressScreen [ID 59776]: Show screen [INFO] com.install4j.runtime.beans.actions.control.RunScriptAction [ID 66036]: Execute action Property rollbackSupported: false Property rollbackScript: null Property script: I4jScript_Internal_396 [ERROR] com.install4j.runtime.beans.actions.control.RunScriptAction [ID 66036]: Execute action not successful [INFO] com.install4j.runtime.beans.actions.control.RunScriptAction [ID 59788]: Execute action Property rollbackSupported: false Property rollbackScript: null Property script: I4jScript_Internal_398 Execute action successful [INFO] com.install4j.runtime.beans.actions.misc.RunExecutableAction [ID 66249]: Execute action Property arguments: null Property executable: ./bin/start-jira.sh Property rollbackSupported: false Property executionType: Run with the same account as the installer process Property workingDirectory: ./bin Property wait: false Property returnCodeVariable: Property logArguments: true Property stdoutFile: null Property stderrFile: null Property stdinFile: null Property failOnRedirectionError: false Property includeParentEnvironmentVariables: true Property environmentVariables: {} Property showWindowsConsole: false Property keepConsoleWindow: false used working dir: /home/jira/jira_server/./bin Execute action successful [INFO] com.install4j.runtime.beans.actions.control.SetProgressAction [ID 59777]: Execute action Property rollbackSupported: false Property timerPeriod: 200 Property timerMaxPercentValue: 100 Property percentValue: 0 Property progressChangeType: Set indeterminate state Execute action successful [INFO] com.install4j.runtime.beans.actions.control.RunScriptAction [ID 59785]: Execute action Property rollbackSupported: false Property rollbackScript: null Property script: I4jScript_Internal_402 Variable changed: app.launch.error=true[class java.lang.Boolean] [ERROR] com.install4j.runtime.beans.actions.control.RunScriptAction [ID 59785]: Execute action not successful [INFO] com.install4j.runtime.beans.screens.CustomizableProgressScreen [ID 59776]: command: move 1 screens, executing actions, checking condition [INFO] com.install4j.runtime.beans.screens.BannerFormScreen [ID 70143]: Show screen Variable changed: executeLauncherAction=true[class java.lang.Boolean] [INFO] com.install4j.runtime.beans.actions.control.RunScriptAction [ID 68582]: Execute action Property rollbackSupported: false Property rollbackScript: null Property script: I4jScript_Internal_405 Execute action successful [INFO] com.install4j.runtime.beans.screens.BannerFormScreen [ID 70143]: command: finish cleaning up Finished

[Nic Brough -Adaptavist-]

That doesn't look like a standard JIRA installation log, so I'm still a bit stuck. What does the catalina.out say is the problem when you start it?

[Julian Radünz]

Lots of errors. See a new generated catalina.out here: https://paste.ee/p/6Pc5w Most of the errors are thrown by third-party-plugins. The errors disappear when removing the plugins, but the startup error is still the same. No effect when removing all plugins. I even tried it now with 4GB memory and 300s plugin enable time - no success. Example error: 'com.atlassian.jira.gadgets' - 'Atlassian JIRA - Plugins - Gadgets Plugin' failed to load. Application context initialization for 'com.atlassian.jira.gadgets-plugin' has timed out waiting for (&(objectClass=com.atlassian.jira.rest.util.ResponseFactory)(objectClass=com.atlassian.jira.rest.util.ResponseFactory)) It has the following missing service dependencies : &responseFactory of type (&(objectClass=com.atlassian.jira.rest.util.ResponseFactory)(objectClass=com.atlassian.jira.rest.util.ResponseFactory)) &responseFactory of type (&(objectClass=com.atlassian.jira.rest.util.ResponseFactory)(objectClass=com.atlassian.jira.rest.util.ResponseFactory))

[alextechadventures]

If upgrading is so troublesome for some people, is it possible to perform a new clean isolated install and after it starts up change database parameter to point it to the old postgresql database where data is stored to continue working? Its pretty typical for PHP applications, is this possible in Java world?

[Julian Radünz]

Good hint. That's what I'll try next.

[alextechadventures]

Some plugins may have been turned off while in 6 that were not a problem then, but are now. Ensure everything needed is turned on with update pluginstate set pluginenabled='true'; Before or after the upgrade.

[Nic Brough -Adaptavist-]

Well, you do have some problems with timeouts, and that would tempt me to increase the enable.wait to more than 300.  You've got several addons as well, including large ones, so I'd also increase the memory to 1.5Gb.

However, the most important thing is the hideous error at the end that tells you it can't write to the disk

org.apache.catalina.startup.HostConfig.start Unable to create directory for deployment: /opt/atlassian/jira/conf/Catalina/localhost

You need to fix that and start again - check the permissions and disk space.

[alextechadventures]

Thanks for pointing that one out. Should have noticed it. Did not work, however. Giving it permissions to JIRA user only changes the last output lines to 2015-10-10 22:40:14,694 localhost-startStop-1 INFO [c.a.jira.startup.LauncherContextListener] Memory Usage: --------------------------------------------------------------------------------- Heap memory : Used: 304 MiB. Committed: 678 MiB. Max: 740 MiB Non-heap memory : Used: 188 MiB. Committed: 206 MiB. Max: 1264 MiB --------------------------------------------------------------------------------- TOTAL : Used: 493 MiB. Committed: 883 MiB. Max: 2004 MiB --------------------------------------------------------------------------------- 2015-10-10 22:40:15,053 localhost-startStop-1 ERROR [c.a.j.web.dispatcher.JiraWebworkActionDispatcher] ****************************************** JIRA startup failed, JIRA has been locked. ****************************************** 11-Oct-2015 08:59:14.869 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 11-Oct-2015 08:59:14.885 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 110407 ms 2015-10-11 08:59:15,850 Modification Check:thread-1 INFO [c.a.jira.startup.JiraStartupLogger] Gave it 600 seconds and 2048 memory. Disk has 33Gb free. Interesting that JIRA installer would not have all permissions correctly applied. Is that something worth reporting to official support/bug? There are no problems starting up 6.4 by restoring backup directories. There are a lot of useless looking plugins popping up indeed, such as trello import and similar, which I would disable on first opportunity.

[Nic Brough -Adaptavist-]

The permissions are not changed by the installer, that's a function of your OS setup. It just creates the directories you need in the home place you tell it to. The reason for startup failing isn't in the piece of log you've given us, so there's not much we can tell you.