Jira Server won't start, even after restoring a server snapshot taken well before the failure

[bryhoyt]

Our Jira Server installation stopped working yesterday, and refuses to start up. On startup it gives two error messages which don't seem to make sense in the context:
1) It says that it can't create the home directory due to permissions (but the directory exists, and updating the permissions and explicitly setting the correct Jira user does not fix this), and
2) It says there is a version mismatch between the Jira build number 801000 and the database build number 804003. Weirdly, the Jira build number is *lower* than the database number. We have not upgraded, and have certainly not downgraded Jira!!

Occasionally it allows me to get past the error screen and view the dashboard, at which point it shows "Test" in the page title, and no project data.

Most importantly, when we restore a FULL server snapshot taken from two days ago, well before the error occured, Jira still does not start, and gives the same error messages. How can it

All this leads me to believe that a) Jira has somehow decided to point at the wrong database, maybe a test database somewhere, and b) it is making that decision based on something outside of the server configuration because a full snapshot restore does not make it go away. Could it be a license key issue somehow?

How can I troubleshoot this, and what are the possible causes?

[bryhoyt]

Yes, very strange. Below is the permissions on the home directory (and a weird grep of the log that shows a "daemon" user trying to start up... I don't know what that means).

Re your questions about the full server snapshot. The database is very much part of the server snapshot. You can see the path to the h2db file in my `stat` below. I have checked this is the correct database (using some SQL to eyeball some of the tickets) -- it is; it has both our oldest and our most recent data in it.

------------ User and home dir permissions: ------------

root@abc1d83ba7a5:/var/atlassian/application-data/jira# grep user.name log/atlassian-

...

user.name : jira
user.name : jira
user.name : jira
user.name : daemon
user.name : jira

root@abc1d83ba7a5:/var/atlassian/application-data/jira# stat .
...

Access: (0700/drwx------) Uid: ( 2001/ jira) Gid: ( 2001/ jira)
...

root@abc1d83ba7a5:/var/atlassian/application-data/jira# stat data
...

Access: (0700/drwx------) Uid: ( 2001/ jira) Gid: ( 2001/ jira)
...

root@abc1d83ba7a5:/var/atlassian/application-data/jira# stat database/h2db.mv.db
...

Access: (0700/-rwx------) Uid: ( 2001/ jira) Gid: ( 2001/ jira)
...

[Nic Brough -Adaptavist-]

That doesn't answer the question.  What are the permissions on the directories here?

Although there is a second problem you have exposed - you're using an unsupported database, one that can (and usually does) fail catastrophically on zero notice, and gives you all sorts of strange errors when it goes wrong.

I would go find the most recent proper backup of the system and use that to populate a new clean system built on a supported database.

[bryhoyt]

Sorry, I'm a little bit lost. What directories do you want to see the permissions for? The Jira home dir is u+rwx and owned by jira:jira as above.

All the subdirs are identical permissions, per `ls -l` below.

Or is there some other information you're looking for?

root@abc1d83ba7a5:/var/atlassian/application-data/jira# ls -l
total 1013496
drwx------ 2 jira jira 4096 Jun 14 00:20 analytics-logs
-rwx------ 1 jira jira 19441 Oct 6 2019 atlassian-jira.log
-rwx------ 1 jira jira 131 Oct 6 2019 atlassian-jira-security.log
drwx------ 4 jira jira 4096 Jun 12 01:52 caches
drwxr-x--- 2 jira jira 4096 Jun 12 02:33 customisations
drwx------ 4 jira jira 4096 May 9 2019 data
drwx------ 2 jira jira 4096 Jun 7 00:00 database
-rwx------ 1 jira jira 884 May 8 2019 dbconfig.xml
drwx------ 3 jira jira 36864 Jun 14 00:00 export
-rwx------ 1 jira jira 37343 Jun 12 02:19 hs_err_pid340.log
-rwx------ 1 jira jira 37598 Jun 12 02:20 hs_err_pid384.log
-rwx------ 1 jira jira 271174 Mar 24 01:36 hs_err_pid8.log
-rwx------ 1 jira jira 260429 Apr 22 07:17 hs_err_pid9.log
drwx------ 4 jira jira 4096 May 8 2019 import
drwx------ 2 jira jira 4096 Jun 14 07:06 log
drwx------ 2 jira jira 4096 May 8 2019 logos
drwx------ 2 jira jira 4096 May 8 2019 monitor
drwx------ 6 jira jira 4096 May 8 2019 plugins
drwx------ 3 jira jira 4096 May 8 2019 tmp

[Nic Brough -Adaptavist-]

And the rest of the permissions we asked for are?

[bryhoyt]

Sorry, I'm really confused. What other permissions did you ask for?

I *think* I've given you the permissions of the Jira home directory and the 1st level of content, which is what I understood you to be asking for in your first post.

Which parts am I still missing?

[Nic Brough -Adaptavist-]

Your list leaves out the ownership and permissions of the home directory (i.e. the one you are in)

There's another odd looking thing too - why are there logs in this directory there?  They should be in the log directory.

But the data you've given us is suggesting the problem may be with the installation rather than the home directory, as it does look healthy, and not lilke it's going to cause a problem.

Could you show us two more things please?

"ps -ef | grep java" would be great.  I'm specifically looking for the command line parameters that the JVM is starting up with.  It's unlikely that there's a problem in them, but it is worth a check (this one bit me once in the past - spent hours looking for settings in files only to find a it was a command line startup parameter overriding the settings we needed)

The active lines in <jira install>/atlassian-jira/WEB-INF/classes/jira-application.properties

[bryhoyt]

Cool, thanks so much. Home directory permissions (yep, looks ok, as you mentioned):

root@abc1d83ba7a5:/var/atlassian/application-data/jira# stat .
...
Access: (0700/drwx------) Uid: ( 2001/ jira) Gid: ( 2001/ jira)

Your question re the log files -- I'm not 100% sure why these ones are sitting here, but they're not actually the real log files (the newest entry is from 2019).  They must be files that somebody copied when setting Jira up initially last year! The real log files are indeed in log/* as you say they should be.

Java process grep:

> ps -ef | grep java

jira 8 6 6 Jun12 ? 04:40:56 /opt/java/openjdk/bin/java -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx3G -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Djira.home=/var/atlassian/application-data/jira -Datlassian.plugins.startup.options=-fg -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -classpath /opt/atlassian/jira/bin/bootstrap.jar:/opt/atlassian/jira/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp org.apache.catalina.startup.Bootstrap start

jira-application.properties:

cat /opt/atlassian/jira/atlassian-jira/WEB-INF/classes/jira-application.properties 
# Do not modify this file unless instructed. It is here to store the location of the JIRA home directory only and is typically written to by the installer.
jira.home =

Thank you!

[bryhoyt]

For the record, we got our Jira installation working again. The issue was that Jira had attempted to restart at some stage, and another Jira process attempted to start up at the same time (not sure why), and the two interfered with each other and one created a lock file under a different user.

I'm not sure all the details or reasons, but glad it's resolved!

Thanks for your help.

[Nic Brough -Adaptavist-]

On the home directory bit, could you show us the permissions and settings on the top-level home directory, and the 1st level of content, and let us know which account is expected to be running Jira?

The rest of your question is a bit fuzzy.  Not what you are reporting, that's very clear, but in my head, because I'm really struggling to work out how you might get there!

The one thing that really jumps out is, as you say, the version mismatch.  The only ways I know of getting there are:

• Someone  pointed a Jira 8.4.2 installation at a database that an 8.1.0 installation was using and told it to upgrade it. 
• Someone has taken your installation and pointed it at a pre-existing 8.4.2 database that they were using somewhere else.

Your restoration of the full server snapshot is near, if not at, the top of the list of tests to start with.  It leaves us with one question though - where is the database?   Is it part of the snapshot (i.e. running your Jira on the same server as a database server that provides the data store) or separate? 

Oh, lastly, cross off the licence idea.   Absolutely right to check up, but Atlassian's licences are not quite that complicated.  They cap the number of people you have using the system and limit you to using versions of the software released before the end of the license.  I don't think they are the problem here.