Error with plugins when installing jira on linux server

[Admin]

Details:

• Ubuntu 18.04 LTS 64bit
• Installing jira software 8.1.0
• Postgres 9.6.13
• Installing following the guide https://confluence.atlassian.com/adminjiraserver071/installing-jira-applications-on-linux-802592173.html
• Response of ulimit -a:

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1546431
max locked memory (kbytes, -l) 16384
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 62987
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Problem:

After configuring the database connection jira proceeds to start up, but everytime fails with 

System plugins failed to start.

I did try the solutions proposed in https://confluence.atlassian.com/jirakb/troubleshooting-jira-startup-failed-error-394464512.html, without luck. Almost every solution proposed that I found was setting:

-Datlassian.plugins.enable.wait=300

 I tried to changed to a higher number and to a lower without success.

I did also try to increase the maximun running processes following this link https://confluence.atlassian.com/jirakb/jira-applications-crash-due-to-outofmemoryerror-unable-to-create-new-native-thread-224397455.html because when running jira in foreground, while starting the plugins an OutOfMemoryError ocurred.

I don't know what else to try.

[Admin · Accepted solution]

We ended up migrating to the cloud instead of to our own server, thanks for the responses

[Dave Theodore [Coyote Creek Consulting]]

Make sure that all of the files and directories are owned by the user that Jira is running as. The easiest way to do so, as root:

chown -R <jirauser>:<jiragroup> /path/to/jira-application

then

chown -R <jirauser>:<jiragroup> /path/to/jira-home-directory

From there, become the user that Jira runs as and try starting it up. I'd recommend deleting or moving all the old log files, then starting it up. This will isolate the logs to only this one startup.  If it still doesn't start, you can read through the logs and the stack traces should be helpful.

Pro tip to start up Jira:

/path/to/jira/bin/start-jira.sh;tail -f /path/to/jira/logs/catalina.out

This will start Jira and immediately start tailing the catalina.out log file so you can watch the whole startup. Good luck! 

[Admin]

It didn't work here is the error of the catalina.out log file, I don't know if I can upload the full log anywhere for you to read it.

 Database configuration OK

___ Database Configuration _________________

Loading entityengine.xml from : file:/opt/atlassian/jira/atlassian-jira/WEB-INF/classes/entityengine.xml
Entity model field type name : postgres72
Entity model schema name : public
Database Version : PostgreSQL - 9.6.13
Database Driver : PostgreSQL Native Driver - PostgreSQL 9.4.1212
Database URL : jdbc:postgresql://localhost:5432/jira
Database JDBC config : postgres72 jdbc:postgresql://localhost:5432/jira

2019-05-14 15:56:30,881 JIRA-Bootstrap INFO [c.a.jira.startup.JiraStartupLogger] 

___ Starting the JIRA Plugin System _________________

ERROR: Bundle org.eclipse.gemini.blueprint.extender [14] EventDispatcher: Error during dispatch. (java.lang.OutOfMemoryError: unable to create new native thread)
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)

Thanks for the response!

[Dave Theodore [Coyote Creek Consulting]]

Can you post your -Xms and -Xmx values in the output of `ps`? 

Also, verify that you are running a supported Java.  In the output of `ps,` verify that this Java is the one that you think you are running.

[Admin]

Is this what you mean for the output of `ps`?

root@h2835342:~# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 77356 5768 ? Ss may14 0:01 init -z
root 2 0.0 0.0 0 0 ? S may14 0:00 [kthreadd/283534]
root 3 0.0 0.0 0 0 ? S may14 0:00 [khelper]
root 73 0.0 0.1 115496 14812 ? Ss may14 0:01 /lib/systemd/systemd-journald
root 133 0.0 0.0 42092 1912 ? Ss may14 0:00 /lib/systemd/systemd-udevd
systemd+ 135 0.0 0.0 71832 3296 ? Ss may14 0:00 /lib/systemd/systemd-networkd
root 143 0.0 0.0 32536 1496 ? Ss may14 0:00 /usr/sbin/cron -f
syslog 145 0.0 0.0 189012 2156 ? Ssl may14 0:00 /usr/sbin/rsyslogd -n
message+ 149 0.0 0.0 47616 1812 ? Ss may14 0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 164 0.0 0.0 62004 2956 ? Ss may14 0:00 /lib/systemd/systemd-logind
root 318 0.0 0.0 100632 1420 ? Ss may14 0:00 /usr/sbin/saslauthd -a pam -c -m /var/run/saslauthd -n 2
root 319 0.0 0.0 100632 1028 ? S may14 0:00 /usr/sbin/saslauthd -a pam -c -m /var/run/saslauthd -n 2
root 328 0.0 0.0 72284 3392 ? Ss may14 0:00 /usr/sbin/sshd -D
root 338 0.0 0.0 24176 1048 ? Ss may14 0:00 /usr/sbin/xinetd -pidfile /run/xinetd.pid -stayalive -inetd_compat -inetd_ipv6
postgres 393 0.0 0.1 316040 16544 ? S may14 0:08 /usr/lib/postgresql/9.6/bin/postgres -D /var/lib/postgresql/9.6/main -c config_file=/etc/postgresql/9.6/main/postgresql.conf
postgres 434 0.0 0.6 316552 51096 ? Ss may14 0:01 postgres: 9.6/main: checkpointer process
postgres 435 0.0 0.0 316040 3312 ? Ss may14 0:01 postgres: 9.6/main: writer process
postgres 436 0.0 0.0 316040 6468 ? Ss may14 0:01 postgres: 9.6/main: wal writer process
postgres 437 0.0 0.0 316440 3188 ? Ss may14 0:02 postgres: 9.6/main: autovacuum launcher process
postgres 438 0.0 0.0 171568 2484 ? Ss may14 0:05 postgres: 9.6/main: stats collector process
root 494 0.0 0.0 67364 1712 ? Ss may14 0:00 /usr/lib/postfix/sbin/master -w
postfix 496 0.0 0.0 69464 2768 ? S may14 0:00 qmgr -l -t unix -u
jira 18592 0.5 16.8 6193784 1414052 ? Sl may14 7:32 /opt/atlassian/jira/jre//bin/java -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.lo
postfix 23008 0.0 0.0 69416 2744 ? S 13:45 0:00 pickup -l -t unix -u -c
root 23141 0.0 0.0 13008 860 ? Ss+ 14:28 0:00 /sbin/agetty -o -p -- \u --noclear tty2 linux
root 23142 0.0 0.0 17172 924 ? Ss+ 14:28 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
postgres 23169 0.0 0.0 316944 4708 ? Ss 14:36 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48574) idle
postgres 23170 0.0 0.0 316944 4708 ? Ss 14:36 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48576) idle
postgres 23181 0.0 0.0 316944 4700 ? Ss 14:41 0:00 postgres: 9.6/main: xci jira 127.0.0.1(58126) idle
postgres 23182 0.0 0.0 316944 4708 ? Ss 14:41 0:00 postgres: 9.6/main: xci jira 127.0.0.1(58128) idle
postgres 23183 0.0 0.0 316944 4704 ? Ss 14:41 0:00 postgres: 9.6/main: xci jira 127.0.0.1(58130) idle
postgres 23204 0.0 0.0 316944 4704 ? Ss 14:46 0:00 postgres: 9.6/main: xci jira 127.0.0.1(39224) idle
postgres 23205 0.0 0.0 316944 4708 ? Ss 14:46 0:00 postgres: 9.6/main: xci jira 127.0.0.1(39226) idle
postgres 23206 0.0 0.0 316944 4708 ? Ss 14:46 0:00 postgres: 9.6/main: xci jira 127.0.0.1(39228) idle
postgres 23218 0.0 0.0 316944 4704 ? Ss 14:51 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48304) idle
postgres 23219 0.0 0.0 316944 4704 ? Ss 14:51 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48306) idle
postgres 23220 0.0 0.0 316944 4708 ? Ss 14:51 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48308) idle
postgres 23234 0.0 0.0 316944 4708 ? Ss 14:56 0:00 postgres: 9.6/main: xci jira 127.0.0.1(57446) idle
postgres 23235 0.0 0.0 316944 4712 ? Ss 14:56 0:00 postgres: 9.6/main: xci jira 127.0.0.1(57450) idle
postgres 23236 0.0 0.0 316944 4708 ? Ss 14:56 0:00 postgres: 9.6/main: xci jira 127.0.0.1(57452) idle
postgres 23248 0.0 0.0 316944 4700 ? Ss 15:01 0:00 postgres: 9.6/main: xci jira 127.0.0.1(38608) idle
postgres 23249 0.0 0.0 316944 4700 ? Ss 15:01 0:00 postgres: 9.6/main: xci jira 127.0.0.1(38610) idle
postgres 23250 0.0 0.0 316944 4708 ? Ss 15:01 0:00 postgres: 9.6/main: xci jira 127.0.0.1(38612) idle
root 23254 0.0 0.0 97176 4144 ? Ss 15:02 0:00 sshd: root@pts/0
root 23268 0.0 0.0 22820 2372 pts/0 Ss+ 15:02 0:00 -bash
postgres 23293 0.0 0.0 316944 4708 ? Ss 15:06 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48408) idle
postgres 23294 0.0 0.0 316944 4704 ? Ss 15:06 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48410) idle
postgres 23295 0.0 0.0 316944 4708 ? Ss 15:06 0:00 postgres: 9.6/main: xci jira 127.0.0.1(48412) idle
root 23302 0.0 0.0 97176 4140 ? Ss 15:07 0:00 sshd: root@pts/1
root 23316 0.0 0.0 22820 2332 pts/1 Ss 15:07 0:00 -bash
root 23336 0.0 0.0 38660 1760 pts/1 R+ 15:08 0:00 ps aux

Also I'm using the linux installer from Jira so I don't have any version of Java installed.

[Admin]

In the catalina.out log there is a part which show Xms and Xmx I don't know if is what you asked, the values are -Xms384m -Xmx2048m

 ___ Environment _____________________________

JIRA Build : 8.1.0#801000-sha1:2e1cd1bb771978cda2c5e8f3f10539ab180613f6
Build Date : Thu Apr 04 00:00:00 CEST 2019
JIRA Installation Type : Standalone
Application Server : Apache Tomcat/8.5.35 - Servlet API 3.1
Java Version : 1.8.0_181 - Oracle Corporation
Current Working Directory : /opt/atlassian/jira
Maximum Allowable Memory : 1820MB
Total Memory : 911MB
Free Memory : 845MB
Used Memory : 66MB
Memory Pool: Code Cache : Code Cache: init = 33554432(32768K) used = 9334144(9115K) committed = 33554432(32768K) max = 536870912(524288K)
Memory Pool: Metaspace : Metaspace: init = 0(0K) used = 22562624(22033K) committed = 23330816(22784K) max = -1(-1K)
Memory Pool: Compressed Class Space : Compressed Class Space: init = 0(0K) used = 2606728(2545K) committed = 2883584(2816K) max = 1073741824(1048576K)
Memory Pool: PS Eden Space : PS Eden Space: init = 100663296(98304K) used = 53137688(51892K) committed = 656408576(641024K) max = 656408576(641024K)
Memory Pool: PS Survivor Space : PS Survivor Space: init = 16777216(16384K) used = 0(0K) committed = 30408704(29696K) max = 30408704(29696K)
Memory Pool: PS Old Gen : PS Old Gen: init = 268435456(262144K) used = 21882168(21369K) committed = 268435456(262144K) max = 1431830528(1398272K)
JVM Input Arguments : -Djava.util.logging.config.file=/opt/atlassian/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms384m -Xmx2048m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xloggc:/opt/atlassian/jira/logs/atlassian-jira-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCCause -Dignore.endorsed.dirs= -Dcatalina.base=/opt/atlassian/jira -Dcatalina.home=/opt/atlassian/jira -Djava.io.tmpdir=/opt/atlassian/jira/temp
Java Compatibility Information : JIRA version = 8.1.0, Java Version = 1.8.0_181

[Dave Theodore [Coyote Creek Consulting]]

Hmmm... you have 2GB allocated to heap, which should be enough to start up.  How much physical memory is on the server? You can see this by running `free`I would recommend a minimum of 8GB of memory, given that you are also running Postgres on the same server. Output will be something like:

$ free

             total       used       free     shared    buffers     cached

Mem:      16226128   14645592    1580536     122636       1188    6085984

-/+ buffers/cache:    8558420    7667708

Swap:      4055036          0    4055036

$ 

Also verify that you have a 64-bit Java.  In your case, the command will be:

/opt/atlassian/jira/jre/bin/java -version

[Admin]

I do have 8GB:

root@h2835342:~# free
             total      usado      libre     compartido     búfer/caché     disponible
Memoria:   8388608    1786016    5900360          26036          702232        6576556
Swap:            0          0          0

The java version:

root@h2835342:~# /opt/atlassian/jira/jre/bin/java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

[Dave Theodore [Coyote Creek Consulting]]

Is this a new installation with no data? 

Try raising the heap memory values and see if it starts. I'm surprised that you would need to do this, but it might get you past this problem.  In /opt/atlassian/jira/setenv.sh change the following lines to the values that I have indicated and see what happens:

JVM_MINIMUM_MEMORY="3072m"

JVM_MAXIMUM_MEMORY="3072m"

[Admin]

Yes, this is a new installation with no data. I applied the changes and restarted jira without luck. Still gives the plugin error, always the same plugin: