Bitbucket HA not working on Ubuntu-16

[John Test]

OS: Ubuntu-16 LTS
Bitbucket server: 4.11.0
Openstack VMs
Openstack Juno (Version: 2014.2.3) 

I am setting up Bitbucket in COLD STANDBY HA following this guide:
https://confluence.atlassian.com/bitbucketserver045/administering-bitbucket-server/advanced-actions/high-availability-for-bitbucket-server

I have external postgres DB

FIRST NODE

Brought up first node, got the error
"Required plugin com.atlassian.stash.ssh-plugin has failed to start"
Tried all solutions here:
https://confluence.atlassian.com/bitbucketserverkb/required-plugin-com-atlassian-stash-ssh-plugin-has-failed-to-start-is-displayed-when-accessing-the-bitbucket-server-web-interface-779171299.html

Did not help. Scrapped it and re-deployed. Now everything came up.

SECOND NODE

Installed Bitbucket server.
Made sure service is stopped.
Synced contents  of $HOMEDIR only (not $INSTALLDIR) from FIRST NODE to this node
Stopped atlbitbucket service on FIRST NODE 
Started atlbitbucket service on SECOND NODE 

I've now tried maybe 10 times and either I get this
"Required plugin com.atlassian.stash.ssh-plugin has failed to start"

or BLANK PAGE when trying to access Bitbucket web UI

Also in between scrap and re-install, even though Bitbucket service is stopped, it still holds onto the ports and shows it is listening on some of the various ports that it opens.

Any ideas on how to fix these problems?

Could this be related to Bitbucket not working well with SystemD, if so is it better to revert to Ubuntu-14 ?

Thanks

[John Test]

Thanks Felix. I am now using the one found here
https://confluence.atlassian.com/bitbucketserver/running-bitbucket-server-as-a-linux-service-776640157.html

[Felix]

Hi John, 

the installer does indeed not create the systemd service. There is an outstanding issue to provide systemd unit files in the Bitbucket Server distribution (https://jira.atlassian.com/browse/BSERV-9148).

From your original question I assumed you had created these yourself (you can check Google for a valid example of a Bitbucket Server systemd unit file). I believe systemd can and will generate unit files on the fly for init.d services, which is, I think, how you are currently launching Bitbucket. This means, however that you can't apply custom properties to the service (such as TasksAccounting). 

If you wish to use systemd with custom properties, you will need to manually configure the unit file until the aforementioned issue (BSERV-9148) is resolved, or apply the property to the systemd defaults.

Felix

[John Test]

Felix, I was just looking into this and by default Bitbucket installer doesn't create atlbitbucket.service file in /etc/systemd/system or /lib/systemd/system

If I create the file with these permissions
-rwxr-xr-x 1 root root   32 Nov 14 16:31 atlbitbucket.service

cat atlbitbucket.service
[Service]
TasksAccounting=false

 
systemctl status atlbitbucket

● atlbitbucket.service
   Loaded: error (Reason: Invalid argument)
   Active: active (exited) since Mon 2016-11-14 16:31:07 UTC; 3min 14s ago

Nov 14 16:32:16 bitbucket-server.local systemd[1]: atlbitbucket.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

Nov 14 16:32:16 bitbucket-server.local systemd[1]: atlbitbucket.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

Is this a bug and should the installer be creating this file?

[John Test]

Kindly appreciated. Thanks Felix!

[Felix]

You have 2 options for the TasksMax issue: either disable systemd's TasksAccounting, which keeps track of the number of tasks a service may spawn, or set TasksMax to a higher limit. 

Both of these changes are made in the systemd unit definition file for Bitbucket (/etc/systemd/system/atlbitbucket.service).

In the [Service] section, add either: 

TasksAccounting=false

or

TasksMax=12288

Felix

[John Test]

Felix, thank you. Can you please advise how to set this and/or any other limits for Bitbucket?

[Felix]

Hi John, 

in order to help you troubleshoot this, it would help if you attached some of the log messages from the atlassian-bitbucket.log when Bitbucket fails to start cleanly.

Re: systemd, Bitbucket Server will work with systemd, provided the limits are set correctly. You may run into issues scaling a production instance with Ubuntu's default limits under heavy load. One such limit Bitbucket Server may hit is the TasksMax limit, which in Ubuntu 16 LTS is, I believe, 512.

Felix