Getting 503 Errors when using Apache2 as load balancer in front of jira datacenter cluster

[Niklas Klein]

Hi everyone,

Me and a colleague are currently installing a JIRA datacenter cluster for educational purposes. The system consists of:

• 1 Data-Server which hosts a postgresql db and an NFS-Server with a share for the shared home folder
• 3 Servers with JIRA datacenter as nodes
• 1 Server with an Apache2 webserver as a loadbalancer

We are roughly following those two guides

https://confluence.atlassian.com/adminjiraserver/set-up-a-jira-data-center-cluster-993929600.html

https://confluence.atlassian.com/enterprise/jira-data-center-load-balancer-examples-781200827.html

But now the issue: We can reach all of the three JIRA nodes via the Apache2 and we get the Cookie for the Session. However, we get a lot of 503 Errors when accesing each of the JIRA dashboards or the setup wizard.

sdsdsd

However, when I access the JIRA node directly via the Tomcat we get a perfectly fine webpage

The same happens when I remove the balancer part of my vhost apache2 config and replace it with a direct path to only one of the JIRA nodes. Therefore we believe the apache2 and especially the balancer configuration shredds the site.

Here is the vhost config:

<VirtualHost *:80>

ServerName jira-cluster.foobar.local

ProxyRequests Off
ProxyPreserveHost On
ProxyVia Off

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://jira-http>
BalancerMember http://10.0.0.11:8080/jira route=synnode1
BalancerMember http://10.0.0.12:8080/jira route=synnode2
BalancerMember http://10.0.0.13:8080/jira route=synnode3
ProxySet stickysession=ROUTEID
Require all granted
</Proxy>

# ProxyPass /jira http://10.0.0.11:8080/jira
ProxyPass /jira balancer://jira-http/

# ProxyPassReverse /jira http://10.0.0.11:8080/jira
ProxyPassReverse /jira balancer://jira-http/

<Location /jira >
Require all granted
</Location>

RewriteEngine on

</VirtualHost>

And here is the connector from the tomcats server.xml

<Connector port="8080" relaxedPathChars="[]|" relaxedQueryChars="[]|{}^\`&quot;&lt;&gt;"
maxThreads="150" minSpareThreads="25" connectionTimeout="20000" enableLookups="false"
maxHttpHeaderSize="8192" protocol="org.apache.coyote.http11.Http11NioProtocol" useBodyEncodingForURI="true"
redirectPort="80" acceptCount="100" disableUploadTimeout="true" bindOnInit="false" proxyName="jira-cluster.foobar.local" scheme="http"/>

Any solutions or ideas are appreciated and if this is the wrong place for such a question, please point me to where else I can ask :)

[Niklas Klein · Accepted solution]

To finish this up. Looks like the missing ProxyPort was the Issue. It was creating "Cross Site Request Forgery" Errors in the logs and after I added it, the website was shown as expected.

[Kurt Klinner]

@Niklas Klein 

Hi Niklas

welcome to the Atlassian community.

Can you check if moving the stickysession attribute to the proxypass directive as described in 

https://confluence.atlassian.com/enterprise/jira-data-center-load-balancer-examples-781200827.html

The context path is set to /jira in your server.xml - right?

Cheers

Kurt

[Niklas Klein]

Hey Kurt,

thanks for your answer. Context path was set correct and moving the stickysession attribute did not help.

What did help though was fixing the connector attributes in the server.xml as in https://confluence.atlassian.com/kb/cross-site-request-forgery-csrf-protection-changes-in-atlassian-rest-779294918.html. Especially adding the proxyPort. I also changed all permissions in the JIRA home directory to 755 (was 750 before) so that could have also played a role in my solution but I will verify when I spin up the environment again (working with vagrant VMs).

Greetings,

Niklas

[Kurt Klinner]

@Niklas Klein 

Great catch did also not realize that the proxyport was missing.

Keep me posted on your test

All the best

Kurt