How to configure Apache for Confluence 6.2 Synchrony cluster

[Areg Vrtanesyan _Work_]

Hi

I am wondering if someone has managed to configure Apache for Synchrony Cluster proxy in?

I have well established configuration for Jira and Confluence DC cluster based on Apache 2.4 + MOD_JK + AjpNio connector.

Now I would like to add Synchrony in it but I can't find out any sample on Atlassian for that.

There is only one reference to single server option.

Regards,
Areg

[Areg Vrtanesyan _Work_ · Accepted solution]

Hi All

With help of my collegaues, PS Yilin Mo and TAM Kevin we have solved this puzzle.

So the steps are involving:

• Use Apache 2.4.x
• Download, compile and install MOD_JK - https://tomcat.apache.org/download-connectors.cgi
• Configure and terminate SSL in Apache
• Configure MOD_JK for Confluence Nodes:

#===================== Worker Section Start =====================
# Confluence Load Balancer Worker
worker.list=confluence-lb
worker.confluence-lb.type=lb
worker.confluence-lb.balance_workers=confluence-n1,confluence-n2
worker.confluence-lb.sticky_session=true
worker.confluence-lb.session_cookie=JSESSIONID
# Worker for Confluence Node 1 - HTTPS
worker.confluence-n1.type=ajp13
worker.confluence-n1.ping_mode=I
worker.confluence-n1.socket_connect_timeout=5000
worker.confluence-n1.host=confluence-node1
worker.confluence-n1.port=8600
worker.confluence-n1.route=confluence-node1
# Worker for Confluence Node 2 - HTTPS
worker.confluence-n2.type=ajp13
worker.confluence-n2.ping_mode=I
worker.confluence-n2.socket_connect_timeout=5000
worker.confluence-n2.host=confluence-node2
worker.confluence-n2.port=8600
worker.confluence-n2.route=confluence-node2
#=====================  Worker Section End  =====================

• Configure Confluence to accept connections from MOD_JK

        <!-- Main Connector Port -->
        <Connector port="8600"

            maxThreads="250"
            minSpareThreads="25"
            connectionTimeout="20000"

            enableLookups="false"
            protocol="org.apache.coyote.ajp.AjpNioProtocol"
            redirectPort="8443"
            acceptCount="10"
            debug="0"
            URIEncoding="UTF-8"

            proxyName="confluence.something.com"
            proxyPort="443"

        />

• Configure Apache for MOD_JK

    JkMount     /*                 confluence-lb
    JkUnMount   /html/*            confluence-lb
    JkUnMount   /synchrony/*       confluence-lb
    JkUnMount   /balancer-manager  confluence-lb

• Configure Load Balancer and proxy for Synchrony. This has been worked out with Yilin help and long time digging in Internet. Finally with network debugging I have found that Synchrony url's ending with "/websocket" should be handled by proxy as Websocket connection not http. I have found nice simple explanation - https://serverfault.com/questions/616370/configuring-apache-2-4-mod-proxy-wstunnel-for-socket-io-1-0

    <Location /balancer-manager>
        SetHandler balancer-manager

        Require                <access list>
    </Location>

    #Synchrony configuration
    ProxyRequests Off
    ProxyPreserveHost On

    # Proxy Balancer
    <Proxy balancer://confluence-synchrony-http>
        # Synchrony Node 1
        BalancerMember        http://synchrony-node1:9100 route=confluence-synnode1
        # Synchrony node 2
        BalancerMember        http://synchrony-node2:9100 route=confluence-synnode2

        # Security
        Require                all granted

        # Load Balancer Settings
        # We are not really balancing anything in this setup, but need to configure this
        ProxySet            lbmethod=byrequests
        ProxySet            stickysession=JSESSIONID
    </Proxy>
    <Proxy balancer://confluence-synchrony-ws>
        # Synchrony Node 1
        BalancerMember        ws://synchrony-node1:9100 route=confluence-synnode1
        # Synchrony Node 2
        BalancerMember        ws://synchrony-node2:9100 route=confluence-synnode1

        # Security
        Require                all granted

        # Load Balancer Settings
        # We are not really balancing anything in this setup, but need to configure this
        ProxySet            lbmethod=byrequests
        ProxySet            stickysession=JSESSIONID
    </Proxy>

    # https://serverfault.com/questions/616370/configuring-apache-2-4-mod-proxy-wstunnel-for-socket-io-1-0x
    RewriteEngine     On
    RewriteCond       %{REQUEST_URI}       ^/synchrony/(.*)/websocket            [NC]
    RewriteRule       /(.*)                balancer://confluence-synchrony-ws/$1  [P,L]
    RewriteCond       %{REQUEST_URI}       ^/synchrony/(.*)/bayeux-sync1                           [NC]
    RewriteRule       /(.*)                balancer://confluence-synchrony-pro0-ws/$1      [P,L]    

    #ProxyPass        /balancer-manager    !
    ProxyPass         /synchrony            balancer://confluence-synchrony-http/synchrony
    ProxyPassReverse  /synchrony            balancer://confluence-synchrony-http/synchrony

• Start Confluence and Synchrony as in https://confluence.atlassian.com/conf62/installing-confluence-data-center-902075269.html with correct posrts set
• Remember to secure Synchrony servers as there is a security issues with the Synchrony command line - https://jira.atlassian.com/browse/CONFSERVER-53085
• Enable Collaborative Editing
• Enjoy

Hopefully this is all.

[Sean McManus]

i created a DNS entry for the VIP on the load balancer in front of the two synchrony nodes.  I changed my mod proxy vhost file to call the vip name.

e.g.

ProxyPass /synchrony https://synchrony.xxx.com/synchrony

    ProxyPassReverse /synchrony https://synchrony.xxx.com/synchrony 

    <Location /synchrony>

        Require all granted

        RewriteEngine on

        RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]

        RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]

        RewriteRule .* ws://synchrony.xxx.com%{REQUEST_URI} [P]

    </Location>

because i made the call over SSL i also had to add this to the vhost file.

    SSLProxyCheckPeerCN off

    SSLProxyCheckPeerName off

[sarath sasi]

Hi Sean,

How you accomplished ws load balancing.?

I have a F5 in front of two synchrony nodes it is load balancing http/s traffics but not ws .Do we have to enable some configuration in F5?

[RJ Confluence Admins]

I had to drop the SSL connection.  =(  I call the vip as 8091 and pass through now.  

[sarath sasi]

Thanks.

Will that mean you put a rewwrite rule like this.

 RewriteRule .* ws://synchrony.xxx.com:8091%{REQUEST_URI} [P]


1.Is Ws traffic passing through F5 with this config?
2.Do we have to do any settings in F5 for this?

[sarath sasi]

That will be fine but my apache load balancing is configured only for synchrony servers as I have F5 for confluence load balancing.I had discussion with PS on this session stickiness part for synchrony.I guess for internal requests to synchrony JSESSION id will not be passed.So not sure apache sticky session with JSESSION ID makes sense for synchrony hub.

Will post my observation once I get a concrete evidence.

[sarath sasi]

As per vendor update LB for synchrony doesn't need session stickiness.

[sarath sasi]

Hi All,

We are observing below behavior.

1.intermittently in one node suddenly saving will fail and it will fix itself after 5 min.

2.Browser will give 204 response code.

3.Logs are not giving much info.

Does anyone faced this issue?.We are also using apache for loadbalancing synchrony cluster.

[sarath sasi]

HI Areg,

I tried configuring the same in  apache for synchrony hub.

Also i noticed that the synchrony traffic is wss:xxxx/synchrony

So i used the configuration for rewrite in apache

<Location /synchrony>

    Require all granted

   RewriteEngine on

    RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]

    RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]

   RewriteRule /(.*) balancer://confluence-synchrony-ws/$1  [P,L]

</Location>

But I am getting response code of 204 no content even though connection is getting success.

[Areg Vrtanesyan _Work_]

Hi

Which version of Confluence you are running?

In latest versions it has changed.

You can check it by using Firefox network monitor.

I will post fixed configuration once will be online.

[RJ Confluence Admins]

What changed and in which version???

[Areg Vrtanesyan _Work_]

In new version of Synchrony it is looking ^/synchrony/(.*)/bayeux-sync1 URL instead of ^/synchrony/(.*)/websocket and it is not documented anywhere ...

It is possible to detect by usung Firefox or Chrome Network Monitor and see which request fails on Websocket and add rewrite condition for it.

I have updated my answer for Apache - working as a charm for us.

[sarath sasi]

Hi Areg,

Thanks for sharing this .I configured the apache to act as lb for synchrony hub.

Configured jsession id for session stickiness.Is there any way to confirm this is working or not?

[RJ Confluence Admins]

Use developer tools in chrome 

application tab

expand the cookies and watch if the jsessionid cookie stays the same when running more than 1 app server.

[RJ Confluence Admins]

Slight hijack... and apologies for that but very related.

I have synchrony behind it's pown VIP and just call it as 8091 and all that works well.  My issue is for the app side.  When I have more than 1 server on the VIP for the app side active, users could not check boxes or change status macros with out being in edit mode.  I was able to fix this with changing to use the JSESSIONID cookie in the F5 in front of the app.  It appears though that the Apache/mod_proxy server in front of the app tier is breaking the JSESSIONID cookie.  For the life of me I can't figure out why / how.  I am about to switch to NGiNX instead.

[AnnWorley]

In front of your Data Center cluster there should be a load balancer. Configure Apache in front of the IP being used for the cluster, using the same Apache settings you would use for the single server.

This doc has a sample config in an expand macro just under the text "If you're using Confluence 6.0 or later with Synchrony (which is required for collaborative editing), you'll need to use Apache 2.4.10 or later..."

[Areg Vrtanesyan _Work_]

Hi Ann

That document doesn't describe how to do that for multiple synchrony instances running on a multiple nodes.

I know how to configure MOD_JK with Sticky Session for 2 node Confluence DC and it is working quite good.

Something like this https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html but again it is for single host. How about multiple hosts?

Now when I have started to prepare for 6.2 update I have a feeling that there is a lack of documentation around it.

Regards,

Areg

[Areg Vrtanesyan _Work_]

Right

For the Datacenter Jira or Confluence you can do following:

* with mod_proxy and mod_ajp

 # Proxy Balancer
 <Proxy balancer://confluence-cluster>
  # Confluence Node 1
  BalancerMember ajp://confluence-node1:8600 route=node1 timeout=1800
  # Confluence Node 2
  #BalancerMember ajp://confluence-node1:8600 route=node2 timeout=1800

  # Security
  Order  Deny,Allow
  #Deny from None
  Allow from All

  # Load Balancer Settings
  # We are not really balancing anything in this setup, but need to configure this
  ProxySet lbmethod=byrequests
  ProxySet stickysession=JSESSIONID
 </Proxy>

 ProxyRequests  Off
 ProxyPreserveHost On

 ProxyPass  /  balancer://confluence-cluster/
 ProxyPassReverse /  balancer://confluence-cluster/

* with mod_jk

"workers.properties" file

 #===================== Worker Section Start =====================
 # Load Balancer Worker
 worker.list=confluence-lb
 worker.confluence-lb.type=lb
 worker.confluence-lb.balance_workers=confluence-node1,confluence-node2
 worker.confluence-lb.sticky_session=true
 worker.confluence-lb.session_cookie=JSESSIONID
 # Worker for Node 1 - HTTPS
 worker.confluence-node1.type=ajp13
 worker.confluence-node1.ping_mode=I
 worker.confluence-node1.socket_connect_timeout=5000
 worker.confluence-node1.host=confluence-node1
 worker.confluence-node1.port=8600
 worker.confluence-node1.route=confluence-node1
 # Worker for Node 2 - HTTPS
 worker.confluence-node2.type=ajp13
 worker.confluence-node2.ping_mode=I
 worker.confluence-node2.socket_connect_timeout=5000
 worker.confluence-node2.host=confluence-node2
 worker.confluence-node2.port=8600
 worker.confluence-node2.route=confluence-node2
 #=====================  Worker Section End  =====================

Apache section

 JkMount    /*     confluence-lb

For Synchrony I can only see example in https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html I can only see section 6 for Synchrony like

6. Configure your load balancer for Synchrony.
Your load balancer must support WebSockets (for example NGINX 1.3 or later, Apache httpd 2.4, IIS 8.0 or later) and session affinity. 
SSL connections must be terminated at your load balancer so that Synchrony can accept XHR requests from the web browser. 

and nothing

Then in https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html?_ga=2.209825301.1803497839.1495787573-1935709168.1489762288

# Put this after the other LoadModule directives
LoadModule jk_module modules/mod_jk.so

# Adding mod_proxy w/ wstunnel and mod_rewrite for Synchrony support
LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so
LoadModule proxy_http_module /usr/lib/apache2/modules/mod_proxy_http.so
LoadModule proxy_wstunnel_module /usr/lib/apache2/modules/mod_proxy_wstunnel.so
LoadModule rewrite_module /usr/lib/apache2/modules/mod_rewrite.so

# Put this in the main section of your configuration (or virtual host, if using Apache virtual hosts)
JkWorkersFile conf/workers.properties
JkLogFile logs/apache2/mod_jk.log
JkLogLevel info

<VirtualHost *:80>
 #Synchrony configuration
 ProxyRequests Off
 ProxyPreserveHost On

 <Proxy *>
     Require all granted
 </Proxy>

 ProxyPass /synchrony http://localhost:8091/synchrony
 <Location /synchrony>
  Require all granted
  RewriteEngine on
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
  RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
  RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P]
 </Location>
 
 # mod_jk configuration
 JkMount /confluence worker1
 JkMount /confluence/* worker1

</VirtualHost>

I have also asked PS and TAM and looks like there is no exact example of 2 node and more Synchrony configuration on Apache or else even the section 5 in https://confluence.atlassian.com/doc/installing-confluence-data-center-203603.html tells to start process on each node.

Small section in https://confluence.atlassian.com/confkb/how-to-configure-apache-mod_jk-to-proxy-confluence-6-x-or-later-867351565.html?_ga=2.209825301.1803497839.1495787573-1935709168.1489762288 tells how to do the WebSockets redirection:

 ProxyPass /synchrony http://localhost:8091/synchrony
 <Location /synchrony>
  Require all granted
  RewriteEngine on
  RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
  RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
  RewriteRule .* ws://localhost:8091%{REQUEST_URI} [P]
 </Location>

But not how to load balance between the Synchrony nodes.

Regards,

Areg

[AnnWorley]

As you mentioned, you have a ticket open with Atlassian's Premier Support. It looks like they are still investigating. When you get a configuration that works, I hope you will come back to the Community and share an example.