unable to create page after upgrade to 6.0.1 with apache 2.2 as proxy

[Andreas Nitsche]

Hi,

after I upgraded to latest 6.0.1 I'm not able to create pages anymore. The loading symbol does not disapear and after some time I see the message:

Loading the editor's taking longer than usual. Give it a few moments, then refresh your page if it still doesn't load. Speak to your Confluence admin if that doesn't fix it.

Refreshing the page was not helpful and I had a look at the catalina log and the atlassian-confluence.log, both do not show an error while loading a "add page".

Do you have any ideas what to try to solve this problem?

Cheers
Andi

[Roman Samorodov]

Hi I just changed one system property and it worked for me.

In file <Confluence home>\confluence.cfg.xml I changed synchrony.proxy.enabled property to false.

Documentation

[Rachel Robins]

Hi everyone, 

Sorry to hear you've had problems getting collaborative editing up and running with your reverse proxy.  It's important to know your reverse proxy must support websockets. This means Apache 2.2 won't work, you need Apache 2.4.    

The following docs have some example configs that you can use as a basis for your own config.  Note that the order of directives and blocks is important in the examples. 

• Proxying Atlassian server applications with Apache HTTP Server (mod_proxy_http) (be sure to expand the Confluence example in Part B, step 2)
• Using Apache with mod_proxy (note the info on required modules that you may need to enable) 

[Vojtech Galda]

I don't agree with you. Is possible use proxy_wstunnel_module module with 2.2. @Andreas Nitsche wrote down how to get this module, see below

[Hal9000]

I am running Apache 2.4, but all I needed to make it work was set synchrony.proxy.enabled property to false. 

[小白]

set synchrony.proxy.enabled property to false  work for me.

my confluence version is 6.6.1

[Panos]

The problem comes the new collaborative script which surprise surprise is running at your port 8091 by default. So, open the port in the firewall and go to Admin -> Left panel down is the Collaborative Editing -> restart the synchrony.

[Panos]

Oh! And there is a new file called atlassian-synchrony in the log folder of your app data to help you debug.

[stalamoni]

thanks Panos, this was the solution to my problem. but in my case i've temporary turned off collaborative editing feature and i will check firewall / router problem later  with our network admin. 

[Mick Carroll]

Thanks! Worked for me. I just turned off Collaborative Editing (we dont use it)

[Adrian Zhang]

I am using Nginx to listen 80 and forward requests to localhost 8090，only 80/tcp port opened on firewall, and meet the problem that cannot edit any page. This solution works like charming. Thanks Panos!

[Jason Hihn]

I am using NGINX, and I am also interested in the setting to make this work on NGINX.

[Panos]

For you, (hopefully all of you) that still have problems upgrade to 6.0.2. According to their release notes there are some bug fixes regarding synchrony.

[Andreas Müller]

Hi,

I changed from Cloud to Server 6.0.1 yesterday, imported everything from the Cloud instance, have it up and running now BUT editor doesn't work. Same message as you have. I can't create Pages and also can not edit pages. 

Tried direct connection without SSL, no go.
Running NginX with SSL as proxy now, still, no go. 

[Panos]

And thats the reason I switched back to lovely 5.10  Anyway, look the inspection console to identify the error (could be different than ours?) which will give you clues of what is going on. Further more in the application folder/logs there is a new log file, something with synchrony.log. Take a look there also

[Andreas Müller]

As a workaround I turned off 'collaborative editing'. Let's see if future versions will get a fix for that 

[Siddheshwar mhetre]

@Panos Karampis, how do you downgrade the Conflunece to 5.10?

[Andreas Nitsche]

If you did not made a backup before your upgrade, don't downgrade but simply turn off the collaborative editing in gneral settings -> collaborative editing.

[Siddheshwar mhetre]

Ohh.. wow.. thanks @Andreas Nitsche, it works for me 

[FOCUS S.A.]

Same here, i'm not able to make the collaborative editing working with nginx. Hopefully i'm in a test env.

[Andreas Nitsche]

Which Version of nginx is in use for your confluence instance?

[Jasper Golze]

I had the same issue. after disabling collaborative editing now everything works again fine.

[Livio Sartori]

Just doing a clean install of confluence 6.0.2 using nginx and I have the same problem the editor does not come back I cannot create new pages

[Andreas Nitsche]

@Livio Sartori, you can simply turn off collaborative editing in general settings.

[Andreas Nitsche]

Hey,

I made some changes to my configuration and it would be great if somebody with an apache 2.2 can verify that.

EDIT: build part of the module was missing in the explanation

Build mod_proxy_wstunnel for your apache version

# Check apache version (on my centos 6 box it's 2.2.15)
rpm -qa | grep httpd

# I would recommend to do all the build stuff in the next steps as an 
unprivileged user to not install or break anything when doing this as 
root.
# ensure these packages are installed, otherwise commands below will fail:
# - svn
# - libtool

# Checkout apache source
svn checkout http://svn.apache.org/repos/asf/httpd/httpd/tags/2.2.15/ httpd-2.2.15

# Get patch and apply it
wget https://gist.github.com/vitkin/6661683/raw/873dd8b4de4ad1ff69757ffe48fc574374aedc57/apache-2.2-wstunnel.patch
cd httpd-2.2.15
patch -p1 -i ../apache-2.2-wstunnel.patch

# Build Apache
svn co http://svn.apache.org/repos/asf/apr/apr/branches/1.4.x srclib/apr
svn co http://svn.apache.org/repos/asf/apr/apr-util/branches/1.3.x srclib/apr-util
./buildconf
./configure --enable-proxy=shared --enable-proxy_wstunnel=shared
make

# Copy the module to apache installation
sudo cp modules/proxy/.libs/mod_proxy_wstunnel.so /etc/httpd/modules

# Create module load file
cd /etc/httpd/conf.d
echo "LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so" > mod_proxy_wstunnel.conf

Default proxy for the confluence application.

<VirtualHost <IPv4>:443 <IPv6>:443>
	...

	ProxyRequests Off
    ProxyVia Off
    SSLProxyEngine On

    ProxyPass / http://127.0.0.1:8080/
    ProxyPassReverse / http://127.0.0.1:8080/

	...
</VirtualHost>

Connector in server.xml

<Connector port="8080"
        maxHttpHeaderSize="8192"
        maxThreads="200"
        minSpareThreads="10"
        maxSpareThreads="75"
        enableLookups="false"
        redirectPort="8443"
        acceptCount="100"
        connectionTimeout="20000"
        debug="0"
        URIEncoding="UTF-8"
        proxyName="confluence.example.com"
        proxyPort="443"
        secure="true"
        scheme="https" />

Ensure to change the proxyName.

Add an additional VirtualHost for the synchrony service

<VirtualHost <IPv4>:8091 <IPv6>:8091>
	...
	SSLEngine on
	<rest of ssl configuration>
	...

    ProxyRequests Off
    ProxyVia Off
    SSLProxyEngine On

    ProxyPassMatch "^/synchrony/(.*/websocket.*$)" ws://127.0.0.1:8091/synchrony/$1
    ProxyPass /synchrony http://127.0.0.1:8091/synchrony

	...

</VirtualHost>

Ensure only the external IPv4 and IPv6 addresses of your server are used here because confluence itself needs to be able to open the port 8091 on 127.0.0.1. The ProxyPassMatch statement is the trick. As long as confluence only calls websockets when there's a websocket in the url this will work.

Change synchrony.bind ip for your confluence instance

JAVA_OPTS="$JAVA_OPTS -Dsynchrony.bind=127.0.0.1"

Disable synchrony.proxy

The service is enabled by default, you can find this configuration statement in <confluence-home>/confluence.cfg.xml.

<property name="synchrony.proxy.enabled">false</property>

Restart Services

Depends on how you installed confluence, hopefully you know how to stop and start the application.

For apache you should do:

apachectl -S
service httpd restart

Would be great if somebody could check this explanation and reply if it's working this way for your setup as well.

Cheers
Andi

[Andreas Nitsche]

I do have an apache in front of my confluence to do the SSL handshake for me. Looks like there is the problem because the OPTION request to port 8091 is also called via https. Is there already a solution documented?

I would start to proxy this service through apache as well. Shrink the listen for port 8091 to 127.0.0.1, setup the proxy on port 8091 for the external ip including ssl and test again. I will try that later or tomorrow.

Cheers
Andi

[Panos]

ah so i am not the only one with problem when ssl

[Panos]

https://confluence.atlassian.com/doc/administering-collaborative-editing-858772086.html scroll down to Change your Synchrony configuration

[Andreas Nitsche]

Just an update on how far I am at the moment.

The idea to configure synchrony to localhost was more difficult than I thought at the beginning because the documentation for synchrony.bind and synchrony.port are well hidden. I checked this site to see how I can configure synchrony for data center even if I do not have one: https://confluence.atlassian.com/doc/configuring-synchrony-for-data-center-858772125.html

Once the two options

JAVA_OPTS="$JAVA_OPTS -Dsynchrony.bind=127.0.0.1"
JAVA_OPTS="$JAVA_OPTS -Dsynchrony.port=8091"

are set, synchrony starts at localhost:8091 as expected. After that I configured apache to listen on the external ip on port 8091. Make to to not only use "Listen 8091" but use "Listen x.y.z.a:8091" and if you've enabled IPv6 according "Listen a:b:c::x:8091"

At the moment my collaboration service is NOT working yet, even if the service can be accessed through Apache with SSL. This is because a websocket is used for synchrony and my apache 2.2.15 of the CentOS 6 box I run confluence, doesn't support mod_proxy_wstunnel out of the box. This will be my next try to get this working via SSL.

Sooner or later I will setup a CentOS 7 box with apache 2.4.x which brings already the module mod_proxy_wstunnel with it.

Cheers and good night
Andi

[Andreas Nitsche]

Ok, now it's getting a good night! This is what I did to make synchrony work through apache as ssl proxy for the websocket service.

# Check apache version (on my centos 6 box it's 2.2.15)
rpm -qa | grep httpd

# I would recommend to do all the build stuff in the next steps as a unprivileged user to not install or brake anything when doing this as root.
# ensure these packages are installed, otherwise commands below will fail:
# - svn
# - libtool

# Checkout apache source
svn checkout http://svn.apache.org/repos/asf/httpd/httpd/tags/2.2.15/ httpd-2.2.15

# Get patch and apply it
wget https://gist.github.com/vitkin/6661683/raw/873dd8b4de4ad1ff69757ffe48fc574374aedc57/apache-2.2-wstunnel.patch
cd httpd-2.2.15
patch -p1 -i ../apache-2.2-wstunnel.patch

# Build Apache
svn co http://svn.apache.org/repos/asf/apr/apr/branches/1.4.x srclib/apr
svn co http://svn.apache.org/repos/asf/apr/apr-util/branches/1.3.x srclib/apr-util
./buildconf
./configure --enable-proxy=shared --enable-proxy_wstunnel=shared
make

# Copy the module to apache installation
sudo cp modules/proxy/.libs/mod_proxy_wstunnel.so /etc/httpd/modules

# Create module load file
cd /etc/httpd/conf.d
echo "LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so" > mod_proxy_wstunnel.conf

# add proxy pass to the synchrony virtual host
<VirtualHost $external_IP4:8091 $external_IP6:8091>
    ServerName your.confluence.fqdn.org

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM
    SSLCertificateKeyFile /path/to/key/file
    SSLCertificateFile /path/to/crt/file
    SSLCertificateChainFile /path/to/intermediate/if/needed
    SSLCACertificateFile /path/to/ca/file
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown

    ProxyPass /synchrony/sockjs/v1/ ws://127.0.0.1:8091/synchrony/sockjs/v1/
    ProxyPass / http://127.0.0.1:8091/
    ProxyPassReverse / http://127.0.0.1:8091/
</VirtualHost>

# check configuration and reaload
apachectl -S
service httpd reload

Maybe it's useful to put this all together in a single post but that can be done after some sleep.

Cheers from http://www.sixhop.net/ side of life
Andi

[Vojtech Galda]

Hello Andreas,

could you post whole apache config file for confluence virt host? We have the same problem with "wss:". Thanks

Regards,

Vojta

[Andreas Nitsche]

Hi Vojta,

it depends on your setup how your virtual host configuration need to look like. These is the VirtualHost I use for the synchrony proxy service:

<VirtualHost <IPv4>:8091 <IPv6>:8091>
    ServerAdmin <webmaster@example.com>
    DocumentRoot /var/www/html
    ServerName <confluence.example.com>

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM
    SSLCertificateKeyFile </etc/ssl/server.key>
    SSLCertificateFile </etc/ssl/server.crt>
    SSLCertificateChainFile </etc/ssl/intermediate_ca.pem>
    SSLCACertificateFile </etc/ssl/ca.pem.cer>
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown

    ProxyPass /synchrony/sockjs/v1/ ws://127.0.0.1:8091/synchrony/sockjs/v1/
    ProxyPass / http://127.0.0.1:8091/
    ProxyPassReverse / http://127.0.0.1:8091/

    ErrorLog logs/confluence-synchrony-error_log
    CustomLog logs/confluence-synchrony-access_log
</VirtualHost>

It can be that this virtual will work for you, but nobody can give a guarantee on that. All values in <> should be checked and adjusted to your needs starting by IPv4 and ending with the SSLCACertificateFile.

Would be great to hear if that works for you.

Cheers
Andi

[Vojtech Galda]

Hi Andi,

thank you for response. I will try it. And what about virt confluence host :443? Do you use this - 

# Auth changes in 2.4 - see http://httpd.apache.org/docs/2.4/upgrading.html#run-time
	ProxyPass /synchrony http://<internal_domain>:8091/synchrony

	<Location /synchrony>
		Require all granted
		RewriteEngine on
		RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
		RewriteCond %{HTTP:CONNECTION} Upgrade$ [NC]
		RewriteRule .* ws://<internal_domain>:8091%{REQUEST_URI} [P]
	</Location>

how is mentioned on https://confluence.atlassian.com/kb/proxying-atlassian-server-applications-with-apache-http-server-mod_proxy_http-806032611.html? 

Regards,

Vojta

[Thomas Heisterhagen]

Hi - I'm running two concfluence instances with a path : my.domain.tld/confluenc1 & my.domain.tld/confluence2 . But the System is calling my.domain.tld/synchrony/... in both instances instead of my.domain.tld/confluence[1|2]/synchrony/ . Did anyone found a workaround how to configure it this way?

[Andreas Nitsche]

Hi,

https://confluence.atlassian.com/doc/configuring-synchrony-for-data-center-858772125.html

There is a config value named synchrony.context.path, looks like this is exactly what you're searching for. Have a try.

Cheers
Andi