Fail upgrade from 7.2 to 7.16

[Juan Ramirez]

We ran into a problem upgrading from Confluence 7.2 to 7.16. We backed up the previous version and ran the 7.16 executable. The Confluence service starts but when we try to connect we get an 

HTTP Status 500 – Internal Server Error Unable to acquire JDBC Connection

Is this recoverable? Do we need to go back to 7.2? We have the backup. I would need instructions on how how to go back to 7.2 and recover from backup please.

Thanks

Juan Ramirez

[Robert Wen_Cprime_]

Hello, @Juan Ramirez ! Welcome to the Atlassian Community!

Is there a way you can copy the ERRORS ONLY from your log file?  That way we can see why you can't access your database.

For context, what database (vendor and version) are you using?

[Juan Ramirez]

Thanks for your reply. We are using an old SQL Server 2008 R2.

This is the log file

HTTP Status 500 – Internal Server Error

---

Type Exception Report

Message Cannot call sendError() after the response has been committed

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

java.lang.IllegalStateException: Cannot call sendError() after the response has been committed
 org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:456)
 javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:120)
 com.atlassian.confluence.web.filter.DebugFilter$LoggingResponseWrapper.sendError(DebugFilter.java:149)
 javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:120)
 com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter$1.sendError(FourOhFourErrorLoggingFilter.java:52)
 javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:120)
 com.atlassian.core.filters.HeaderSanitisingResponseWrapper.sendError(HeaderSanitisingResponseWrapper.java:89)
 javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:120)
 javax.servlet.http.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:120)
 com.opensymphony.webwork.dispatcher.ServletDispatcher.sendError(ServletDispatcher.java:349)
 com.atlassian.confluence.servlet.ConfluenceServletDispatcher.sendError(ConfluenceServletDispatcher.java:104)
 com.atlassian.confluence.servlet.ConfluenceServletDispatcher.serviceAction(ConfluenceServletDispatcher.java:97)
 com.opensymphony.webwork.dispatcher.ServletDispatcher.service(ServletDispatcher.java:199)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
 com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter$SwapOnBootstrapFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:55)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:35)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
 com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
 com.atlassian.confluence.util.ClusterHeaderFilter.doFilter(ClusterHeaderFilter.java:57)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 org.springframework.orm.hibernate.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:170)
 com.atlassian.confluence.web.filter.ConfluenceOpenSessionInViewFilter.doFilterInternal(ConfluenceOpenSessionInViewFilter.java:41)
 org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
 com.atlassian.confluence.util.ConfluenceErrorFilter.doFilter(ConfluenceErrorFilter.java:24)
 com.atlassian.util.profiling.filters.ProfilingFilter.doFilter(ProfilingFilter.java:105)
 com.atlassian.confluence.core.datetime.RequestTimeThreadLocalFilter.doFilter(RequestTimeThreadLocalFilter.java:37)
 com.atlassian.core.filters.cache.AbstractCachingFilter.doFilter(AbstractCachingFilter.java:31)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.johnson.plugin.servlet.filter.JohnsonServletFilterModuleContainerFilter.doFilter(JohnsonServletFilterModuleContainerFilter.java:47)
 com.atlassian.confluence.tenant.TenantGateFilter.lambda$doFilter$0(TenantGateFilter.java:35)
 com.atlassian.confluence.impl.tenant.ThreadLocalTenantGate.lambda$wrap$0(ThreadLocalTenantGate.java:25)
 com.atlassian.confluence.tenant.TenantGateFilter.doFilter(TenantGateFilter.java:37)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.util.MobileAppRequestFilter.doFilter(MobileAppRequestFilter.java:36)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.internal.web.filter.spring.IgnoreWebAsyncManagerFilter.doFilter(IgnoreWebAsyncManagerFilter.java:59)
 com.atlassian.confluence.web.filter.validateparam.RequestParamValidationFilter.doFilter(RequestParamValidationFilter.java:51)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.web.filter.TranslationModeFilter.doFilter(TranslationModeFilter.java:39)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.plugin.servlet.filter.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:71)
 com.atlassian.confluence.web.filter.LanguageExtractionFilter.doFilter(LanguageExtractionFilter.java:39)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.doFilter(VCacheRequestContextFilter.java:58)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.util.LoggingContextFilter.doFilter(LoggingContextFilter.java:33)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.util.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:67)
 com.atlassian.confluence.web.filter.ZipkinTracingFilter.doFilter(ZipkinTracingFilter.java:51)
 com.atlassian.confluence.web.filter.ResponseOutputStreamFilter.doFilter(ResponseOutputStreamFilter.java:25)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30)
 com.atlassian.confluence.setup.ConfluenceEncodingFilter.doFilterInternal(ConfluenceEncodingFilter.java:35)
 com.atlassian.confluence.web.filter.AbstractStaticResourceAwareFilter.doFilter(AbstractStaticResourceAwareFilter.java:43)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32)
 com.atlassian.core.filters.HeaderSanitisingFilter.doFilter(HeaderSanitisingFilter.java:37)
 com.atlassian.confluence.servlet.FourOhFourErrorLoggingFilter.doFilter(FourOhFourErrorLoggingFilter.java:64)
 com.atlassian.confluence.internal.diagnostics.HttpRequestMonitoringFilter.doFilter(HttpRequestMonitoringFilter.java:35)
 com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:63)
 com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:32

TThanks for your help

Note The full stack trace of the root cause is available in the server logs.

---

Apache Tomcat/9.0.54

[Robert Wen_Cprime_]

Well, SQL Server 2008 is not supported by 7.16.  It accepts SQL Server 2016 (deprecated), 2017, and 2019.

[Peter DeWitt]

Compatibility Matrix:

https://confluence.atlassian.com/doc/supported-platforms-207488198.html

[Juan Ramirez]

Thanks for clarifying that. How can I downgrade to 7.2 given that I have the backup?

Thanks,

[Robert Wen_Cprime_]

1. Shut down Confluence

2. From your backup, replace the contents of

   a. install directory 

   b. Confluence home directory

3. restore database (probably not needed)

4. start up Confluence

[Juan Ramirez]

I have a Confluence folder under C:\Program Files\Atlassian\Application Data. Is that the one I have to replace with the backup?

Thanks,

[Robert Wen_Cprime_]

That's probably the Confluence home directory.  The installer prompted you for a destination directory.  Do you remember where that is?

Maybe you should see if there's a Confluence directory in C:\Program Files\Atlassian.  That's my bet where the install directory is.

[Juan Ramirez]

I have copied the contents of Confluence  7.2 backup in C:\Program Files\Atlassian\Application Data to the same path C:\Program Files\Atlassian\Application Data. This is the home directory.

Then I copied the 7.2 backup from the directory C:\Program Files\Atlassian to the same path C:\Program Files\Atlassian. I tried to restart confluence in Services but it failed to start. It seems that we missed one step here. I believe this is because I still have 7.16 installed (Control Panel/ Programs).Should I remove 7.16 and install 7.2 back? Removing 7.16 would delete the current home and installation directory?

[Juan Ramirez]

Can I just downgrade the 7.16 to 7.2 by just running the executable?

Thanks,

[Juan Ramirez]

Robert, I will need assistant on how to restore to 7.2 please. I have the backup files. I still have 7.16 installed. 

Thanks,

[Robert Wen_Cprime_]

I don't think the executable allows you to downgrade (puts the 7.2 install back).

What I think you need to do is keep everything as it is and manually set up Confluence 7.2 as a service. Here's the documentation for doing so: https://confluence.atlassian.com/doc/start-confluence-automatically-on-windows-as-a-service-172071.html

[Robert Wen_Cprime_]

@Juan Ramirez : I need the following since I'm not the one in front of the computer:

Screenshots of:

The directory structure below C:\Program Files\Atlassian so I can verify

- The install directory

- the Confluence home directory

What the services panel looks like.

In the meantime, for when you try to do the upgrade again, here's a link to the upgrade guide: https://confluence.atlassian.com/doc/upgrading-confluence-4578.html

[Juan Ramirez]

Hi Robert,

Thanks for all your help. I am including the screenshots of what I see for the install and home directory with the backup files I used to restore to each directory. I am including the service general and log on screens too.

Thanks again

[Juan Ramirez]

I would like to add that error message I get when I try to start the service is related to a permissions problem. The error code is 4.

[Robert Wen_Cprime_]

OK

1. Copy/verify that the contents of 2022_02_03-Confluence-7.2.0-back\Confluence are in C:\Program Files\Atlassian\Confluence - That is the root level of your install directory.

2. Copy/verify that the contents of 2022_02_03-Confluence-7.2.0-back\Confluence are in C:\Program Files\Atlassian\Application Data\Confluence (basically your Confluence home directory).  The only file of importance here is confluence.cfg.xml. You want to make sure that's the original file (from 7.2). Everything else may not be touched since the upgrade broke.  But if you want to be really cautious, entire contents can work.

3. Start Confluence MANUALLY. Run C:\Program Files\Atlassian\Confluence\bin\start-confluence.bat

4. Once you've verified that Confluence starts up and is 7.2, stop Confluence MANUALLY. Run C:\Program Files\Atlassian\Confluence\bin\stop-confluence.bat

5. Look at the entry for the Confluence service.  Verify that the correct version of Tomcat is being called (I don't think "tomcat9.exe" is in 7.2 version of C:\Program Files\Atlassian\Confluence\bin).  Refer to the guide I posted previously for setting up Confluence as a Windows Service.

Good luck!

[Juan Ramirez]

I did follow the instructions on how add manually a new service. The new service started but I get now a 404 error when I open the link for confluence. Before the 404 error, I get the  message your confluence instance is already setup. Then when I click on return to Confluence I get the 404 error below. The new service that was created manually still has the \Tomcat9.exe instead of the Tomcat9w. How do I change that parameter? Thanks,

[Juan Ramirez]

We are making progress. I restored both directories using a backup from 2 days ago. Before restoring, I renamed the old directories and restore so all files from Wednesday were copied. I started the service and it worked fine. 

Now I get a database connection error from Confluence. The configuration file shows the correct sql database, path, username and password but it cannot connect. Does it mean  that the SQL database was changed when I updated to 7.16? I can restore the SQL database to a 2 days ago version.

[Juan Ramirez]

This is one of the error messages

url: /bootstrap/selectsetupstep.action | traceId: 00841ff8f5694b85
com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Server chose TLSv1, but that protocol version is not enabled or not supported by the client.". ClientConnectionId:71272e17-6eb1-43f4-b5d1-5f8e73e2c847

[Robert Wen_Cprime_]

Yes, it looks like there were some writes to the database during the upgrade process.    

Shut down Confluence.

Restore the database with the backup from 2 days ago.

Restart Confluence.

[Juan Ramirez]

No, it was not the database. It was TLS. I modified the Java option on the service and it worked. I did not have to restore the database.

I found the solution here Confluence 7.x throws "Can't reach database server or port" error when connecting to Microsoft SQL server | Confluence | Atlassian Documentation

under Workaround

n Windows, Confluence running as a service:
Identify the name of the service that Confluence is installed as in Windows (Go to Control Panel > Administrative Tools > Services)

We are expecting to see a service name like this Confluence251017164028

Open the command window (Choose Start > cmd.exe)

cd to the bin directory of your Confluence installation folder and run the following command:

tomcat9w.exe //ES//%service_name%
Click on the Java tab to see the list of current start-up options
In the Java Options box, update

from:

-Djdk.tls.server.protocols=TLSv1.1,TLSv1.2
-Djdk.tls.client.protocols=TLSv1.1,TLSv1.2
to:

-Djdk.tls.server.protocols=TLSv1,TLSv1.1,TLSv1.2
-Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2
Restart the service

Thanks,

[Robert Wen_Cprime_]

So all looks good?  If so, I'm glad.

For your next attempt, go through the Upgrade Guide and understand that some dependencies (like the database) may need to be upgraded first before upgrading Confluence.

Good luck with your next upgrade!

[Juan Ramirez]

Thanks. I think our next move will be migrating to Azure or to the cloud option

[Peter DeWitt]

@Juan Ramirez This may be as simple as you configuration file not pointing to the correct JDBC driver or you are using an older non-compatible driver. Check out this compatibility matrix to ensure you are using version that play nice together:

https://confluence.atlassian.com/doc/supported-platforms-207488198.html

You can also verify that you config files are setup correctly with the aid of this article.

https://confluence.atlassian.com/doc/configuring-a-datasource-connection-937166084.html

You can also open a support request directly with Atlassian Support and they may have additional things for you to check.

-pjd

[Juan Ramirez]

Hi Robert,

Thanks for all your help. I am including the screenshots of what I see for the install and home directory with the backup files I used to restore to each directory. I am including the service general and log on screens too.

Thanks again,