Errors after upgrading Confluence Data Center 7.19 to 8.5

[Tim Schüder]

Hi,

after upgrading Confluence DataCenter from 7.19 to 8.5 running with Windows Server 2019 64bit we experience the following problems :

- the Confluence-Windowsservice will not stop normally

- we get http500 errors on one node of the Confluence-Cluster

The following lines appear ..

ERROR [https-jsse-nio2-8443-exec-22] [ContainerBase.[Standalone].[localhost].[/]] log Unhandled exception occurred whilst decorating page

 -- url: / | userName: xxxxxxx | traceId: f37bc8698694e3f9

java.lang.RuntimeException: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo

Caused by: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo

Caused by: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo

Caused by: java.lang.IllegalAccessException: class org.apache.velocity.runtime.parser.node.PropertyExecutor cannot access class sun.util.calendar.ZoneInfo (in module java.base) because module java.base does not export sun.util.calendar to unnamed module @65ea0089

2023-09-01 09:36:15,123 ERROR [https-jsse-nio2-8443-exec-22] [[Standalone].[localhost].[/].[default]] log Servlet.service() for servlet [default] in context with path [] threw exception

java.lang.RuntimeException: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo

Caused by: javax.servlet.ServletException: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo

Caused by: org.apache.velocity.exception.VelocityException: ASTIdentifier() : exception invoking method for identifier 'rawOffset' in class sun.util.calendar.ZoneInfo

2023-09-01 09:28:32,515 ERROR [Catalina-utility-2] [atlassian.plugin.parsers.XmlDescriptorParser] createModuleDescriptor There were problems loading the module 'job'. The module and its plugin have been disabled.

2023-09-01 09:28:32,515 ERROR [Catalina-utility-2] [atlassian.plugin.parsers.XmlDescriptorParser] createModuleDescriptor There was a problem loading the descriptor for module 'job' in plugin 'Confluence JIRA Metadata Plugin'.\n Job module descriptors are no longer supported. Use JobConfig modules instead.

Any suggestions?
Kind regards

Tim

Update:

With help by Atlassian I solved the problem :

Add the following to your setenv.bat  :

set CATALINA_OPTS="--add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED ${CATALINA_OPTS}"


Add the following to your JAVA-Properties:

--add-opens=java.base/sun.util.calendar=ALL-UNNAMED

[Bernhard Pressler]

The problem origins in the reuse of the 7.19 setenv.sh or setenv.bat.

I used the one delivered with 8.5.x and it works now.

[Tim Schüder]

Hi Berhard, I try that ... as the copied installation will probably use the parameters set up with setenv.bat the 7.19-way ...

Thanks

[Bernhard Pressler]

As Confluence 8.5.x uses JDK17 there may be specific differences 

[Benjamin W.]

Hi all

we also stumbled upon this issue (Windows Server 2019, Upgrade Confluence 7.19.5 to 8.5.0).

For us, using 8.5.1 did not solve the issue at all. Some errors disappeared, but Confluence did not start eitherway.

We also tried making use of JRE / JDK, but this showed not difference.

First when again using JDK 11.0.14.1_1 (instead of 17.0.7_7) we were able to successfully launch Confluence 8.5.1.

Regards

[Bernhard Pressler]

I Think it is the same issue i had - on the upgrade the setenv.bat is not replaced by the new one. So do a clean 8.5.x installation on a test-environment without the ongoing setup-process and pick the new, fresh setenv.bat file for you upgraded environment.

[Tim Schüder]

With help by Atlassian I solved the problem :

Add the following to your setenv.bat  :

set CATALINA_OPTS="--add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED ${CATALINA_OPTS}"


Add the following to your JAVA-Properties:

--add-opens=java.base/sun.util.calendar=ALL-UNNAMED

[Bernhard Pressler]

The new setenv.bat / setenv.sh includes another file:

# Add various JPMS arguments to allow Confluence to work on Java 17
CATALINA_OPTS="@$CATALINA_HOME/confluence/WEB-INF/jpms-args.txt ${CATALINA_OPTS}"

inside there are those parameters even shown by Tim

# Add various JPMS arguments to allow Confluence to work on Java 17.
# These also work with Java 11, but are optional.
--add-opens java.base/java.lang=ALL-UNNAMED
--add-opens java.base/java.lang.reflect=ALL-UNNAMED
--add-opens java.base/java.net=ALL-UNNAMED
--add-opens java.base/java.io=ALL-UNNAMED
--add-opens java.base/java.util=ALL-UNNAMED
--add-opens java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens java.base/sun.net.www.protocol.jar=ALL-UNNAMED
--add-opens java.base/sun.net.www.protocol.http=ALL-UNNAMED
--add-opens java.base/sun.net.www.protocol.https=ALL-UNNAMED
--add-opens java.base/sun.util.locale=ALL-UNNAMED
--add-opens java.management/javax.management=ALL-UNNAMED
--add-exports java.base/sun.security.action=ALL-UNNAMED
--add-exports java.base/sun.util.calendar=ALL-UNNAMED
--add-exports java.naming/com.sun.jndi.ldap=ALL-UNNAMED
--add-exports java.xml/com.sun.org.apache.xml.internal.utils=ALL-UNNAMED
--add-exports java.desktop/sun.font=ALL-UNNAMED
--add-exports java.base/sun.security.util=ALL-UNNAMED

[Benjamin W.]

Thank you very much for the fast feedback.

I can confirm that with those hints I was able to run Confluence 8.5.1 on Java 17.

As we run Confluence as a service (Windows), we added the parameters from %CATALINA_HOME%\confluence\WEB-INF\jpms-args.txt onto the registry under Computer\HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Apache Software Foundation\Procrun 2.0\Confluence\Parameters\Java\Options9

[Aron Gombas _Midori_]

By the way, the problem is caused by the <job> definition in the app called "Confluence JIRA Metadata Plugin". 

You may want to try if you can start Confluence without this app. (There is some commandline parameter to start up Confluence without certain apps.)

[Aron Gombas _Midori_]

The culprit here is the `sun.util.calendar.ZoneInfo` class. It is a Java class part of the Java platform and the corresponding time management packages went through changes from one Java version to another.

The first thing you should check is whether your particular Confluence version is running on a recommended Java version and distribution!

[Tim Schüder]

As I installed via the Windows installer I would think that this should work... both nodes are running with

openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Temurin-17.0.7+7 (build 17.0.7+7)
OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (build 17.0.7+7, mixed mode, sharing)

[Aron Gombas _Midori_]

And this is what you see in the Jira admin screen (in the web UI)?

I think it is quite typical on Windows that there are multiple JREs installed, and a Java program is executed with an unexpected JRE. Therefore, the best is if you check it in the Jira admin screen.

[Tim Schüder]

Checking the JVM-Parameters in the Jira admin screen does not sho any differences or problems. THe used version is still 17.0.07 running in sun.boot.library.path  Confluence-Home\jre\bin.

There is just one java.exe installed on the whole system...

[Aron Gombas _Midori_]

OK, then this one is eliminate! Check the hint in my other answer.

[Tim Schüder]

Sorry, I don't get it. I should delete the one and only java.exe an the system?? If this is really the causing problem, why is the first node in this dataCenter-Cluster running without any problems and the copied one expereinces such problems?

[Aron Gombas _Midori_]

No, sorry. I said that Java version is "eliminated" (removed, ruled out) from the potential root causes.

And I suggest that you see my other answer above. As you see, from the stack trace, it can be identified in which app the problem comes up. And I remember that Jira can be started with disabling certain apps. I think it would worth a try to start Jira with the problematic app turned off.

It is not a solution, but an experiment to validate if that app is the problem and there is nothing else.

Does that make sense?

[Tim Schüder]

Ah ... sorry for misunderstanding :-) ... and we are talking about Confluence ... I check to disable this App.

:-)