Unable to create space after upgrading to Confluence 6.11.2 from 6.8.1

[Chun-Chi Su]

After the upgrade I'm not able to create any space

The ONLY log that shows from atlassian-confluence.log when clicking on the button "Create Space" 

2018-09-21 10:39:30,312 WARN [http-nio-8090-exec-1] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
 ->[PluginReadWriteTx]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #204045758)
 -- referer: http://dev.wiki.company.com/ | url: /rest/create-dialog/1.0/space-blueprint/dialog/web-items | traceId: b745e438eca8cbcc | userName: 60054973

There was no any other error messages, which is very puzzling.

The confluence site has entered safe mode, so all plugins are disabled, but this problem still persists.

Database Information

Database name	MySQL
Database version	5.7.17
Database Dialect	com.atlassian.confluence.impl.hibernate.dialect.MySQLDialect
Database Driver Name	com.mysql.jdbc.Driver
Database Driver Version	mysql-connector-java-5.1.46 ( Revision: 9cc87a48e75c2d2e87c1a293b2862ce651cb256e )
Database Connection URL	jdbc:mysql://localhost/confluence
Database Connection Transaction Isolation	Read committed
Database Latency	0

I would like to troubleshoot this problem myself, but I need more logs from atlassian-confluence.log to be able to do that, do any of you have insights on how this could be happening? Or where else can I be looking at to discover more logs?

---------------------------------Updated---------------------------------

Upon further investigation, I realized I'm missing Space Blueprints when viewing blueprints from the administration panel, so that must be the culprit.

I've restored a backup of my site from 6.8.1, and tried upgrading to 6.11.1, and the site still works. So I can confirm this bug only exists on 6.11.2.

Log from atlassian-confluence.log when clicking on Global Templates and Blueprints.

(The page renders, but the Space Blueprints section is missing from the page)

2018-09-21 11:54:35,576 ERROR [http-nio-8090-exec-2] [plugin.descriptor.web.ConfluenceWebInterfaceManager] getHtml Failed to render web panel: com.atlassian.plugin.web.descriptors.DefaultWebPanelModuleDescriptor$ContextAwareWebPanel@39e51b6f
 -- url: /pages/templates2/listglobaltemplates.action | traceId: aa4f2f1485551d23 | userName: 60054973 | action: listglobaltemplates
java.lang.NullPointerException
 at java.util.Objects.requireNonNull(Objects.java:203)
 at com.atlassian.confluence.plugins.createcontent.impl.DefaultSpaceBlueprintManager.build(DefaultSpaceBlueprintManager.java:205)
 at com.google.common.collect.Iterators$8.transform(Iterators.java:799)
 at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
 at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
 at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258)
 at com.atlassian.confluence.plugins.createcontent.impl.DefaultSpaceBlueprintManager.lambda$getAoSpaceBlueprints$1(DefaultSpaceBlueprintManager.java:128)
 at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)
 at com.atlassian.sal.spring.component.SpringHostContextAccessor$1.doInTransaction(SpringHostContextAccessor.java:71)
 at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
 at com.atlassian.sal.spring.component.SpringHostContextAccessor.doInTransaction(SpringHostContextAccessor.java:68)
 at com.atlassian.confluence.spring.transaction.interceptor.ConfluenceSpringHostContextAccessor.access$001(ConfluenceSpringHostContextAccessor.java:21)
 at com.atlassian.confluence.spring.transaction.interceptor.ConfluenceSpringHostContextAccessor.lambda$doInTransaction$3(ConfluenceSpringHostContextAccessor.java:72)
 at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
 at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
 at com.atlassian.confluence.spring.transaction.interceptor.ConfluenceSpringHostContextAccessor.doInTransaction(ConfluenceSpringHostContextAccessor.java:72)
 at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
 at com.sun.proxy.$Proxy317.doInTransaction(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:136)
 at com.sun.proxy.$Proxy317.doInTransaction(Unknown Source)
 at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:18)
 at sun.reflect.GeneratedMethodAccessor255.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
 at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
 at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
 at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
 at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
 at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
 at com.sun.proxy.$Proxy925.execute(Unknown Source)
 
 ......

[Athirah Zolkefli]

Hey @Chun-Chi Su,

Thanks for adding in the last bit in your description about the NPE in your logs.

We've had several reports on the same issue occurring in 6.11.2 and they were all resolved by following the steps in the following KB to drop the Blueprint AO tables.

• Creating new page or space loads forever

Let us know if it helps to resolve your issue too.

Regards,
Athirah

[Chun-Chi Su]

Thank you @Athirah Zolkefli, I will try this resolution ASAP, but the KB also stated that the workaround won't work on Oracle databases, which unfortunately, is the environment our production server is running on. Will there be a workaround for Oracle databases as well?

[Athirah Zolkefli]

Hi @Chun-Chi Su and everyone, the KB is now updated with a workaround for Oracle too! There are 3 additional commands to run if you're using Oracle.

There is also a bug report created to report this behaviour, which you can find here:

• CONFSERVER-57943 - Unable to create spaces after upgrading from 6.x to 6.11.x or later versions

[Lazom Dzemaili Moz Technologie]

I was unable to create a space with the same error "An unknown server error has occured".

This was a problem with the sql_mode parameter in mysql database.

The value was set to 'NO_AUTO_VALUE_ON_ZERO'. This is not good.

I modified the my.cnf configuration file of mysql, I added : sql_mode=""

Restart mysql server

Restart Confluence and the error is gone on my server instance.

[Jasper Golze]

Hi,

I am using postgres DB.

[Lazom Dzemaili Moz Technologie]

It would be interesting to find the equivalent of NO_AUTO_VALUE_ON_ZERO on postgres DB.

In Mysql, NO_AUTO_VALUE_ON_ZERO is :

NO_AUTO_VALUE_ON_ZERO affects handling of AUTO_INCREMENT columns. Normally, you generate the next sequence number for the column by inserting either NULL or0 into it. NO_AUTO_VALUE_ON_ZERO suppresses this behavior for 0 so that only NULL generates the next sequence number.

(https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html)

[Julian Rdnz]

Hi,

the same problem occurred on my Confluence instance after updating to 6.12.1. It seems to be caused by corrupted data in some template tables.

Solution:

1. Backup database or even better the whole system ( VM snapshot / XM backup ). You better try this on your staging / dev system!
2. Shutdown Confluence
3. Delete tables:

   DROP TABLE "AO_54C900_SPACE_BLUEPRINT_AO";
   DROP TABLE "AO_54C900_C_TEMPLATE_REF";
   DROP TABLE "AO_54C900_CONTENT_BLUEPRINT_AO"; 

4. Clean plugin cache: https://confluence.atlassian.com/confkb/how-to-clear-confluence-plugins-cache-297664846.html 
5. Restart Confluence.

Now creating spaces should work again.

[Chun-Chi Su]

Thank you for your input! But according to this post https://confluence.atlassian.com/confkb/creating-new-page-or-space-loads-forever-744329880.html this won't work for servers running on Oracle.

[Mirko Schäfer (Seibert Solutions)]

@Julian Rdnz , great, works for me (with Postgres). You safed my day!

Thanks a lot.

[Eddie Moore]

I had the same issue going from 6.0.5 to 6.13.0.  Implemented the changes above on my test server and it's fixed. plan on doing production server next.  Thanks for the post on Nov 12!

[Jasper Golze]

Hi I do have 6.15.1, still cant't create spaces. Try to delete

DROP TABLE "AO_54C900_SPACE_BLUEPRINT_AO";
DROP TABLE "AO_54C900_C_TEMPLATE_REF";
DROP TABLE "AO_54C900_CONTENT_BLUEPRINT_AO"; 

 But they don't exist. Cleared Cache and restarted. Still the problem.

Even when I use safe mode, I can't create a space.

Log shows:

2019-03-17 13:50:44,121 WARN [http-nio-8090-exec-1] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:
->[PluginReadWriteTx]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #2019209192)
-- referer: https://wiki***.de/ | url: /rest/create-dialog/1.0/space-blueprint/dialog/web-items | traceId: c56235d31de745d8 | userName: jazz

I am using postgresql as DB and Apache as Reverse Proxy with SSL.

Any Idea?

[kriegalex]

We have found that similar problem was due to not reinstalling QMS Blueprints (softcomply eQMS) plugin after update to 6.12.0 because the trial was expired.

After reinstalling the plugin (with trial still expired) on 6.12.0, the create space worked again. But after uninstalling it again, it goes back to not working, so it is only a temporary workaround.

Do we really have to manually drop those tables after uninstalling this plugin ? Should the uninstall procedure not clean what is needed to be cleaned ?

[Chun-Chi Su]

Hello! Are you able to provide links to the plugin? We can't find any plugins named QMS Blueprints on the Marketplace. Thank you!

[kriegalex]

https://marketplace.atlassian.com/apps/1216218/softcomply-eqms?hosting=server&tab=overview