Bitbucket 6 denies push with remote: Communication breakdown with Bitbucket.

[Romulo Ritter Contini]

This week I upgraded the stack Jira + Bitbucket to versions 8.0.0 and 6.0.0. Since then, the bitbucket constantly prevents users to push changes to any repository / branch with the message on client:

POST git-receive-pack (812 bytes)
remote: Communication breakdown with Bitbucket.

The atlassian-bitbucket.log, when this happends, show:

2019-02-20 09:38:24,833 DEBUG [http-nio-7990-exec-2] @5L11PRx578x6308x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.b.i.c.s.CrowdSsoAuthenticationHandler Skipping Crowd SSO as it is not enabled
2019-02-20 09:38:24,833 DEBUG [http-nio-7990-exec-2] @5L11PRx578x6308x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ProviderManager.providerNotFound and locales [en_US, en]
2019-02-20 09:38:24,883 DEBUG [http-nio-7990-exec-6] @5L11PRx578x6309x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.b.i.c.s.CrowdSsoAuthenticationHandler Skipping Crowd SSO as it is not enabled
2019-02-20 09:38:24,886 DEBUG [http-nio-7990-exec-6] @5L11PRx578x6309x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.s.i.s.s.PluginAuthenticationProvider Authentication failed - Bad credentials
2019-02-20 09:38:24,886 DEBUG [http-nio-7990-exec-6] @5L11PRx578x6309x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.s.i.a.PluginHttpAuthenticationFailureHandler onAuthenticationFailure - delegating to com.atlassian.stash.internal.auth.RememberMeAuthenticationFailureHandler
2019-02-20 09:38:24,887 DEBUG [http-nio-7990-exec-6] @5L11PRx578x6309x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.s.i.a.PluginHttpAuthenticationFailureHandler onAuthenticationFailure - delegating to com.atlassian.stash.internal.rest.auth.RestAuthenticationFailureHandler
2019-02-20 09:38:24,887 DEBUG [http-nio-7990-exec-6] @5L11PRx578x6309x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.s.i.a.PluginHttpAuthenticationFailureHandler onAuthenticationFailure - delegating to com.atlassian.stash.internal.auth.ScmAuthenticationFailureHandler
2019-02-20 09:38:24,887 DEBUG [http-nio-7990-exec-6] @5L11PRx578x6309x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.s.i.a.PluginHttpAuthenticationFailureHandler onAuthenticationFailure - com.atlassian.stash.internal.auth.ScmAuthenticationFailureHandler handled authentication failure
2019-02-20 09:38:25,166 DEBUG [http-nio-7990-exec-9] @5L11PRx578x6310x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.b.i.c.s.CrowdSsoAuthenticationHandler Skipping Crowd SSO as it is not enabled
2019-02-20 09:38:25,422 DEBUG [http-nio-7990-exec-9] romulo.contini @5L11PRx578x6310x0 192.168.41.90 "GET /scm/crdweb/credbellweb.git/info/refs HTTP/1.1" c.a.bitbucket.scm.BaseCommand Executed /usr/bin/git http-backend
2019-02-20 09:38:25,977 DEBUG [http-nio-7990-exec-8] @5L11PRx578x6311x0 192.168.41.90 "POST /scm/crdweb/credbellweb.git/git-receive-pack HTTP/1.1" c.a.b.i.c.s.CrowdSsoAuthenticationHandler Skipping Crowd SSO as it is not enabled
2019-02-20 09:38:26,184 DEBUG [http-nio-7990-exec-8] romulo.contini @5L11PRx578x6311x0 192.168.41.90 "POST /scm/crdweb/credbellweb.git/git-receive-pack HTTP/1.1" c.a.bitbucket.scm.BaseCommand Executed /usr/bin/git http-backend

On the web interface of bitbucket, trying to access the /projects is shown a HTTP 500 error with the message

Handler dispatch failed; nested exception is java.lang.IncompatibleClassChangeError

And an exception is thrown:

2019-02-20 09:49:34,675 ERROR [http-nio-7990-exec-1] romulo.contini @5L11PRx589x6515x0 wng8fm 192.168.41.90 "GET /projects HTTP/1.1" c.a.s.i.w.HandlerExceptionResolverImpl Handler dispatch failed; nested exception is java.lang.IncompatibleClassChangeError
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.IncompatibleClassChangeError
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1053)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.bitbucket.internal.scm.git.lfs.servlet.filter.GitLfsLockingFilter.doFilter(GitLfsLockingFilter.java:50)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:37)
at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:38)
at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
at com.atlassian.bitbucket.internal.scm.git.lfs.servlet.filter.GitLfsLockingFilter.doFilter(GitLfsLockingFilter.java:50)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:87)
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.lang.Thread.run(Thread.java:745)
... 290 frames trimmed
Caused by: java.lang.IncompatibleClassChangeError: null
at org.hibernate.query.criteria.internal.path.AbstractPathImpl.get(AbstractPathImpl.java:127)
at com.atlassian.stash.internal.project.HibernateProjectDao.findAll(HibernateProjectDao.java:77)
at com.atlassian.stash.internal.project.DefaultProjectService.findAll(DefaultProjectService.java:171)
at com.atlassian.stash.internal.web.admin.ProjectAdminController.getProjects(ProjectAdminController.java:229)
... 32 common frames omitted

Since the push is been rejected by an authentication failure, I think is valid to inform that we use the JIRA User Directory.

When the bitbucket is restarted, all works well for some time. It´s been necessary to restart the service 3 or 4 times a day.

Since it was an upgrade, I already tried to recreate the Application Links between Bitbucket and Jira, but the problem persists.

[Stephen Sifers]

Hello Romulo,

Thank you for working through this issue initially and taking the time to send detailed information, steps you took, and errors logs with the caused by. These are extremely valuable.

1. First, I would suggest recreating the application link, but you’ve already taken this step (well done!)
2. Secondly, I would suggest reviewing the time sync between the Jira and Bitbucket server to ensure they’re in sync with one another. If they not in sync, this may cause authentication issues (especially as time drifts throughout the day).
3. Third, ensure your Server.XML files are configured properly per the new versions. Copying the file and replacing it from an older version may not work as expected due to changes within the Server.XML file itself. It is suggested to edit the new file and insert any customization after upgrading (Don’t replace the file with an older version).

There are other steps when you may review at Application Links Troubleshooting Guide.

If we’re still having an issue with the application link after attempting the above 2, please let us know.

Regards,
Stephen Sifers

[Romulo Ritter Contini]

Hi Stephen, thank you for your answer.

As I said before, I already tried the step 1 you provided and did not work. The problem still occours.

The second one does not apply because the Jira and Bitbucket services are hosted on the same machine and OS.

And the third one, I've never touched the Server.XML files.

What focus my attention is the fact that when the problem is fired, some funcionalities of Bitbucket still works, like the settings page. But when trying to open the /projects route, an exception with the message "Handler dispatch failed; nested exception is java.lang.IncompatibleClassChangeError" is shown on the screen.

Maybe another problem not related to application links?

My team uses multiple git client, including Sourcetree, Eclipse, Android Studio, Visual Studio Code.

The server was on git's version 2.14.1 and I updated it to 2.20.1. But it doesn't solve the problem.

[Romulo Ritter Contini]

Adding some info about the problem.

I recently try to visualize a Pull Request to review when the problem started. It was the second time today.

Here is the log on bitbucket:

2019-02-22 15:30:05,315 ERROR [http-nio-7990-exec-10] romulo.contini @1XUDRM7x930x667x0 gddgxn 192.168.41.90 "GET /rest/git/latest/projects/CRDWEB/repos/credbellweb/pull-requests/3754/rebase HTTP/1.1" c.a.p.r.c.e.j.ThrowableExceptionMapper Uncaught exception thrown by REST service: null
java.lang.IncompatibleClassChangeError: null
at org.hibernate.query.criteria.internal.path.AbstractPathImpl.get(AbstractPathImpl.java:127)
at com.atlassian.stash.internal.repository.HibernateRepositoryHookDao$2.visit(HibernateRepositoryHookDao.java:200)
at com.atlassian.stash.internal.repository.HibernateRepositoryHookDao$2.visit(HibernateRepositoryHookDao.java:185)
at com.atlassian.bitbucket.scope.RepositoryScope.accept(RepositoryScope.java:26)
at com.atlassian.stash.internal.repository.HibernateRepositoryHookDao.scopePredicate(HibernateRepositoryHookDao.java:185)
at com.atlassian.stash.internal.repository.HibernateRepositoryHookDao.lambda$scopesPredicate$1(HibernateRepositoryHookDao.java:208)
at com.atlassian.stash.internal.repository.HibernateRepositoryHookDao$$Lambda$1923/26873660.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:576)
at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:255)
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
at com.atlassian.stash.internal.repository.HibernateRepositoryHookDao.scopesPredicate(HibernateRepositoryHookDao.java:209)
at com.atlassian.stash.internal.repository.HibernateRepositoryHookDao.findByKeys(HibernateRepositoryHookDao.java:95)
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService.getHooksForScope(DefaultRepositoryHookService.java:678)
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService.lambda$getHooksForDescriptors$8(DefaultRepositoryHookService.java:655)
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService$$Lambda$1921/2105130193.doInTransaction(Unknown Source)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService.getHooksForDescriptors(DefaultRepositoryHookService.java:654)
at com.atlassian.stash.internal.hook.repository.DefaultRepositoryHookService.preUpdate(DefaultRepositoryHookService.java:360)
at com.atlassian.stash.internal.scm.git.pull.DefaultGitPullRequestService.canRebase(DefaultGitPullRequestService.java:101)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at com.atlassian.stash.internal.scm.git.rest.pull.GitPullRequestResource.canRebase(GitPullRequestResource.java:97)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24)
at com.atlassian.analytics.client.filter.UniversalAnalyticsFilter.doFilter(UniversalAnalyticsFilter.java:92)
at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
at com.atlassian.bitbucket.internal.xcode.web.XcodeUserAgentFilter.doFilter(XcodeUserAgentFilter.java:36)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.plugin.connect.plugin.auth.scope.ApiScopingFilter.doFilter(ApiScopingFilter.java:81)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:85)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.plugin.connect.plugin.auth.oauth2.DefaultSalAuthenticationFilter.doFilter(DefaultSalAuthenticationFilter.java:69)
at com.atlassian.plugin.connect.plugin.auth.user.ThreeLeggedAuthFilter.doFilter(ThreeLeggedAuthFilter.java:109)
at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:37)
at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:38)
at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:39)
at com.atlassian.bitbucket.internal.scm.git.lfs.servlet.filter.GitLfsLockingFilter.doFilter(GitLfsLockingFilter.java:50)
at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:87)
at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.lang.Thread.run(Thread.java:745)
... 338 frames trimmed

[Stephen Sifers]

Hello Romulo,

We have created a support request on your behalf to have your issue looked into. You should be receiving an email shortly from the support request.

If you do not receive the email, you may find your request at SSP-34640.

Regards,
Stephen Sifers

[Christian S]

Is there any public solution available already? Seems we have the very same problem. Didn't look at the logs yet, but symptoms are the same:

• Problem occured since update.
• Restarting Bitbucket temporarily solves the problem.
• Pushes, looking at projects and pull requests doesn't work. Rest seems to work.

[David Maye]

Hi Christian,

The issue appeared to be an older version of Java on the BB machine and after upgrading, it fixed the issue.

See https://confluence.atlassian.com/bitbucketserver/supported-platforms-776640981.html for supported Java versions.

Cheers,
-David
HipChat Server/Data Center Premier Support Engineer II | Atlassian

[Dovid Bender]

I had the same issue as well. I just updated Java, restarted BitBucket and everything started working.

[Alan Schwartz]

Experiencing the same issue. I'm running the BitBucket server in an AWS Linux machine. I already updated to the latest Java sdk 1.8 version which I think should aslo be supported.

[Dovid Bender]

@Alan Schwartz Make sure the version you installed is listed here https://confluence.atlassian.com/adminjiraserver072/supported-platforms-828787550.html . Not always is the latest version supported.