Jira Crashed. Giving java.lang.OutOfMemoryError: Java heap space

[Tayyab Bashir]

Hi, 
My JIRA server crashed and in the logs its continuously giving Java Heap space and Out of memory errors. 

I am also using a third party plugin "Bug watcher".
There are many errors from that aswell, But before that JVM ran out of space.  

java.lang.NullPointerException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
	at com.google.common.collect.SingletonImmutableList.<init>(SingletonImmutableList.java:41)
	at com.google.common.collect.ImmutableList.of(ImmutableList.java:80)
	at net.java.ao.DatabaseProvider.renderAction(DatabaseProvider.java:244)
	at net.java.ao.schema.SchemaGenerator$1.apply(SchemaGenerator.java:117)
	at net.java.ao.schema.SchemaGenerator$1.apply(SchemaGenerator.java:114)
	at com.google.common.collect.Iterators$8.next(Iterators.java:782)
	at net.java.ao.schema.SchemaGenerator.migrate(SchemaGenerator.java:92)
	at net.java.ao.EntityManager.migrate(EntityManager.java:133)
	at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.migrate(EntityManagedActiveObjects.java:43)
	at com.atlassian.activeobjects.internal.AbstractActiveObjectsFactory.create(AbstractActiveObjectsFactory.java:52)
	at com.atlassian.activeobjects.internal.DelegatingActiveObjectsFactory.create(DelegatingActiveObjectsFactory.java:39)
	at com.atlassian.activeobjects.osgi.ActiveObjectsServiceFactory.createActiveObjects(ActiveObjectsServiceFactory.java:136)
	at com.atlassian.activeobjects.osgi.ActiveObjectsServiceFactory.access$000(ActiveObjectsServiceFactory.java:51)
	at com.atlassian.activeobjects.osgi.ActiveObjectsServiceFactory$1$1.get(ActiveObjectsServiceFactory.java:65)
	at com.atlassian.activeobjects.osgi.ActiveObjectsServiceFactory$1$1.get(ActiveObjectsServiceFactory.java:61)
	at com.atlassian.activeobjects.osgi.DelegatingActiveObjects$MemoizingSupplier.get(DelegatingActiveObjects.java:149)
	at com.atlassian.activeobjects.osgi.DelegatingActiveObjects.count(DelegatingActiveObjects.java:121)
	at sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at sun.proxy.$Proxy1379.count(Unknown Source)
	at com.metainf.jira.plugin.watch.ao.GenericDao.count(GenericDao.java:53)
	at com.metainf.jira.plugin.watch.listener.DefaultNotificationManager.getCountUnread(DefaultNotificationManager.java:169)

[Chander Inguva · Accepted solution]

Hey Muhammad,

Please find following articles

This is because of Bug Watcher Notifications plugin

https://confluence.atlassian.com/display/JIRAKB/Unable+to+obtain+a+connection+from+the+underlying+connection+pool

and 

https://confluence.atlassian.com/display/JIRAKB/JIRA+slow+and+throwing+error+There+was+an+error+getting+a+DBCP+datasource+in+logs

Please try to increase Java JVM with increments of 512Mb as instructed in https://confluence.atlassian.com/jira/increasing-jira-memory-5896.html

 Hope these will help you resolve issue.

Regards

Chander

[Tayyab Bashir]

Yes its happening because of the bug watcher. Any idea how is this happening? And what can we do to prevent it? Because we do use bug watcher extensively. (until this happened)

[Chander Inguva]

As suggested by Nic above, contact Bug Watchers Support and report the issue of Memory Leak. https://marketplace.atlassian.com/plugins/com.metainf.jira.plugin.bug-watcher/server/support Support Link: http://www.meta-inf.hu/wiki/display/PLUG/Support;jsessionid=C43ADE445C8028FF6E51C9ED5D6A3D17

[Nic Brough -Adaptavist-]

How much memory is it running with?  Have you tried increasing that?

[Tayyab Bashir]

JVM has 250MB min. and 1GB max

[Tayyab Bashir]

Following are the 3 prominent errors occurring repeatedly in the logs. h4. Error 1: 2016-01-08 11:19:56,598 http-bio-8080-exec-878 ERROR sali 674x8237567x13 1wj37br 10.17.4.4,10.64.255.52 /rest/api/1.0/menus/find_link [common.error.jersey.ThrowableExceptionMapper] Uncaught exception thrown by REST service java.lang.OutOfMemoryError: Java heap space h4. Error 2: 2016-01-08 11:20:59,498 http-bio-8080-exec-943 ERROR dbar 680x8237744x37 wvleq 10.0.1.67,10.64.255.52 /rest/quickwatch/1.0/notification/count/unread [NoModule] There was an error getting a DBCP datasource. java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool h4. Error 3: 2016-01-08 11:59:44,002 http-bio-8080-exec-13 ERROR smun 719x12603x2 1l38r3w 10.17.4.28,10.64.255.52 /rest/quickwatch/1.0/notification/count/unread [plugin.watch.rest.NotificationResource] java.lang.NullPointerException

[Nic Brough -Adaptavist-]

Ok, have you tried increasing the memory for it?

[Tayyab Bashir]

No, I haven't tried increasing the memory. I'll try doing that on Monday

[Nic Brough -Adaptavist-]

Ok, bear in mind that this is just a diagnostic change - if it helps, then we know that you're just overloading the memory. If it does not prevent the error, then we know it's resource leak somewhere!

[Tayyab Bashir]

Hi Nic, I did increase the JVM but the errors were still coming. I think its because of a third party plugin (Bug Watcher). Whenever i disable the plugin, the errors stop. I've tried deleting the cache, reinstalling the plugin, just installing the bug watcher (disabling everything else); in each case, errors are produced as long as bug watcher is running. Just a couple of days back it was running fine, I don't know why this behavior is occurring. Do you have any ideas about it?

[Nic Brough -Adaptavist-]

Yes, you need to remove the bug-watcher plugin. Are you using a valid version for your version of JIRA? If you are not, then move to a valid version. If you are, then I'm afraid you need to report an issue with the developers of the add-on and wait for them to fix the memory leak it is causing.

[Tayyab Bashir]

{code} Exception in thread "http-bio-8080-exec-889" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space at org.apache.tomcat.util.http.parser.SimpleCharStream.<init>(SimpleCharStream.java:261) at org.apache.tomcat.util.http.parser.SimpleCharStream.<init>(SimpleCharStream.java:270) at org.apache.tomcat.util.http.parser.HttpParser.<init>(HttpParser.java:226) at org.apache.catalina.connector.Response.setContentType(Response.java:804) at org.apache.catalina.connector.ResponseFacade.setContentType(ResponseFacade.java:245) at javax.servlet.ServletResponseWrapper.setContentType(ServletResponseWrapper.java:123) at com.atlassian.core.filters.encoding.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:39) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at com.atlassian.jira.web.filters.PathMatchingEncodingFilter.doFilter(PathMatchingEncodingFilter.java:45) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.atlassian.jira.startup.JiraStartupChecklistFilter.doFilter(JiraStartupChecklistFilter.java:74) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(ChainedFilterStepRunner.java:84) at com.atlassian.jira.web.filters.JiraFirstFilter.doFilter(JiraFirstFilter.java:57) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) {code}