Failed to add rrd metrics for key: Error

[Maria Bonello]

Hi

After upgrading from version 7.6.4 to version 7.9.2, we are getting the below error. We are using jira data center, with 2 nodes (Aincrad is the name of one of the nodes.) We've also changed the jira.index.lock.waittime from 30000 to 90000 from jira-config.properties file, however the error remained. 

2019-02-06 15:47:26,444 pool-29-thread-1 ERROR [c.o.s.r.diag.rrd.RrdManagerImpl] Failed to add rrd metrics for key: 20301
java.io.IOException: Could not obtain exclusive lock on file: /mnt/jira-db-shared/jira/scriptrunner/rrd/Aincrad/20301.rrd4j] after 3000 milliseconds
at org.rrd4j.core.RrdSafeFileBackend.lockFile(RrdSafeFileBackend.java:65)
at org.rrd4j.core.RrdSafeFileBackend.<init>(RrdSafeFileBackend.java:33)
at org.rrd4j.core.RrdSafeFileBackendFactory.open(RrdSafeFileBackendFactory.java:28)
at org.rrd4j.core.RrdBackendFactory.open(RrdBackendFactory.java:434)
at org.rrd4j.core.RrdDb.<init>(RrdDb.java:270)
at org.rrd4j.core.RrdDb.<init>(RrdDb.java:258)
... 2 filtered
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:258)
at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl.createOrOpenDatabaseFile(RrdManagerImpl.groovy:181)
at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl$createOrOpenDatabaseFile.callCurrent(Unknown Source)
at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl.addSample(RrdManagerImpl.groovy:146)
... 2 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnCurrentN(ScriptBytecodeAdapter.java:95)
at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl.this$dist$invoke$2(RrdManagerImpl.groovy)
at com.onresolve.scriptrunner.runner.diag.rrd.RrdManagerImpl$RrdDataProcessor.methodMissing(RrdManagerImpl.groovy)
... 2 filtered

Can someone advice regarding this error? Thanks 

[Andy Heinzer]

It looks like you might be seeing problems such as those described in Jira Data Center Functionalities Loss Due to Cluster Wide Lock.  It explains how one node in a data center cluster can create a lock globally, and then explains some ways to correct that.

However I'm not familiar with the code packages listed in your stacktrace at the beginning:

at org.rrd4j.core.RrdSafeFileBackend.lockFile(RrdSafeFileBackend.java:65)
at org.rrd4j.core.RrdSafeFileBackend.<init>(RrdSafeFileBackend.java:33)
at org.rrd4j.core.RrdSafeFileBackendFactory.open(RrdSafeFileBackendFactory.java:28)
at org.rrd4j.core.RrdBackendFactory.open(RrdBackendFactory.java:434)

I see below these are some scriptrunner packages being used.  However my concern is over these rrd4j packages.  Perhaps the version of these packages has not been updated to correspond to the changes in Jira and Jira data center.  

Could you share with us the plugins and the versions of those plugins you are using here with Jira Data Center 7.9.2?

[Maria Bonello]

Hi Andrew,

The ScriptRunner version is 5.4.48, which was updated few days ago from version 5.3.9. This is the only plugin that is seems to be used from the stacktrace.

Also, I went through the clusterlockstatus table and there isn't any node holding the cluster wide lock as all the records in the table has null datavalue in the locked_by_node field. 

[Joshua Yamdogo @ Adaptavist]

Hi @Maria Bonello 

Are you still having this issue? If you are, could you please a support ticket through our support portal here: https://marketplace.atlassian.com/apps/6820/scriptrunner-for-jira?hosting=server&tab=support

Regards,

Josh