Problem Re-Indexing with Groovy Runner plugin

[Andy Grace]

I had an index fail with NoClassDefFoundError - this then meant that all Jira functionality was lost (couldn't search for or create issues, admin tools all messed up)

Stopping Jira, deleting indexes and removing plugins from installedplugins and then re-starting overcame and I gradually added plugins re-indexing as I went and re-index went fine at each stage

I upgraded Groovy Runner from 2.1 Beta 7 to 2.1 Beta 8 and I didn't get the NoClassDefFoundError but the re-index did fail, this time with Too Many Open Files

I upped ullimit to 4096 and it now re-indexes successfully

I don't have that many Groovy Files (1 scripted field is a file and a few workflow transitions have groovy files) and this is a test system with relatively low volume of issues so I'm reticent to move it to production until I overcome the issue.

Any ideas why it might be keeping too many files open (or indeed why the first time failed with NoClassDefFoundError and not the 2nd when I didn't change any scripted files or script transitions/fields)?

[Andy Grace · Accepted solution]

Was resolved by https://jamieechlin.atlassian.net/browse/GRV-211 apologies for the late answer

[ITSupport]

I was wondering if there is a way to disable indexing Custom Scripted Field made by Script Runner? In our environment we are creating html content. Apparently, when we remove groovy script and add "return '' " indexing works fine but when complete script is generating html index is corrupt.

[JamieA]

to disable indexing just set the custom field to have no indexer.

[ITSupport]

Thank you very much for your tip! Accept my apologies for not thinking and rushing with question.

[JamieA]

no apology necessary, but thanks.

[SABVARX]

Thank you for posting the solution on how to disable the indexer! I was really in troubles because the index couldn't run through with script runner enabled. We have two html customfields. After editing the customfiels to have "no search template" it all works fine again.

[Pierre EK]

We have the same problem here - we had it before and it still exists with the upgraded versions of JIRA and Script Runner. We are now at JIRA 6.3.7, and Script Runner 3.0.6.

If we keep Script Runner enabled, and perform a Reindex of JIRA (full or background), the indexes become corrupted. The problem is solved by disabling Script Runner and relaunching the indexing 

I am surprised that we are just a few to report this problem, even though it has been existing with our company for a while. We are keeping the plugin disabled, and we're enabling it only when we need to Copy a project in the Built In scripts section...

[JamieA]

Are you sure that's the reason - can you test with a script that just returns null?

What indexer are you using?

[Sumit Kumar]

is this resolved ?? I am also facing the same issue.

Script Runner version :: 2.1.9 .. index goes corrupt after adding the scripted fields into the project.

when a null is returned as an output of the issue on which the scripted field are loaded.

[JamieA]

> NoClassDefFoundError and not the 2nd when I didn't change any scripted files or script transitions/fields

Which class was not found?

No... sounds improbable that nothing changed but you got different results. Have you got the log files?

> I upped ullimit to 4096 and it now re-indexes successfully

4096 is really very low... the ulimit for open files is probably not going to be related to the volume of issues.

[JamieA]

Hi Andy,

> it would need to open a file for each issue that has a scripted field that has groovy file?

Possibly... although it would also close it. Seems related to https://jamieechlin.atlassian.net/browse/GRV-206 - maybe you could watch that issue.

[Andy Grace]

Hi Jamie

Thanks for the super quick answer - the only change between the 2 times was upping the version of the groovy plugin.

It didn't specify the class that it couldn't find

I upped ulimit from 1024 to 4096 as outlined as it

https://confluence.atlassian.com/display/JIRAKB/Loss+of+Functionality+due+to+Too+Many+Open+Files+Error

I'm not sure how the indexing works and whether it would need to open a file for each issue that has a scripted field that has groovy file?

I do have log files and screenshots - what would be the best way to get them to you?

Cheers

Andy

[Andy Grace]

Would I be better changing to inline groovy for the custom fields and transitions rather than opening the file?

Here is a snippet of the log from the first failure the NoClassDef.. so it appeared then that it failed to find the class for the Index failure.. but the original error was the same too many files open - I suspect it must have failed to read from the class path due to too many files open issue

Exception in thread "ISSUE-indexQueue:thread-1" java.lang.NoClassDefFoundError: com/atlassian/jira/index/DefaultIndex$Failure

at com.atlassian.jira.index.DefaultIndex.perform(DefaultIndex.java:45)

at com.atlassian.jira.index.QueueingIndex$Task.index(QueueingIndex.java:144)

at com.atlassian.jira.index.QueueingIndex$Task.run(QueueingIndex.java:125)

at java.lang.Thread.run(Unknown Source)

...

at java.lang.Thread.run(Unknown Source)

Mar 8, 2013 1:09:22 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run

SEVERE: Socket accept failed

java.net.SocketException: Too many open files

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(Unknown Source)

at java.net.ServerSocket.implAccept(Unknown Source)

at java.net.ServerSocket.accept(Unknown Source)

at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)

[Andy Grace]

Hi - so it seems GRV-206 is not related to my problem but to the groovy version.

I've unearthed something deeper in the log - there was a nullpointer being thrown on one of my equalsIgnoreCase checks which was occuring on a null object

Perhaps the script runner is not closing files when hitting exceptions like this and that is why it is now hitting the Too many open files issue?

Shall I create a GRV issue so I can upload logs etc?

[JamieA]

No, it will be not be leaving files open. Why don't you see what files are open with lsof? You can create an issue, sure, but I'm not seeing a real problem yet.

[Andy Grace]

OK I now have it narrowed down to one scripted field that uses a file - if I delete that field it indexes fine, with the field there it dies with too many files open

It could just be that the script is taking a while to pull results so that during indexing more and more files get open

I'll raise an issue and send you the lsof output and further details