Why is JIRA 4.1.2 upon startup, calls to delete from db all users

[Elijah]

Yes it is Saturday, yes it is nice outside and yes we are at work 

We are troubleshooting a very large and unknown issue. System loads some pages but other pages the site hangs until it errors out. We have many levels of various HW attached so it may not be the issue.

Anyhow, right now I have 2 focused questions.

1. We noticed that in the log upon startup that the system logs a db statement to delete every user that is in the system. This is the log statement for the query: EXECUTE <unnamed> = DELETE FROM PUBLIC.MEMBERSHIPBASE WHERE USER_NAME=$1 AND GROUP=$2
2. Another thing we are confused about is that the user account which is managed in LDAP is entered into the JIRA db 2 times for each user. 1 is lowercase the other is uppercase. Is this a feature of JIRA or is it something wrong with the setup of our instance through some configuration file?

We greatly appreciate any help any of you out there can provide.

[Daniel Wester · Accepted solution]

For #2 - did you have an internal directory previously where these users could have been created? Then you added the LDAP directory? If that's the case I think they'd be listed twice.

I would be more concerned about the site hanging only on certain pages. Which pages? Do you have a common add-on on these pages?

[Elijah]

@Daniel Wester [Wittified] I greatly appreciate your response.

From my understanding, LDAP has always been used and it has been a known issue for a couple years from what I have heard. So basically what occurs is that a user is added to LDAP and then the user logs into the site to activate that user and then it is in the JIRA database 2 times for the user.

The hanging occurs on several areas.

1. Administration : User Browser page 
   1. On our test servers after we do a reboot of the server, the User Browser page hangs for about 5 minutes. After it loads then it comes up within a second or 2.
   2. On our production server after we do a reboot of the server, the User Browser page hangs for about 5 minutes. Every reload of the page takes just as long to load it. Sometimes we get a timeout issue.
2. Create a ticket page
   1. I'm not sure how JIRA runs by default but when the user clicks on the top navigation to create a ticket the page loads to select an organization and a ticket type. User then clicks next and the page just hangs and never loads and end up with an error page saying Internal error but nothing in any log.
3. Network path 
   1. User calls to the server run in this fashion. Call comes into Data Power --> Apache -->Tomcat -->LDAP
   2. On a call that does not hang, we can see log entries everywhere
   3. On a call that hangs we don't see a log anywhere except for Data Power for the exception when it times out saying something along the lines of 'No header response'

[Daniel Wester]

Are you getting any errors in the atlassian-jira.log or catalina.out in Tomcat? It really sounds like you've got an issue in the Tomcat/Database area. 5 minutes to compile the jsp and initialize the cache for users doesn't sound right to me. I'd suggest dropping in a support.atlassian.com ticket. Out of curiously - how's the performance of the database and tomcat? Either one of the them to close to be maxing out cpu or memory(both system wide and assigned) or connections? What's the latency between JIRA and the database?

[Elijah]

No errors are logged in the jira or tomcat logs that help us in any direction to look. We know on Tomcat the memory gets eaten up quickly and we have 8GB. That has been an ongoing issue for a very long time where we have been forced to restart the service once a week to regain memory. Thats another issue to look into as to why it gets eaten up. As for the Latency between JIRA and the DB I'm not sure of and the people that can tell us are not in until Monday morning. Our SA is submitting a ticket right now. Do you by any chance know of a diagram that explains what JIRA does behind the scene when the server starts up and starts to try and delete from the public.membershipbase? We would love to see some process guide UML flow something to help us know what goes on. We have none and know no SME with JIRA and are just best guessing our way through this.

[Elijah]

Another thing just for one days log we had over 36 million queries to EXECUTE <unnamed> = DELETE FROM PUBLIC.MEMBERSHIPBASE WHERE USER_NAME=$1 AND GROUP=$2 and we have no idea what calls that or why it is called. Can you tell us anything about this query that execute from JIRA?

[Elijah]

Another important aspect of what we are dealing with. If the user uses the web services to submit data they do not have any issues pushing 1 or 100 items. It is only through the GUI that the application will totally hang. There have not been any updates pushed to the application in 2 months and the problem had started on July 1st after a patch to the Postgres EnterpriseDB was made. Patch was a minor patch and I put a request into EnterpriseDB asking for what was in that patch. As for requesting help from Atlassian, although we have a support contract or whatever its called they responded quickly to our request for help with something along the lines of '4.1.2 is not supported try the community for help'. I'm sorry but any place that would just shove you off rather than to at least provide an attempt to help is not much in the way of customer service. I told management yesterday they can save the customer money by dropping the service for support since we won't ever get any support for the version of JIRA we are stuck with.