How often should we run the integrity checker for a large Jira instance?

[Jonas Silvano Lassi]

I'm one of the new admins of a large Jira data center instance with over 17k users. There are lots of activities going on, including migrations, scripting, and database manipulation (unfortunately). We're reviewing the governance, reducing admin access as much as possible and locking down database access.

We know that database data gets corrupted relatively quickly, given the multiple concurrent activities and mistakes along the way. 

We discovered that the integrity checker fixes most of the critical issues we've been facing, such as inconsistencies in workflows. Since it's a large installation, it is a painful process every time as we have to take the node out of the cluster, increase the heap memory, and make some additional configurations (reverse proxy) to ensure it doesn't get a time-out.

References: 

• https://jira.atlassian.com/browse/JRASERVER-17593
• https://confluence.atlassian.com/kb/proxying-atlassian-server-applications-with-apache-http-server-mod_proxy_http-806032611.html
• https://confluence.atlassian.com/jirakb/sql-equivalents-for-jira-server-s-workflow-integrity-checks-658179102.html

Now, our leadership wants us to run it monthly as part of keeping things as consistent as possible and to allow us to spot the actions/activities that have been causing those inconsistencies.

1) What recommendations and best practices would be around running the Jira native integrity checker? How frequently should we run it and why? 

2) I was hoping to find all SQL equivalents, but I only found the workflow ones. Is there any API, SQL, or script alternative that would allow us to execute it?

Thanks in advance.

[Nic Brough -Adaptavist-]

This is going to sound quite harsh, but your organisation needs to sack a load of people for incompetence.

Ideally, we never need to run the integrity checker.  In real life, we do, things happen like crashes, network outages, failover, DR testing, etc.  I've seen a couple of cases where a user's browser has messed up the issue they were working on.

There are very good times to run the Integrity Checker, and I'd actually like to see it built-in.  A regular one - yes, do that.  For my users, I'd run it daily, but as an admin, I want to see what it finds and fixes, so I get an idea of what is going wrong.

Now, the bad bit:

You said "database manipulation"

No

Never touch an Atlassian database

If you change the database behind an Atlassian system, you are stuffed - you have broken it.

Please can you tell me that "database manipulation" means "we don't touch the actual Atlassian database, we take a copy or feed and change that"

[Jonas Silvano Lassi]

I appreciate your candidness. Our team is taking over; we're now educating others and making the necessary adjustments to do things the right way.

I like they you put it. Ideally, we never need to run it, but we do in real life. We want to implement a similar approach as you mentioned.

You said you'd run it daily. Based on your experience with a large DC instance like ours, how frequently would you run it? Daily still? Also, how do you manage the time-outs or the system getting overloaded and going down because of the integrity checker? I imagine you dedicate one node to perform this task.

I totally agree with "the bad bit."

For the database manipulation, it was mostly to fix stuff in bulk and read-only tasks for reporting purposes. As I said, we're locking down database access. We're now considering a read-replica and tools like eazybi for reporting.

[Kevin Dalton]

Sorry to say there is really no automation for this and the SQL alternatives will cause you to need to re-index.  My recommendation is to set the admin logout higher than default and have a monthly recurring task to go in and handle this.  We run 3 large instances and run it once a month before we re-index. 

[Jonas Silvano Lassi]

Exactly. I couldn't find anything anywhere. What do you mean by setting the admin logout higher? Is this change enough to avoid getting time-outs in your case?

[Kevin Dalton]

We have to change the timeout due to the amount of time it takes for us to run the checker.   

https://confluence.atlassian.com/jirakb/change-the-default-session-timeout-to-avoid-user-logout-in-jira-server-604209887.html