JIRA Project Sidebar not loading via IIS Reverse Proxy(HTTPS) but works fine with localhost-HTTP URL

[Venkat Ravichandran]

We've recently installed the JIRA 9.9.0 on our new Windows server and configured the DNS with IIS as a reverse proxy. The project sidebar is loading fine when we use the local host as a base URL(http://localhost:8080), but the same is failing to load with the custom DNS via IIS. The below HTTP GET calls are raising Bad Request(400) error and it's preventing the sidebar to load and giving an error "We couldn't load the project sidebar. Refresh the page to try again". The issue started happening after we migrate the JIRA software from an old Windows server to a new Windows server. We've followed all the steps provided in the Atlassian article such as IIS Configurations, URL Rewrite, and Proxy Configurations but it's still failing with 400 errors.

 

Failing URL:  https://xxx.customdns.com/s/0f324b341a73725e6c77af284f7119e3-CDN/-ew6rv8/990000/sd12uq/3173942cf55c369ec7f2e658b750bc29/_/download/contextbatch/css/jira.webresources:mentions-feature,jira.rich.editor.api,jira.rich.editor,-_super,-jira.view.issue,-jira.global,-project.issue.navigator,-atl.general,-com.atlassian.jira.jira-frontend-plugin:84,-com.atlassian.jira.plugins.jira-development-integration-plugin:0,-jira.webresources:calendar-localisation-moment/batch.css?agile_global_admin_condition=true&b...

Failing URL: https://xxx.customdns.com/s/be3bff21099c255e83dc4a71281b3e99-CDN/-ew6rv8/990000/sd12uq/3173942cf55c369ec7f2e658b750bc29/_/download/contextbatch/js/jira.webresources:mentions-feature,jira.rich.editor.api,jira.rich.editor,-_super,-jira.view.issue,-jira.global,-project.issue.navigator,-atl.general,-com.atlassian.jira.jira-frontend-plugin:84,-com.atlassian.jira.plugins.jira-development-integration-plugin:0,-jira.webresources:calendar-localisation-moment/batch.js?agile_global_admin_condition=true&bas... 

We've increased the URLSegmentLength and configured the proxy and HTTPS in the connector(Server.xml) file but nothing works. The following are the few articles that we followed to set up and configure JIRA and resolve the 400 errors but no luck.

Ref 1: https://confluence.atlassian.com/adminjiraserver072/integrating-jira-applications-with-iis-828788154.html

Ref 2: https://confluence.atlassian.com/jirakb/elements-in-jira-server-fail-to-render-with-400-bad-request-error-860033769.html

Ref 3: https://confluence.atlassian.com/jirakb/jira-users-receiving-the-error-400-bad-request-message-request-header-is-too-large-1032258616.html

There was a similar issue reported by someone in the past but there is no answer available in the post. 

URL: https://community.atlassian.com/t5/Jira-Service-Management/IIS-Proxy-Issue-with-Project-Sidebar/qaq-p/1612299?utm_source=atlcomm&utm_medium=email&utm_campaign=mentions_reply&utm_content=topic

Any help or guidance on this would be greatly appreciated.

[Artur Moura]

Hi @Venkat Ravichandran

While working on a similar case, the issue was related to the Reverse rewrite host in response headers parameter under the Application Request Routing configuration page on IIS.

Could you please confirm if this parameter is unchecked?

Hope it helps.

[Venkat Ravichandran]

Hi @Artur Moura ,

Thank you so much for your response! 

Yes, I've checked this attribute again in IIS now and can see the checkbox is already unchecked. Please refer to the screen shot below.

[Artur Moura]

You are very welcome @Venkat Ravichandran

OK, this is good. Thanks for confirming it. Let's try another thing now.

On IIS, you should have UrlRewrite rules under an IIS site or an IIS server level. 

Could you please confirm if you don't have any configuration applied to the server level that might be messing up with the connection?

The issue would be almost the same as mentioned in this KB.

[Venkat Ravichandran]

Hi @Artur Moura

Thank you for the assistance again!

The UrlRewrite rules are currently present at IIS site level not at the server level. I've also verified the settings mentioned in the KB article and it's exactly the same. Please refer to the screen shot below.

Our Infra team had already completed their investigation and confirmed that there is nothing blocking the request from server, network and firewall policies.

Please let me know if you need any further details. Thanks in advance!

[Artur Moura]

Hi again @Venkat Ravichandran

OK, the configuration looks good, so the next step is to double-check the server.xml file with the proxy settings you are using.

Are you using AJP or HTTP connector? If it is HTTP, is the connector looks like the sample config?

<Connector port="8080"

               maxThreads="100"

               minSpareThreads="10"

               connectionTimeout="20000"

               enableLookups="false"

               protocol="HTTP/1.1"

               redirectPort="8443"

               acceptCount="10"

               secure="true"

               scheme="https"

               proxyName="xxx.customdns.com"

               proxyPort="443"

               relaxedPathChars="[]|"

               relaxedQueryChars="[]|{}^\`&quot;&lt;&gt;"

               bindOnInit="false"

               maxHttpHeaderSize="8192"

               useBodyEncodingForURI="true"

               disableUploadTimeout="true" />

In addition, the BaseURL you have in Jira. Does it match with the proxyName/IIS site configuration?

Are you seeing any Gadget Feed URL errors on your Jira while using IIS?

[Venkat Ravichandran]

Hi @Artur Moura ,

We're currently using the HTTP connector and below are the configurations of Server.xml

<Connector 

maxSpareThreads="75" 

maxThreads="150" 

minSpareThreads="25"

port="8080" 

URIEncoding="UTF-8"

relaxedPathChars="[]|"

relaxedQueryChars="[]|{}^&#x5c;&#x60;&quot;&lt;&gt;"                    

connectionTimeout="20000"

enableLookups="false"

maxHttpHeaderSize="12288" 

protocol="HTTP/1.1"

useBodyEncodingForURI="true" 

redirectPort="8443"

acceptCount="100" 

disableUploadTimeout="true" 

bindOnInit="false" 

secure="true" 

scheme="https"

proxyName="CustomDNSName.com.au" 

proxyPort="443"/>

The proxy name configured in the above connector is same as the base URL(https://CustomDNSName.com.au) provided in the JIRA settings.

We couldn't see any gadget feed URL error in the JIRA application. The following are the errors captured in the browser console and it's again related to the project sidebar.

I've also run the log analyzer to check if it captures anything but it says "Hercules didn't find any known problems in the following log file"

Please let me know if you need any further details. Thanks in advance!

[Artur Moura]

Hi @Venkat Ravichandran

Thanks again for confirming it and for sharing the server.xml file.

So from Jira end, it seems all ok. Now we need to try to move the troubleshooting to the IIS side.

I found this IIS troubleshooting page. I believe it is worth trying to increase the MaxFieldLength and the MaxRequestBytes parameters a little bit.

Hoping this helps. Please let me know how it goes.

[Venkat Ravichandran]

Hi @Artur Moura 

Thank you for the assistance!

As per the article, I've created these new registry entries in the specified path as shown below but no luck.

We've also received the below Gadget Feed URL error after rebooting the server today but it occurred only once and not after that. I've also checked the health check menu by clicking the Find out more link in the below image and it's showing the result as "Jira is able to access itself through the Gadget feed URL"

The below HTTP/2 GET calls are failed again with same 400 errors.

2023-08-04 00:33:27 123.2.2.1 58679 192.x.x.x 443 HTTP/2 GET /s/3608a30700ca957c8d29e1f7ab136a1a-CDN/-ew6rv8/990000/sd12uq/2b6b456a75fc97d4b250d9c86888a34f/_/download/contextbatch/css/com.atlassian.jira.plugins.jira-development-integration-plugin:0,com.atlassian.jira.plugins.jira-quicksearch-plugin:5,com.atlassian.jira.jira-frontend-plugin:84,-_super,-jira.view.issue,-project.issue.navigator,-jira.global,-jira.project.sidebar,-jira.webresources:calendar-localisation-moment,-atl.general/batch.css?agile_global_admin_condition=true&baseurl-check-resources=true&healthcheck-resources=true&jag=true&jaguser=true&jira.create.linked.issue=true&richediton=true 381 - 400 - URL - TCP

2023-08-04 00:33:27 123.2.2.1 58679 192.x.x.x 443 443 HTTP/2 GET /s/f8b5697645ab53c7701598af5f4b1577-CDN/-ew6rv8/990000/sd12uq/2b6b456a75fc97d4b250d9c86888a34f/_/download/contextbatch/js/com.atlassian.jira.plugins.jira-development-integration-plugin:0,com.atlassian.jira.plugins.jira-quicksearch-plugin:5,com.atlassian.jira.jira-frontend-plugin:84,-_super,-jira.view.issue,-project.issue.navigator,-jira.global,-jira.project.sidebar,-jira.webresources:calendar-localisation-moment,-atl.general/batch.js?agile_global_admin_condition=true&baseurl-check-resources=true&healthcheck-resources=true&jag=true&jaguser=true&jira.create.linked.issue=true&locale=en-AU&richediton=true 383 - 400 - URL - TCP

Thanks in advance!

[Venkat Ravichandran]

Hi @Artur Moura , Hope you are doing well! Just following up on this one to check if you have any other recommendations or advise to troubleshoot the issue. Thanks in advance!

[Artur Moura]

Hi @Venkat Ravichandran

I'm sorry I missed your reply!

It's very weird behavior so far. 

I believe it is worth trying to increase the header size on the server.xml file.

Would you please try to increase the maxHttpHeaderSize parameter to 65536 and see if it works?

I'm mentioning this because I found this StackOverflow page where it seems the same issue.

[Venkat Ravichandran]

Hi @Artur Moura , No worries. Thank you so much for your response!

I've made this change and given a reboot but no luck.

I've also gone through the StackOverflow article and created some additional registry entries as well but the issue still persists. 

Thanks in advance!

[Tavon Hannah]

Has this issue been resolved. I am getting the same error with the matching IIS ARR and Tomcat configurations.

[Venkat Ravichandran]

Hi @Tavon Hannah , No, it's not resolved yet. 

@Artur Moura  Do you have any other suggestion for us to resolve the issue? if not, we're planning to use Ngnix as a reverse proxy instead of IIS for the JIRA software.

Thanks in advance!

[Artur Moura]

Hi @Venkat Ravichandran and @Tavon Hannah

If both of you bypass the IIS, can you reproduce the issue?

I'm asking this as a performance issue that might be related. 

[Venkat Ravichandran]

Hi @Artur Moura , For us, this issue occurs only with IIS and the project sidebar is loading fine when we use the local host as a base URL(http://localhost:8080), Please let me know if you need any further details.

[Jord Coerse]

folllowing this... 
Same issue and same circumstances