Jira no longer works after server restart.

[Deleted user]

We are using Jira Server deployed in AWS and we have deployed it behind an Application Load Balancer which allows both http and https.
We recently restarted our server due to high utilization.
When the server booted up, it started the jira app, but we can no longer access the jira dashboard.
Initially, we thought it was a Load Balancer issue so we disabled it and tried accessing jira via the IP. When we access the IP:8080, it gets redirected to https://IP:8443 which no longer works.

This was our server.xml settings before when we used Load Balancer:

<Connector port="8080"
maxThreads="150"
minSpareThreads="25"
connectionTimeout="20000"

enableLookups="false"
maxHttpHeaderSize="8192"
protocol="HTTP/1.1"
useBodyEncodingForURI="true"
redirectPort="8443"
acceptCount="100"
disableUploadTimeout="true"
bindOnInit="false"
scheme="https"
proxyName="jira.cloud.oleloapp.com"
proxyPort="443"
/>

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
keystoreFile="conf/keystore" keystorePass="s00perSeeekrit"
/>

I disabled the Load Balancer and re-configured server.xml:

<Service name="Catalina">

<Connector port="8080"

maxThreads="150"
minSpareThreads="25"
connectionTimeout="20000"

enableLookups="false"
maxHttpHeaderSize="8192"
protocol="HTTP/1.1"
useBodyEncodingForURI="true"
redirectPort="8443"
acceptCount="100"
disableUploadTimeout="true"
bindOnInit="false"

Even when I remove redirectPort="8443", it still forces redirection to https.

Testing simulations...

[root@ip-172-31-31-159 conf]# curl -v localhost:8080
* Rebuilt URL to: localhost:8080/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.53.1
> Accept: */*
>
< HTTP/1.1 302
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 UTC
< Location: https://localhost:8443/
< Content-Length: 0
< Date: Mon, 02 Apr 2018 08:11:19 GMT
<
* Connection #0 to host localhost left intact

[root@ip-172-31-31-159 conf]# curl -v https://localhost:8443
* Rebuilt URL to: https://localhost:8443/
* Trying 127.0.0.1...
* TCP_NODELAY set
* connect to 127.0.0.1 port 8443 failed: Connection refused
* Failed to connect to localhost port 8443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8443: Connection refused

[Mona Sood]

Last night I was getting issue in jira functioning which continues to loop into login page. I was getting some hipchat connector error in log file(related to token or identity). So I just removed all setting for SSL/certificates and runing my jira on http only,it was weird :(

[john morrissey]

Ah ok I see it's from a few hours ago so answer is different now and will post

[Andy Heinzer]

@Mona Sood Have you seen @john morrissey's post about using Jira behind AWS ELB with SSL offloading and http/https redirect?  Perhaps this could help you with your specific configuration here as well.

The title of this thread though also has me concerned about Jira getting hung during a restart.  There was a known bug where this could happen in some versions of Jira that is documented in https://jira.atlassian.com/browse/JRASERVER-62486

This bug has been fixed since the 7.9.2 and 7.10.0 versions, but in some previous versions it was possible that and end user trying to load certain jsp pages could cause Jira to fail to startup.

[Mona Sood]

Hi ,

I have similar setup but unfortunately I am not able to do redirection from http to https with8443.
Elb listener setup-
Elb:443 instance:8443

Jira server.xml:
Connector port 8080
Redirctport:8443
SSL setup-
Connector port:8443

This is not working as expected if i use url<http://JIRA.ABC> or <https://JIRA.ABC>
Rather with ipaddress of server it's working from ip:8080 to ip:8443

I am not sure what and where I am missing.

[john morrissey]

If the url<http://JIRA.ABC> or <https://JIRA.ABC> is the AWS ELB you can setup redirection on the ELB http listener configuration 

[Mona Sood]

First of all thank you for giving time on my query.
Unfortunately , we are using classic load balancer not application one.
And classic one is not giving redirection functionality.

Also ,I have gone though your post, when I started working on this requirement, that is stated by Andrew . I haven't used step 3 and 4 mentioned in your discussion.

When I tried elb:443 and instance:8080.
It did work but as soon as I added security constraints jira stoped working. Not sure why.

I have also followed Atlassian post s and they just asked to modify server.xml to add proxy name n port and scheme.
No need to modify web.xml

But if I go with elb cert offloading
It's simple but redirection is a challenge .

[john morrissey]

Ah ok the classic LB.  Can you or would you consider changing to an ALB?  It gives you more functionality where you can content switch based on FQDN to backend systems.  You can also black hole anything just trying to access the IP address by sending it to a target group with no entries in it 

[Mona Sood]

This am not sure whether my organization would proceed to go with ALB or not but as of now I am looking some solutions for classic one :(

For staging env I would try but again the constraints in prod as stated above.

Can end to end security be achieved by classic one?

Just elb:443 instance :8080 that too without adding security cons in web.xm

[john morrissey]

Yes if doing ssl offloading and you security policy allows for http between the ALB and App server I think you just need the redirect stuff in the server.xml.  No need for the security constraints and rewriting I believe.  

Our security policy mandates back end encryption also be enforced that's why we need the security constraints

[Mona Sood]

I think our security policies only allow access for certain set of ips (internal to the organisation) to chanellise over elb .

I will try backend encryption using step3 n 4 and let's see how it works for my applications .

Thank you for suggestion and it's helpful to proceed with easy security setup.

[Andy Heinzer]

I've seen this happen before on Jira instances that were configured to use SSL, but then you change those settings.   I can see that even after the changes you made to the port 8080 connector you still have the redirectPort="8443" parameter.   Sometimes this can still force the web connections to that to try to connect to another port instead.

But here is what I would recommend:

1. Stop Jira
2. Edit the server.xml once more to remove that redirectPort="8443" from the connector on port 8080
3. Save that file
4. Go into the $JIRAINSTALL/work/ directory
5. Clear the contents of this directory (we still want the /work/ directory to exist, but you can safely delete all the files and subfolders here when Jira is stopped)
6. Start Jira again

I have encountered this problem myself when trying to switch an install of Jira from HTTPS back to just HTTP.  I think that the work directory might be hanging on to a cached value here that is causing this to redirect when it should not be.   It's ok to delete the contents of that directory when Jira is stopped, when you start Jira again, it will recreate the files needed.

Please let us know the results.