Need help with SSL on a 64-bit RHEL server running Jira v7.4.1 and confluence v6.3.1

[John Ellis]

We recently had to move our jira and confluence instances over from a 32-bit to a 64-bit server due to new confluence versions being pushed out only being 64-bit. We had SSL connections setup and working on our previous server. At one time I had it working on this 64-bit server; until I upgraded jira and confluence to the latest versions. I saved then copied the server.xml files back over into the appropriate folders as I knew they got overwritten but still cannot get the SSL connection to work for either jira or confluence since the upgrades done to them.

[Nic Brough -Adaptavist-]

Ahh, the joys of https - always the right thing to do, but a pain in the neck to get it right

How are you doing your SSL?  Terminated at a proxy, or with JIRA and Confluence's Tomcats?

[John Ellis]

I used a free certificate from cacert.org, used portecle to set it all up then edited the server.xml files with all the needed settings for secure port, secure store location, etc. As I say I did have it working until I uprgaded jira and confluence to the latest versions and now they will only come up on the 8080 and 8090 ports; not the 8444 and 8443 ports that they were working on before. If any of this does not make sense please keep in mind that I am not a programmer or Linux guru and I had never even heard of Jira, Confluence, or Atlassian until I took this position a little over 5 years ago. All I know about them has been OJT! In fact, one of my 1st tasks here was to upgrade jira and confluence and they were so far back in their versions I had to do it in three steps.

[Nic Brough -Adaptavist-]

Ah, ok.  The upgrade process has removed your settings.  I did a JIRA 7.4 upgrade the other day, and that at least had the decency to tell me it had trashed my server.xml, cacerts and setenv.sh files.  From memory, older versions do it silently.

[John Ellis]

Yes I had allso finally figured out that a jira upgrade always replaces the exiting server.xml file with a new one but as I said I had saved the server.xml after I edited it to point to the secure port; in this case 8443 and then after the jira upgrade I copied that server,xml file back into the jira/bin folder and restarted jira so it seems to me that it should be working on the SSL port but it still doesnt. I can only get to jira on the non-secure port; 8080.

[Nic Brough -Adaptavist-]

Copying the old file back might not be the right thing to do, as it may have changed as you upgraded.

Also, server.xml does not go in jira/bin, it's for the <jira install>/conf directory, and you should take a look at cacerts and setenv.sh as well, in case those have changed.

[John Ellis]

Sorry yes, my bad; I knew the server.xml file goes into the /conf folder. I think I remember that I did have to start editing the server.xml file even after an upgrade rather than copying the file back over but I thought that only happened due to upgrading from one complete version up to the next. Thanks, I will check into this as well.

[John Ellis]

Nic I have replied to this email thread but it looks like maybe you are not seeing these. Can you confirm if you have gotten any email replies to this thread today?

Thanks,

John Ellis

[John Ellis]

Nic one of the things that I emailed you about was that when I was looking at the Jira server.xml file just now I saw this right in the middle of it- "Note, you no longer configure your database driver or connection parameters here.
These are configured through the UI during application setup."

I wonder if that means we can no longer edit this file to setup SSL?

[John Ellis]

Maybe that's why I can't ever get a jira login on the secure port (8443) when I edit the server.xml file to use it?

[Nic Brough -Adaptavist-]

Yes, I've not been emailed about this until your last post!  I'll blame my email servers for now, until I know better.

I need to step back and establish what you're doing here though.  Otherwise I could chase a load of problems that I'm not qualified to talk about. 

Almost all of my experience with Atlassian stuff is with it running behind a proxy server where the SSL is terminated.  A user connects to the proxy, using SSL, the proxy fowards/retreives the actual data over unencrypted http to the server that only talks to the proxy.

Is that what you are doing?  Or have you gone with implementing SSL inside the Tomcat?

[John Ellis]

No it is not setup through a proxy server; I have no idea how to even do that. As I think I stated before, I am trying to get the SSL to work inside Tomcat; that's why I am trying to modify the server.xml files to point towards the secure ports of 8443 for jira and 8444 for confluence. Over the weekend I received several email messages indicating a temporary problem with Atlassian emails not being delivered so that's probably why you didn't get any of them.

Thanks again for your help,

John Ellis

[Nic Brough -Adaptavist-]

Ok, did you follow https://confluence.atlassian.com/adminjiraserver074/running-jira-applications-over-ssl-or-https-881684081.html ?  Starting from the server.xml that was installed by the installer, rather than your old one?

[John Ellis]

Yes I did. After upgrading jira to the latest version then I used the Portecle plugin to create a keystore and went through all those steps to create the cert. csr the imported the signed cert. back in. I then tried to edit the server.xml file to change the port from 8080 to 8443 but once I made that change I could not access jira at port 8443, still only at 8080. When I was looking at the server.xml file is when I saw that notice I saw in the middle of the server.xml file that I previously posted; "Note, you no longer configure your database driver or connection parameters here.
These are configured through the UI during application setup."

Please note that I was NOT able to use the config.sh GUI to setup the https part. Due to an issue with the java  version I am having to do the https change all manually, by editing the server.xml files.

[Nic Brough -Adaptavist-]

Ok, we'll need to see the server.xml really.  If it's still active on 8080, there must be something lingering in there telling it to do that after the restart.

[John Ellis]

I can't figure out any way to attach the server.xml file here so this is a DropBox link to go into the folder called 64-bit server to look at the server.xml file that I modified-

https://www.dropbox.com/sh/3xrdvb7st3h7q4l/AABFqFhN1CH-jO_sUWA62R-Xa?dl=0

[Nic Brough -Adaptavist-]

Ok, I'd expect that to leave your JIRA apparently available on port 8080 as well as 8443, but redirecting.

You say you're not getting a login - does it churn and timeout, or does it immediately give you an error message when you visit the url?

What do the logs say?

[John Ellis]

Nic I just tried it again and I told you wrong; once I try changing the server.xml file jira will not startup on 8443 OR 8080, the original port it gives the error almost immediately; within a second or so. Here is a DropBox link to the log files that were created just now when I tried it.

https://www.dropbox.com/sh/wqsiu6nfjngdd17/AAD5rkxY2i9g_xEghCH93RCza?dl=0

Thanks,

[Nic Brough -Adaptavist-]

Ah, that makes sense!  I didn't spot it before because I thought it was running ok, but not serving.

  Have a re-read of https://confluence.atlassian.com/adminjiraserver074/running-jira-applications-over-ssl-or-https-881684081.html and note the red box at the top!

[John Ellis]

OK that fixed the SSL issue but now I have a weird issue with the Jira dashboard. I noticed this happening when I tried to access jira outside the office but now it is doing it inside as well. Don't know if you have ever seen this or not. Some of the Titles & entries don't look right on the dashboard now. Here is a link to it in DropBox-

https://www.dropbox.com/s/p9hmft3x12nny83/64-bit_ssl_jira_issue.jpg?dl=0

Thanks,

[John Ellis]

I also noticed that if I just try to connect to jira on the 8080 port it gives me a login page but when I try to login it just keeps going back to the login page; never actually lets me login. However I can login fine on the 8443 port now.

[Nic Brough -Adaptavist-]

Ok, the 8080 should be redirecting, as you don't really want to use it.

The _msg_ problem with gadgets is a known one with SSL behind a proxy - it means the server cannot talk to itself through the proxy for some reason, usually that the client certificate is not installed/accessible in the JVM.  I suspect it's the same for direct Tomcat SSL termination as well.

[John Ellis]

OK when I 1st logged into the jira on the SSL port it gave me an "unsecure connection" page first and also said the certificate was not valid for this site so maybe that's causing this, even though the certificate seemed to install OK?

[Nic Brough -Adaptavist-]

Yes, a certificate install doesn't mean it's valid.

[John Ellis]

OK that makes sense. So now I now I can at least login on the sure port I will focus back on getting the right SSL certificate to work on this server. I will go back to the 8080 port for now so the Dashboard looks right.

Thanks for all your help Nic.

[John Ellis]

Nic I have been trying to edit my confluence server.xml file to support the secure port 8443 but when I add the https part at the bottom it keeps giving me errors;-

"Using Java: /opt/atlassian/confluence/jre//bin/java
[Fatal Error] server.xml:35:2: The content of elements must consist of well-formed character data or                                                                                                  markup.
[Fatal Error] server.xml:35:2: The content of elements must consist of well-formed character data or                                                                                                  markup.
The content of elements must consist of well-formed character data or markup.
[Fatal Error] server.xml:35:2: The content of elements must consist of well-formed character data or                                                                                                  markup.
The content of elements must consist of well-formed character data or markup.
2017-08-22 10:28:18,475 INFO [main] [atlassian.confluence.bootstrap.SynchronyProxyWatchdog] The Conf                                                                                                luence context path is empty. No further action is required."

Here is a link to the server.xml file; if you could take a look and see what you think the problem is that would be great.

https://www.dropbox.com/s/ywusit7r84jkzug/server.xml?dl=0

Thanks again,

John Ellis

[Nic Brough -Adaptavist-]

There's something broken in the file, but it's being lost as you copy and paste.

Could you open the file with an xml editor that could check the format for you?

[John Ellis]

[John Ellis]

Downloaded a free XML editor program; It shows the error to be on line 35 but doesn't tell me what I need to do to fix it- just says:

"Error at line 35, column 2, not well-formed (invalid token)"

[Nic Brough -Adaptavist-]

Have a look at the bit it's underlined in red!  Editors with syntax highlighting and checking are really handy, and much better at spotting small errors than I am!

[John Ellis]

Nic I see where it gives me the line that has the error I just don't know what to change to fix it. Do you have a working confluence server.xml file that is setup for SSL running in Tomcat? Maybe I could just compare them to see what's wrong with mine. I had confluence setup this way on our old server but it died so I can't look at those xml files.

[Nic Brough -Adaptavist-]

Sorry, I assumed you knew enough xml to spot it.  It's most of a comment enclosure, but it's missing a shriek.  Have a look up the file for other examples.

[John Ellis]

My background was in computer hardware until I took this position here at LSG. I don't understand what you mean by "missing a shriek" and I googled it as well.

Sorry to be such a dummy here.

Thanks,

 John Ellis

[John Ellis]

Say Nic I kept searching on this forum and found a server.xml file setup for SSL that someone had put on a post so I pasted the part I needed into mine and now I have confluence running on SSL. Now I have to go back and try to get the certificate for Jira setup correctly so it won't give me those title issues on the Dashboard.

Thanks again for all your help.

John Ellis

[John Ellis]

Wow Nic this is so frustrating. Last week I said that I finally got the SSL port to work for jira but this week it is no longer working. Now the symptom is that when I try to go to my jira instance on port 8444 it just churns but when I go back to port 8080 I get a jira login page just fine. Here is a DropBox link to the edited jira file-

https://www.dropbox.com/s/ywusit7r84jkzug/server.xml?dl=0

If you can take a look at it that would be great.

Thanks!

[Nic Brough -Adaptavist-]

What did you change in the file?

[John Ellis]

All I changed was adding the part on line 88, that sets up the SSL port. Last Thursday it was working on port 8444 but then when I checked it on Friday it no longer worked and I had made no changes to the server.xml since it was working.

[John Ellis]

I can put the "before" server.xml file out on Dropbox as well if that would help?

[Nic Brough -Adaptavist-]

You jsut need to check what you changed - that should explain what has broken.

[John Ellis]

That is what I am saying is so frustrating. Last Monday I finally edited the jira server.xml file and got it to work on the ssl port; 8444 but that was when I had the issues with some of the titles not showing up correctly on the dashboard. You said that was probably due to a problem with the SSL certificate so I copied back the previous server.xml file. Then last Friday I finally got the ssl port 8443 to work on confluence so I then copied the edited jira server.xml that had the ssl settings back into jira and restarted it and now it doesn't work. So the same exact jira server.xml file that DID WORK for ssl last Monday now DOES NOT work.

[John Ellis]

Nic I was thinking yesterday; do you think it would be easier to get the SSL and everything working if I reconfigured this 64-bit server as running Windows rather than Linux? How are the ones you have setup running as? Just trying to figure out some way to get this resolved easily.

Thanks,

[Nic Brough -Adaptavist-]

You won't get anything from changing OS, if anything, Windows is harder to make work.

If you had it working before and then changed something, then undid the change and it doesn't work, then you must have missed changing something back.

[John Ellis]

I didn't change anything in the server.xml files; I changed the files themselves. I had copied the server.xml file before I edited it for SSL and so I just copied that file back into the jira/conf directory. I worked on this again on Friday and now I can't get a jira login at all. When I try to go to the SSL port it times out but when I try to go back to the non-secure port of 8080, which was working before, now it goes to the login page for confluence on the secure port of 8443. I even tried changing the jira port from 8080 to 8081 but it still goes to the confluence secure login port of 8081.

[John Ellis]

I finally figured out the issue with the jira login secure port going to the confluence login page. In editing the jira server.xml file I didn't change one of the port "redirect" commands. Still trying to get a valid certificate installed on it though.