502 Bad Gateway

[arcadiaengineering]

This morning, due to no action on our part, our confluence installation on aws shows a 502: Bad Gateway error. Stopping and restarting the EC2 instances where confluence is installed solves the problem....temporarily. Shortly afterwards (in a couple of hours), the site goes down with the same error again.

Everything was working fine up until yesterday, we've made no changes to the AWS setup or installation files at all. How can we fix this?

[arcadiaengineering · Accepted solution]

Formal answer: we were attacked from the vulnerability detailed here: https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337

Following the steps in that thread fixed it for us (kill hkugepads process, clear cron jobs).

[Diego]

Hello there!

Could you clarify some points for us?

1. Which version of Confluence are you running?
2. Is CPU usage unusually high in your server?

Looking forward to your reply!

[arcadiaengineering]

Hi, thanks for offering to help. Answers below.

1. How do I find my confluence version? I cannot access confluence from the url, so I can't just go to the admin panel and find out (bad gateway error). But I can access it via the command line. How do I find out/what command/which file?

2. CPU usage according to AWS is at 100%. That's pretty high, right? Unfortunately I don't have statistics of CPU usage before this morning, because when I stopped and restarted my AWS instances to see if I could solve the problem, they were somehow automagically terminated and re spun/re-built. So the ones I currently have running are somehow "new" ones. Stopping/starting the AWS instances solved the bad gateway problem for an hour or two, then it came back.

[arcadiaengineering]

Our version is 6.12.2 .

I was able to get confluence back up by following instructions in this thread:

https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337

[Diego]

Hey there!

Great to know that you solved the issue! I was a little late to reply here but thankfully everything is ok.

Thanks for sharing your Connie version. With this information I can recommend you to upgrade your instance to at least 6.12.4. This is due to these two CVEs:

Confluence Security Advisory - 2019-03-20 | March 2019 Confluence Server Advisory - WebDAV and Widget Connector vulnerabilities

Confluence Security Advisory - 2019-04-17 | Confluence - Path traversal vulnerability - CVE-2019-3398

Also, try running another malware check on your instance just to be sure that everything is in order.

Further! Before the upgrade or any other change you may apply to your instance, backup these:

• <confluence-home> folder
• <confluence-install> folder
• Database

More information:

Confluence Home and other important directories

Let us know your thoughts!

[arcadiaengineering]

I just ran top on the confluence server. It claims command khugepageds is taking 399% of the CPU. Is this helpful info? Not sure what to do next...

[arcadiaengineering]

Looks like we may be suffering from this same attack: https://community.atlassian.com/t5/Confluence-discussions/khugepageds-eating-all-of-the-CPU/td-p/1055337

Following the steps in that thread, will post if still need more assistance. Thx.

[Gary Pasquale]

Hi @arcadiaengineering ,

A 502 error isn't going to tell you much with further investigation.

I'd start by following this guide:

https://blog.hubspot.com/marketing/502-bad-gateway

That will be a good starting point.

Regards,

Gary