Forums

Articles
Create
cancel
Showing results for 
Search instead for 
Did you mean: 

"Unexpected bytes from remote node, closing socket' when bringing up Confluence 7.19.x EKS cluster

robert.haskins September 17, 2023

We are deploying clustered single node Confluence 7.19.x instance on an EKS cluster on our journey to reliability. I am seeing a number of these error messages in the "atlassian-confluence.log":

Unexpected bytes from remote node, closing socket

In searching for resolutions to this issue, I found two Support articles:

https://confluence.atlassian.com/bitbucketserverkb/bitbucket-data-center-cluster-node-located-in-another-subnet-is-not-joining-the-cluster-1044086974.html

(I discounted this because the setup we have is very simple and everything is in one subnet)

https://confluence.atlassian.com/confkb/unexpected-bytes-from-remote-node-error-after-upgrading-confluence-data-center-to-7-18-1-or-later-and-several-other-versions-1138196530.html#:~:text=Cause,to%20start%20the%20next%20node.

(I discounted this because we have a single node cluster)

 

I also found this StackExchange article regarding a similar issue though on an EC2 cluster:

https://stackoverflow.com/questions/69134181/bitbucket-cluster-authentication-failed-due-to-hazelcast-group-name-and-hazelcas

 

Anyone have any suggestions as to what is causing this and how to fix it? Turn off Hazelcast authentication and see if we still have the issue? If so, what is the procedure for this? All I can find for references are for doing this manually on each node but I think I need a helm based solution:

https://confluence.atlassian.com/confkb/how-to-change-cluster-node-authentication-in-confluence-data-center-1141971144.html

2 comments

Comment

Log in or Sign up to comment
Mathias HD September 18, 2023

Hi, I don't know what your environment looks like or from which point you started.

You can just enable clustering and only have one node in the cluster. It seems to work like that in our environment. But be careful if you set up a test environment, as it will happily connect to your prod if you forget to disable it in the configs after the copying it.

How does your config looks like in the Clustering department? 

robert.haskins September 18, 2023 edited

Hi @Mathias HD 

Yes, the configuration you mention (one replica and clustering enabled) is essentially the setup we have. To wit:

replicaCount = 1

clustering.enabled = true

additionalEnvironmentVariables.name.ATL_FORCE_CFG_UPDATE.value = true

The issue is that we are getting those errors in our logs, and there are things not working as expected. For one, not being able to make "admin" privileged when running with 2 replicas -- it acts like I am putting the wrong password in, but I am putting the correct password in.

But I am getting these cluster errors when there is just one replica running, or two replicas running.

Yevhen
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
September 18, 2023

Wring password sounds like you don't have sticky cookies configured for your ingress (or loadbalancer, depending on how you exposed Bitbucket service).

As to node joining issue, it's weird you have it in a 1 node cluster. Can you please share the entire log?

robert.haskins September 18, 2023

@Yevhen Thanks so much for your help!

Regarding the incorrect password for the admin panel, it seems to have stopped having an issue after adding the following annotations to the Confluence values.yaml: (from: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/guide/ingress/annotations/)

alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60 alb.ingress.kubernetes.io/target-type: ip

Regarding the unxpected bytes error, that seems to have cleared up when I set "readinessProbe.initialDelaySeconds = 60"

Like • Yevhen likes this
TAGS
atlassian, likes for trees, atlassian community, social impact, tree planting campaign, community kudos, atlassian giving, environmental impact, sustainability, likes for good, atlassian social responsibility, community challenge

Make every click count—help us plant 50,000 trees! 🌳

Want to make your everyday Community actions directly contribute to reforestation? The Atlassian Community can achieve this goal by liking a post, attending an ACE, sending your peers kudos, and so much more!

Help us plant more trees
AUG Leaders

Atlassian Community Events