"Unexpected bytes from remote node, closing socket' when bringing up Confluence 7.19.x EKS cluster

[robert.haskins]

We are deploying clustered single node Confluence 7.19.x instance on an EKS cluster on our journey to reliability. I am seeing a number of these error messages in the "atlassian-confluence.log":

Unexpected bytes from remote node, closing socket

In searching for resolutions to this issue, I found two Support articles:

https://confluence.atlassian.com/bitbucketserverkb/bitbucket-data-center-cluster-node-located-in-another-subnet-is-not-joining-the-cluster-1044086974.html

(I discounted this because the setup we have is very simple and everything is in one subnet)

https://confluence.atlassian.com/confkb/unexpected-bytes-from-remote-node-error-after-upgrading-confluence-data-center-to-7-18-1-or-later-and-several-other-versions-1138196530.html#:~:text=Cause,to%20start%20the%20next%20node.

(I discounted this because we have a single node cluster)

I also found this StackExchange article regarding a similar issue though on an EC2 cluster:

https://stackoverflow.com/questions/69134181/bitbucket-cluster-authentication-failed-due-to-hazelcast-group-name-and-hazelcas

Anyone have any suggestions as to what is causing this and how to fix it? Turn off Hazelcast authentication and see if we still have the issue? If so, what is the procedure for this? All I can find for references are for doing this manually on each node but I think I need a helm based solution:

https://confluence.atlassian.com/confkb/how-to-change-cluster-node-authentication-in-confluence-data-center-1141971144.html

[Yevhen · Accepted solution]

Wring password sounds like you don't have sticky cookies configured for your ingress (or loadbalancer, depending on how you exposed Bitbucket service).

As to node joining issue, it's weird you have it in a 1 node cluster. Can you please share the entire log?

[robert.haskins]

@Yevhen Thanks so much for your help!

Regarding the incorrect password for the admin panel, it seems to have stopped having an issue after adding the following annotations to the Confluence values.yaml: (from: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/guide/ingress/annotations/)

alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60 alb.ingress.kubernetes.io/target-type: ip

Regarding the unxpected bytes error, that seems to have cleared up when I set "readinessProbe.initialDelaySeconds = 60"

[Mathias HD]

Hi, I don't know what your environment looks like or from which point you started.

You can just enable clustering and only have one node in the cluster. It seems to work like that in our environment. But be careful if you set up a test environment, as it will happily connect to your prod if you forget to disable it in the configs after the copying it.

How does your config looks like in the Clustering department? 

[robert.haskins]

Hi @Mathias HD 

Yes, the configuration you mention (one replica and clustering enabled) is essentially the setup we have. To wit:

replicaCount = 1

clustering.enabled = true

additionalEnvironmentVariables.name.ATL_FORCE_CFG_UPDATE.value = true

The issue is that we are getting those errors in our logs, and there are things not working as expected. For one, not being able to make "admin" privileged when running with 2 replicas -- it acts like I am putting the wrong password in, but I am putting the correct password in.

But I am getting these cluster errors when there is just one replica running, or two replicas running.