Shaping REST traffic patterns on AWS ALB?

[Jamie Gaard]

I was reading this article on load balancer traffic to a Jira DataCenter cluster - which mentions dedicating a node(s) entirely for incoming REST requests.

This is a very useful traffic pattern for my company's Jira instance that fields many external requests to it. Enough that it can affect performance for users. 

The article gives a recommended configuration to achieve this, which would be great, but in a box near the end it states that the recommendation won't work for AWS load-balancers. This is apparently because AWS uses its own traffic shaping policies. To achieve the desired REST traffic patterns I'll need to deploy 2 load balancers (according to the article) but fails to link to an example or explain more than that.

Hoping someone has solved this problem already and can explain a bit about what they did? I've looked across the web but no luck. Thanks in advance

[Thiago Masutti · Accepted solution]

Hi @Jamie Gaard 
I hope you are well.

The main drawback with ALB is that it doesn't support NOT conditions.

A possible way to overcome this is to base your rule by a list of user-agent that are tied to your internal automation.

Here's an example of what you can try configuring in your environment, translating it to the ALB configuration.
If the mobile App isn't used in your environment, then disregard that condition.

mobile_app_pool: nodes that can serve traffic from the mobile app
rest_pool: nodes that can serve traffic from rest api (automation, scripts, etc)
regular_pool: nodes that can serve traffic for regular access (browser)

If user-agent ~ "*AtlassianMobileApp*"
    Direct traffic to mobile_app_pool

If requested url ~ "/rest/*" && user-agent IN ("Apache-HttpClient*","python-requests*","Go-http-client*",<other-rest-user-agent>):
    Direct traffic to rest_pool

If none of the above matches
    Direct traffic to regular_pool

Kind regards,
Thiago Masutti

[Jamie Gaard]

Thank you for the insight, @Thiago Masutti

This is very helpful, but I do see a few problems with the implementation. 

1. It requires that I know all the user-agent values from all of the possible origins  of internal automations/scripts etc. Luckily we have this recorded in the catalina logs so it's just a mater of sifting through them. 
2. Any rest requests that aren't included in list above will end up going to the regular pool, including valid rest requests that originate from novel environments. 

This seems like a great solution in any case, so I think you. Few follow up questions, if you have the time.

1. Will this interfere with internal rest calls from Jira/plugins?
2. We use cookies to bind sessions to a specific node. Though I haven't tried this implementation yet I did read this KB about problems when using this configuration. Do you think that will be an issue here?
3. Why does the traffic shaping article mention 2 load balancers as a solution? Is that outdated or does another solution exist? 

[Thiago Masutti]

Hi Jamie.

When applying any type of LB routing rules, it's recommended to assess how that rule would impact your current environment.

Inspecting the access logs (ALB or Tomcat) may help identifying patterns on usage and rules would apply.

Personally I like to create reports with https://goaccess.io/ which might help giving a global view based on the access logs.

After applying any rule the admin would still need to periodically assess it for any improvements.

Regarding your questions:

1. Internal rest calls will be dealt by the last rule. On ALB the order matters, so that's why the default rule is at the end.

2. If you have any automation relying on cookie based authentication, then you always need to persist the application session cookie and the ALB cookie.

3. The article suggests using a different load balancer application, such as F5, Nginx, Apache, etc. You could also use an additional LB for REST API, but then you would need to instruct developers to use the different LB URL (which wouldn't be Jira's Base URL).

Kind regards,
Thiago Masutti