URGENT - Elastic Bamboo Instances always stuck on 'Pending'

[Rohan Eldridge]

My Elastic Bamboo instances never transition from 'Pending'. What then happens is that the auto balancer kills the instance and restarts it every ~10 minutes or so! Where can I find the logs to diagnose this problem? I have pasted some of my logs at the bottom of this post but they provide zero information to help with solving this problem. Where can I find more useful logs or how can I work out what the problem is?

I have the following simple instance startup script in my Elastic Image configuration:

sudo yum install -y php-mysql

sudo yum install -y php-mysqli

sudo yum install -y php-pdo

sudo yum install -y php-pecl-xdebug

I originally had more commands in the instance startup script but removed them to try and resolve this problem.

The AMI ID of my Elastic Image is ami-430a452a. The name of the Elastic Image is "Amazon Linux: Default Image S3 x86_64 (linux) (stock image)".

Here is all that I get in the logs:

2013-11-10 02:19:19,855 INFO [main] [AgentContext] Bootstrap Type: 'ELASTIC'

2013-11-10 02:19:19,963 INFO [Thread-2] [RemoteAgent] com.atlassian.bamboo.agent.elastic.client.ElasticAgent starting.

2013-11-10 02:19:20,231 INFO [Thread-2] [RemoteAgentHomeLocatorForBootstrap] Agent home located at '/home/bamboo/bamboo-agent-home'

2013-11-10 02:19:20,239 INFO [Thread-2] [RemoteAgent] Starting Bamboo Remote Agent...

2013-11-10 02:19:20,240 INFO [Thread-2] [RemoteAgent] Starting application context...

2013-11-10 02:19:22,326 WARN [Thread-2] [PropertyPlaceholderConfigurer] Could not load properties from class path resource [bamboo-init.properties]: class path resource [bamboo-init.properties] cannot be opened because it does not exist

2013-11-10 02:19:22,367 INFO [Thread-2] [RemoteAgentHomeLocatorForBootstrap] Agent home located at '/home/bamboo/bamboo-agent-home'

2013-11-10 02:19:22,368 INFO [Thread-2] [RemoteAgentHomeDirectoryLayout] Plugin directory: /home/bamboo/bamboo-agent-home/plugins

2013-11-10 02:19:22,368 INFO [Thread-2] [RemoteAgentHomeDirectoryLayout] Plugin cache directory: /home/bamboo/bamboo-agent-home/caches/plugins

2013-11-10 02:19:22,410 WARN [Thread-2] [AgentConfiguration] Configuration file does not exist. Creating File: bamboo-agent.cfg.xml

2013-11-10 02:19:22,524 INFO [Thread-2] [AgentUuidInitializer] Found agent UUID '3d1fc27e-c6c9-4805-a4b9-c295097134cd' in temporary UUID file '/home/bamboo/bamboo-agent-home/uuid-temp.properties'

2013-11-10 02:19:22,524 INFO [Thread-2] [AgentConfiguration] Setting UUID in main config file to '3d1fc27e-c6c9-4805-a4b9-c295097134cd'

2013-11-10 02:19:22,580 INFO [Thread-2] [BambooActiveMQConnectionFactory] Setting broker URL to 'failover:(tcp://localhost:4527?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&initialReconnectDelay=15000'

2013-11-10 02:19:22,728 INFO [Thread-2] [BambooActiveMQConnectionFactory] Setting broker URL to 'failover:(tcp://localhost:4527?wireFormat.maxInactivityDuration=300000)?maxReconnectAttempts=10&initialReconnectDelay=15000'

2013-11-10 02:19:23,121 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-3] Accepted a local connection on port 4527, tunnel age: 7s

2013-11-10 02:19:33,141 WARN [Thread-2] [FailoverTransport] Transport (tcp://127.0.0.1:4527) failed, reason: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format., attempting to automatically reconnect

2013-11-10 02:19:48,142 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-4] Accepted a local connection on port 4527, tunnel age: 24s

2013-11-10 02:20:43,093 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-3] Accepted a local connection on port 4527, tunnel age: 54s

[mfobrien NA]

I am having the same issue launching a bamboo EC2 instance inside a private subnet on my VPC under 172.21.1.0.  I have a 2 zone VPC with NATs running on the public 0 subnet.

I am able to connect to the instance only via a bastion running on the public subnet by tunnelling to the private bamboo instance.

I checked that the above hostname issue is fixed as of Jan 2015

ec2-user@ip-172-21-1-46 ~$ hostname
ip-172-21-1-46
ec2-user@ip-172-21-1-46 ~$ host ip-172-21-1-46
ip-172-21-1-46.ec2.internal has address 172.21.1.46

I manually associated an EIP (from the VPC console) and this did not help - so the issue is not EIP - because a non-VPC 172.31 (amazon default) public EC2 instance works fine with just a public IP and no EIP.

ports are fine 22 and

tcp 0 0 :::26224 :::* LISTEN -

Running on the public 172.21.0.0/26 subnet works via the public IP - but this is not a secure workaround

see related

Bamboo can't start Amazon Elastic Instance agent after being added to VPC

[Krystian Brazulewicz]

I was able to replicate this problem. In order to fix it I needed to add the following line to /etc/hosts on the Bamboo server instance:

10.0.1.113  ip-10-0-1-113

please adjust to your hostname/IP address. This change is related to the fact that bare EC2 instance is unable to resolve its own name:

ec2-user@ip-10-0-1-113 bamboo-home]$ hostname
ip-10-0-1-113
[ec2-user@ip-10-0-1-113 bamboo-home]$ host ip-10-0-1-113
Host ip-10-0-1-113 not found: 3(NXDOMAIN)
[ec2-user@ip-10-0-1-113 bamboo-home]$ host ip-10-0-1-113.ec2.internal
ip-10-0-1-113.ec2.internal has address 10.0.1.113

[Aleksey Razbakov]

I have the same issue.

My Elastic agent is Pending

[Steffen Opel]

@Rohan - you can check whether you image configuration is in a VPC by going to 'Bamboo admin -> Elastic Bamboo -> Image Configurations' and clicking 'Edit' on the image(s) currently in use. The combo box '<label id="fieldLabelArea_saveElasticImageConfiguration_vpc" for="saveElasticImageConfiguration_vpc">Virtual private cloud subnet</label>' needs to be set to 'None' rather than a VPC subnet, if any (see Managing your elastic image configurations for details).

That being said, I doubt it will be the problem, so a few more questions in case:

• Have your agents always been running in that particular AWSaccount in us-east-1, i.e. they have neither been migrated to a different region or AWS account?
  • Regardless, is this a comparatively new AWS account? In case, could you confirm via the AWS Management Console, whether you are running on the new 'EC2-VPC' platform or can use both 'EC2' and 'VPC' separately (section 'Supported Platforms' in the top right of the EC2 dashboard)?
  • In case you are running on the new 'EC2-VPC' platform, is this AWS account used for other activities too, i.e. is it possible that somebody changed things in that account?

[Przemek Bruski]

It basically means that the agent was unable to contact the server. Does it work if you don't use an instance setup script?

[Rohan Eldridge]

No, it doesn't work even if I remove the instance setup script. It still stays stuck on 'Pending'.

Am I using an old/wrong Bamboo EC2 agent image?

[Przemek Bruski]

No, it does not look like a wrong image. Do you use a VPC?

[Rohan Eldridge]

According to the EC2 console, no Elastic IPs are currently in use in our Bamboo Elastic Agent region (ie. US East). There is also no VPC set up in that region. Any idea as to why the Agents aren't being given Elastic IPs then as I assumed that was a automatic process?

I shall look into the Elastic IP management as well.

[Rohan Eldridge]

I believe we are using a VPC for the actual application that we have running on our AWS account - it has public and private subnets. I just checked the Bamboo Elastic Agent Instance and it doesn't have an Elastic IP, which I assume means it can't talk to the outside world correctly?

Is there a guide or documentation for checking our AWS setup to ensure it'll work with Bamboo?

I know recently we did run into a problem of running out of Elastic IPs, we did request more from AWS however and they gave them to us.

[Przemek Bruski]

Whether you have VPCs enabled on that account doesn't really matter - what matters is whether your image configuration is configured to use VPC.

BTW. you don't have to have EIPs allocated if you enabled EIP management - Bamboo will allocate and deallocate them.

[Rohan Eldridge]

Actually, sorry, we have a VPC set up but it is in the Sydney region - could this still affect the Bamboo agent which runs in a different region (US East)?

[Przemek Bruski]

Yeah, it needs an elastic IP. To get it, you can enable Elastic IP management in Bamboo, but make sure you configure Bamboo to ignore all EIPs you have already allocated on that account.

[Rohan Eldridge]

nvm, see other comment. i think i was getting myself confused.

[Rohan Eldridge]

OK, Elastic IP management doesn't help as we have no Elastic IPs currently in use in that EC2 region. The Elastic IP limits are also per region... so that shouldn't be stopping anything.

[Rohan Eldridge]

I'm guessing that "Enable elastic IP management" will fix the problem... so I guess my new question is, why do I need to do this now when before we didn't need to modify this setting?

It's not the end of the world if we do need to enable that setting, I'm just curious as we didn't need to enable that setting before and we had Elastic Agents working correctly. I'm just trying to understand why things have changed and what we could have done wrong.

[Rohan Eldridge]

Actually... that then raises another question - was it ever possible to run Elastic Agents without an Elastic IP? Cause according to the guy who set up the original (customised) image, it didn't need an Elastic IP...

[Rohan Eldridge]

OK, looking at the original image it's not customised, it seems to be based on a stock image, which is ami-a10b44c8. It does have a snapshot attached but I don't think that would affect how it interacts with the network?!?

As a result my assumption would be that the instances have always needed Elastic IPs and for some reason either a Bamboo setting was changed or the Elastic IPs were manually allocated at the time and then removed later.

I guess enabling "elastic ip management" will fix things, but our Bamboo admin doesn't want me to do that...

Anyway, thanks for your help - and please correct me if any of my conclusions are wrong.

[Przemek Bruski]

You don't need to configure EIPs at all if your image configuration (not the image) is not deployed into a VPC.

[Rohan Eldridge]

OK, how can I confirm that the image configuration is not in a VPC?

As far as I can tell we're not using VPCs (in that region). Would creating a new 'Image configuration' help?

[Rohan Eldridge]

Sorry for all the questions, I'm just confused at to why it has stopped working when it was working. As a result I'm trying to figure out what has changed and if it is a Bamboo setting OR a problem with our AWS.

So it's not just a case of how to fix it, it's also a case of figuring out why it broke :)

[Rohan Eldridge]

Sorry for the bump... kinda urgent problem... :(

[Steffen Opel]

@Rohan - you can check whether you image configuration is in a VPC by going to 'Bamboo admin -> Elastic Bamboo -> Image Configurations' and clicking 'Edit' on the image(s) currently in use. The combo box Virtual private cloud subnet needs to be set to None rather than a VPC subnet, if any (see Managing your elastic image configurations for details).

That being said, I doubt it will be the problem, so a few more questions in case:

• Have your agents always been running in that particular AWS account in us-east-1, i.e. they have neither been migrated from a different region or AWS account?
  • Regardless, is this a comparatively new AWS account? In case, could you confirm via the AWS Management Console, whether you are running on the new 'EC2-VPC' platform or can you use both 'EC2' and 'VPC' separately (section 'Supported Platforms' in the top right of the EC2 dashboard)?
    • In case you are running on the new 'EC2-VPC' platform, is this AWS account used for other activities too, i.e. is it possible that somebody changed things in that account?

[Rohan Eldridge]

Yeah, I have no option in that drop down to pick a subnet.

1. We've always run them in US East as we're using all the original images. As for availability zone, that has always been left on automatic.

2. We have EC2 and VPC seperate under 'Supported Platforms'.

3. Someone may have changed things on the account that I'm unaware of, but from what I can tell there is no VPC or Elastic IPs in use currently in the region. The only thing in the region is an Elastic Beanstalk based EC2 app.

[Przemek Bruski]

Maybe someone modified the elasticbamboo (or ControlledByBamboo) security group? Try deleting these groups.

[Rohan Eldridge]

Sorry but I've been told to leave this alone now as it was taking too long to investigate/resolve.

Thank you for helping...