Bamboo Elastic Agents Stuck in Pending

[Jada White]

Hello,

My Bamboo setup has been broken ever since yesterday (25th Oct 2016) during the scheduled maintenance for Jira.

My Elastic Bamboo agents are forever stuck in Pending status and none of my builds can progress.

The elastic agents seem to come up with the following errors where user-data is not reachable or aws authentication fails when I attempt to start the agent:

```

[ec2-user@bamboo-feb-2015 ~]$ su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo

Starting Elastic Bamboo Agent...

Exception in thread "main" java.io.FileNotFoundException: This program must be run within an EC2 instance started from Bamboo: java.io.FileNotFoundException: http://169.254.169.254/2008-02-01/user-data

 at com.atlassian.aws.utils.URLFetcherUtils.fetchData(URLFetcherUtils.java:52)

 at com.atlassian.aws.utils.URLFetcherUtils.fetchString(URLFetcherUtils.java:40)

 at com.atlassian.aws.ec2.Ec2UtilsThin.getUserData(Ec2UtilsThin.java:47)

 at com.atlassian.aws.ec2.EC2Utils.getUserData(EC2Utils.java:89)

 at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.<init>(ElasticAgentInstaller.java:45)

 at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:65)

```

or 

```

[root@bamboo-feb-2015 ]# su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo

Starting Elastic Bamboo Agent...

2016-10-26 14:47:45,919 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-ap-se2/5.12.4-D20160704T134120/aa02e30788f7683b8f5431d112a3a6dec65b44065b68d45b46c102a9452b0435/ to /opt/bamboo-elastic-agent

2016-10-26 14:47:46,431 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-ap-se2/5.12.4-D20160704T134120/aa02e30788f7683b8f5431d112a3a6dec65b44065b68d45b46c102a9452b0435/ to /opt/bamboo-elastic-agent

2016-10-26 14:47:46,432 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-ap-se2]

2016-10-26 14:47:46,757 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-ap-se2], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 99AC87A327C26D7E), S3 Extended Request ID: Q/u6rusXcd2PbwJh9rjcW77f3iVj5I+XNx+R+AEz2vuZZ92vZHWCfCTNIHz9f/c0a1/3Hx8tFmc=

2016-10-26 14:47:46,757 INFO [main] [S3Utils] Fetching the list of remote objects...

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 981D011696993749), S3 Extended Request ID: gTbXq8/RbI5PFy8oADxXEHoJsAVEqSFM2yg6JttmGufgv102cgsvrYlztVnL0G4n2z+Q3fGl8C8=

 at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)

 at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)

 at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)

 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3714)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3664)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:637)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)

 at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:354)

 at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:162)

 at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)

  at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

```

Ive tried running latest updates. no luck. searching for the aws authentication errors due to requiring a valid date or x-amz-date header, showed a lot of issues >1 year ago with joda-time version and aws sdk.. not sure of the relation, but seems to be the same error. i have attempted to use latest joda-time but no luck.

Additionally, I can connect to the tunnel port 26224 fine.

I am really at a loss with this one.....

[dylantack-metaltoad]

I had this issue when the agent was starting too early. Using systemd on CentOS 7, I resolved it by adding the following to the service unit:

[Unit]
After=network.target cloud-config.target

 This makes sure that cloud-init and the network is ready before the bamboo agent starts.

[Przemek Bruski]

Is this a custom image? Which version of Bamboo is it?