Bamboo OnDemand random Client.InvalidZone.NotFound error

[Lionel Orellana]

I have a custom AMI in the Sydney which I created based on the Bamboo stock image.

I can start a micro instance of this AMI from bamboo and attach an EBS volume from a snapshot.

I can start an m3.medium instane of this AMI from bamboo without attching the EBS volume.

Starting an m3.medium instance of the same AMI attaching the EBS volume doesn't work consistently. It was working this morning. Tonight I get

Creating EBS volume...

Found snapshotId at /root/snapshotId. Snapshot will be restored to /mnt/bamboo-ebs

Cannot find value for 'aws.ebs.deleteOnTermination'. Nothing is written.

mountLocation: /mnt/bamboo-ebs

attachedDeviceLocation: /dev/sdh

instanceId: i-02562d3d

amiId: ami-952db5af

availabilityZone: ap-southeast-2b

Thu Apr 24 11:31:26 UTC 2014

Using snapshotId snap-759bff41

Restoring volume from snap-759bff41...

Client.InvalidZone.NotFound: The zone 'ap-southeast-2b' does not exist.

ec2-create-volume failed (exit code 0). Output:

I have modified the .bash_profile of the root account in the AMI to do

export EC2_URL=https://ec2.ap-southeast-2.amazonaws.com

After doing that it all started working. Until now when it is failing again for no apparent reason.

-----------------------

As I finish writing this post I try to start the m3.medium instance again and it is working again.

So it seems that I am randomly getting this Client.InvalidZone.NotFound error.

Thanks for any help

Lionel.

[Przemek Bruski · Accepted solution]

Is it a recent stock image? Are you creating new image configurations for your tests or is it always the same configuration?

Try the following: start an instance from an image configuration without an EBS, then add the EBS to that configuration and start it again. It should work from then on.

[Lionel Orellana]

I copied ami-270b444e (3.0-SNAPSHOT-bamboo-snapshot-amzn-ami-pv-2013.03.1.x86_64-ebs-1372107828-BDM-1372609769-BDM1378218549) from us-east-1.

What you suggest is what I had done and perhaps that is why it is working again. But why would starting it without an EBS first fix it? Will it happen again? I think it's already happen twice.

[Przemek Bruski]

When you start it without an EBS, the agent starts and reports back the image version - only then Bamboo knows that it should start EBSes, by default it expect the image to do it. We are aware that this is a problem.

It should not happen again, unless there's a bug somewhere.

[Lionel Orellana]

Ok I'll keep an eye on it to see if it happens again. Is there a JIRA issue for this?