Spike week day 1
We are currently having one of our quarterly spike weeks on our team, and for once I am managing to protect my own time and do my own spike. To help me remember what I've learned, I plan on writing what I did each day.
Our theme this time is infrastructure, and I've decided to automate my deployment pipeline. To do this I want to set up so that I can spin up new EC2 instances each time I deploy, and use AWS CodeDeploy to deploy to them. Currently I have a pipeline set up that sets off a build when I push to master in github, but I still need to ssh into my EC2 instance to pull down the build artifact from an S3 bucket. I wrote about how I initially set it up here, and it has moved on a little since.
My plan was to set up an EC2 template which would have all my current parameters like the AMI I am using, the security groups I have set up etc defined to make it easy to reproduce each time I launch a new instance. This part was very straightforward, it was simply a case of working through the options.
I started hitting issues when I went to configure it to install the CodeDeploy agent on start up. I was following the instructions in the documentation, but when I went in to check the CodeDeploy service was running, it wasn't. I went through a few times, checking the variables I was entering, trying out some variations I found in other tutorials. Nothing.
Next, I decided to launch an instance without the User Data instructions for launching the CodeDeploy agent and running them myself after the instance had launched. It worked.
Although this was incredibly frustrating, at least it reassured me that it should work.
These are the instructions I used from the documentation:
1#!/bin/bash 2apt-get -y update 3apt-get -y install ruby 4apt-get -y install wget 5cd /home/ubuntu 6wget https://aws-codedeploy-eu-west-2.s3.amazonaws.com/latest/install 7chmod +x ./install 8./install auto
You don't really get much indication about what is going wrong, so after several attempts to use the logs from the console which only occasionally showed me anything useful, I followed the instructions here to give me some logging data, and added some console logs to help me orientate myself, leaving me with the following instructions in the User Data:
1#!/bin/bash -xe 2exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 3echo "FIRST HELLO" 4apt-get -y update 5echo "hello after update" 6apt-get -y install ruby 7echo "hello after installing ruby" 8apt-get -y install wget 9echo "hello after installing wget" 10cd /home/ubuntu 11wget https://aws-codedeploy-eu-west-2.s3.amazonaws.com/latest/install 12echo "hello after getting the installer" 13chmod +x ./install 14./install auto 15echo "hello at the end"
This narrowed down the issue to the step where we install ruby. It was at this point I remembered with sinking horror that there had in fact been a prompt when I had run through line my line myself where I had had to select yes to continue. I hadn't taken much notice of it at the time, assuming that the -y command would skip through all of those. A bit more googling led me to this which is exactly what I had happily tabbed through when doing it by hand.
Thankfully, the suggestion to add
worked, giving me the following instructions to run on launch:
1#!/bin/bash 2export DEBIAN_FRONTEND=noninteractive 3apt-get -y update 4apt-get -y install ruby 5apt-get -y install wget 6cd /home/ubuntu 7wget https://aws-codedeploy-eu-west-2.s3.amazonaws.com/latest/install 8chmod +x ./install 9./install auto
I have no idea why this isn't in the instructions AWS provides. Maybe it doesn't always run like that? Anyway, what I've learned today:
- find a way to read the logs asap if it isn't already there
- take more notice of what I'm doing differently when I run through something step by step
- everything seems so easy afterwards