ATTENTION: Update 2020: Have a look at Keptn.sh for the next generation of the Unbreakable Pipeline Feel free to move on with this tutorial for AWS but be aware that we have taken these concepts developed back in 2018 and brought them over to our Open Source project Keptn. Keptn provides both automated delivery and operations. It integrates well with any tools due to its event-driven delivery choreographay. It also implements SLI/SLO-based Quality Gates and allows for automatic roll backs of bad deployments. If you want to learn more about Keptn make sure to check out the website and Keptn Tutorials

AWSDevOpsTutorial

Unbreakable DevOps Pipeline Tutorial with AWS CodeDeploy, AWS CodePipeline, AWS Lambda, EC2 and Dynatrace.

The goal of this tutorial is having a full end-to-end AWS DevOps Pipeline (Staging, Approval, Production) that is fully monitored with Dynatrace. With Dynatrace injected into the pipeline we implement the following use cases:

1. FullStack Monitor your Staging Environment
2. Shift-Left: Automate Approve/Reject Promotion from Staging to Production based on Performance Data captured while running continuous performance tests
3. FullStack Monitor your Production Environment
4. Self-Healing: Automatic Deploy of previous revision in case Dynatrace detected problems in Production

ATTENTION: The Tutorial will create several AWS Resources such as 2 EC2 Instances, CodePipeline, CodeDeploy, Lambdas, S3 and a DynamoDB table. In order to save costs make sure to power down the CloudFormation stack after you are done with your tutorial.

Online Performance Clinic: I ran an Online Performance Clinic which you can watch on our Dynatrace Fullstack Live Performance Clinics Channel You can also download the slides I used!

Lets get this party started: Before we launch the CloudFormation stack which will create all required resources (EC2 Instances, Lambdas, CodeDeploy, CodePipeline, API Gateway) lets make sure we have all pre-requisites covered!

Pre-Requisites

1. You need an AWS account. If you don't have one get one here
2. You need a Dynatrace Account. Get your Free SaaS Trial here!
3. Optional: In case you want to use Ansible as automation platform, you need a license for Ansible Tower. Get your license here.
4. You need to clone or copy the content of this GitHub repo to your local disk!

Preparation

Amazon Web Services (AWS)

As we are going to use AWS CodeDeploy, AWS CodePipeline, AWS Lambda, DynamoDB, API Gateway and EC2 make sure the AWS Region you select provides all these services. We have tested this cloud formation on US-West-2a (Oregon) and US-East-2b (Ohio). To be on the safe side we suggest you pick one of these regions!

1. Create an EC2 Key Pair for your AWS Region! Our CloudFormation Template needs an EC2 Key Pair! 1.1. To learn more about Key Pairs and how to connect to EC2 Instances for troubleshooting read Connect to your Linux Instance
2. Create a S3 Bucket with the naming scheme: <yourname>-dynatracedevops and enable versioning. See following screenshots for reference
3. Copy the content from the folder "copytos3" to your newly created S3 bucket. This includes the application package, tests, monspec as well as all Lambda functions. If you want to use Ansible for Auto-Remediation, make sure to also copy the Ansible Tower license and Ansible playbook. The Ansible License file you have to be provide has to be named ansible-license.txt.
4. for ANSIBLE TOWER auto-remediation only: make both the ansible-license.txt as well as the playbook.yaml public in order for the script to download it to the corresponding EC2 instances during installation. Therefore, select both files and click on "More -> Make public" and submit the dialog.

Dynatrace

We need a couple of things to launch the CloudFormation Template

1. Your Dynatrace Tenant ID
2. Your Dynatrace Tenant URL: For SaaS that would be something like http://<yourtenant>.live.dynatrace.com. For Managed it would be http://<yourserver>/e/<your-env-id>
3. Your Dynatrace OneAgent for Linux Download URL: Go to Deploy Dynatrace -> Start Installation -> Linux and copy the URL within the quotes as shown below:
4. A Dynatrace API Token: Go to Settings -> Integration -> Dynatrace API and create a new Token with default settings (screenshot below may differ to what you see. Just go with defaults)

(optional) Setup Dynatrace AWS CloudWatch Monitoring

Dynatrace has a built-in feature to pull in metrics and metadata (e.g: tags ) from CloudWatch. You can setup this integration as explained in the documentation: How do I start Amazon Web Services monitoring?

1. Lets create the CloudFormation Stack

If you haven't cloned or downloaded the full GitHub repo then download the CloudFormation stack definition from here.

Now click on one of the regional links. This brings you to the CloudFormation Web Console where you can upload your downloaded CloudFormation Stack Definition.

Region | Launch Template ------------ | ------------- Oregon (us-west-2) | <a href="https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Oregon</a> N. Virginia (us-east-1) | <a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=dynatracedevopsstack" target="_blank">Launch Dynatrace DevOps Stack into Virginia</a> Ohio (us-east-2) | <a href="https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Ohio</a> Ireland (eu-west-1) | <a href="https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Ireland</a> Frankfurt (eu-central-1) | <a href="https://console.aws.amazon.com/cloudformation/home?region=us-central-1#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Frankfurt</a> Tokyo (ap-northeast-1) | <a href="https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Tokyo</a> Seoul (ap-northeast-2) | <a href="https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-2#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Seoul</a> Singapore (ap-southeast-1) | <a href="https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-1#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Singapore</a> Sydney (ap-southeast-2) | <a href="https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=dynatracedevopsstack" target="blank">Launch Dynatrace DevOps Stack into Sydney</a>

Step 1: Upload your File and click next

Step 2: Define all your input parameters

Step 3: Click through all other wizard steps. Confirm the box and click CREATE

Step 4: Refresh that screen and watch CloudFormation create resources

Step 5: Once Done. Check outputs

There is a lot of useful information here, e.g: the PublicDNS of the two EC2 Instances that got created and the links to the Public API Gateway we created that allows us to execute some Lambda functions.

Lets explore what has been created

Several things have been created.

2 EC2 Instances: Automatically monitored with Dynatrace

The stack created two EC2 Instances. One for Staging, one that we use for Production. Go to your EC2 Section in the AWS Console and explore them:

These EC2 Instances launched with a special launch script that was passed in the UserData section of the EC2 Launch Configuration. This script installed requirement components such as the AWS CodeDeployAgent, Node.js, pm2 as well as the Dynatrace OneAgent. Here is a simplified version of that UserData script:

#1: Install Dynatrace OneAgent
yum update -y
yum install ruby wget -y
wget -O Dynatrace-OneAgent-Linux.sh "{{DynatraceOneAgentURL}}
sudo /bin/sh Dynatrace-OneAgent-Linux.sh APP_LOG_CONTENT_ACCESS=1

#2: Install Depending Software, e.g: httpd, nodejs, pm2
...

#3: Launch dummy PM2 App to pass Env Variables to OneAgent -> Trick to get Vars into Web UI
export DT_CUSTOM_PROP=DEPLOYMENT_GROUP_NAME=GROUP_NAME APPLICATION_NAME=APP_NAME
echo "console.log('dummy app run');" >> testapp.js
pm2 start testapp.js &> pm2start.log

#4: Install AWS CodeDeploy Agent and run CFN Update
...

Lets move to Dynatrace and validate that these two EC2 machines are actually monitored. In your Dynatrace Web UI simply go to Hosts - you should see 2 new hosts. If you have the [Dynatrace AWS CloudWatch](https://www.dynatrace.com/support/help/cloud-platforms/amazon-web-services/h