diff --git a/content/amazon-ec2-spot-cicd-workshop/_index.md b/content/amazon-ec2-spot-cicd-workshop/_index.md index e3b47bc5..2c5bccd1 100644 --- a/content/amazon-ec2-spot-cicd-workshop/_index.md +++ b/content/amazon-ec2-spot-cicd-workshop/_index.md @@ -1,33 +1,22 @@ --- -title: "CI/CD and Test Workloads with EC2 Spot Instances" -menuTitle: "CI/CD and Test Workloads" +title: "CI/CD Workloads with EC2 Spot Instances" +menuTitle: "CI/CD Workloads" date: 2019-02-19T02:02:35 weight: 80 pre: "8. " --- ## Overview -During this workshop, you'll get hands-on with Amazon EC2 Spot and discover architectural best practices through the lens of DevOps and CI/CD. You'll deploy Jenkins build agents and test environments on Spot instances at a fraction of the cost of on-demand instances. You'll also implement mechanisms to ensure that your CI/CD tooling recovers from spot market events by decoupling application state from your compute resources. Finally, you'll migrate your CI/CD environment to a containered environment to eke out maximum performance and cost efficiency. In addition to covering the ins and outs of Spot, we'll share some of the Spot-based mechanisms used by customers to reduce the cost of their test and production workloads. +In this workshop, you'll get hands-on with Spot instances and discover architectural best practices through the lens of DevOps and CI/CD. We'll dive dive deep on how to deploy tools like Jenkins and use Spot instances as build agents. You'll also implement mechanisms to ensure that your CI/CD tooling recovers from Spot interruptions by simulating failures and decoupling application state from your compute resources. Moreover, you'll migrate your CI/CD environment to a containerized environments using ECS to eke out maximum performance and cost efficiency. In addition to covering the best practices to use Spot, we'll share some of the Spot-based mechanisms used by customers to optimize their infrastructure resources. -## Workshop Details -This workshop will be broken down into a series of labs that flow on from each other (that is, you must complete each lab in order before proceeding with the next). The lab exercises that will be covered are: +## Workshop Labs +This workshop will be broken down into a series of labs using differenct CI/CD tools and AWS services, topics covered are: -* Workshop preparation: Deploy pre-requisite resources through Amazon CloudFormation; -* Lab 1: Reduce the cost of builds using Amazon EC2 Spot Fleet; -* Lab 2: Deploy testing environments using Amazon EC2 Spot, Amazon CloudFormation & Amazon EC2 Launch Templates; -* Lab 3: Externalize state data to add resiliency and reduce cost for your CI/CD tooling; -* Lab 4: Using containers backed by Auto Scaling Groups comprised of both on-demand and Spot instances; -* Workshop clean up. +* [Jenkins with Auto Scaling groups](/amazon-ec2-spot-cicd-workshop/jenkins-asg.html) +* [Jenkins with ECS](/amazon-ec2-spot-cicd-workshop/jenkins-ecs.html) +{{% notice note %}} As a reminder, you should have a laptop device (Windows/OSX/Linux are supported - tablets are not appropriate) with the current version of Google Chrome or Mozilla Firefox installed. You should also have a clean AWS account, with **AdministratorAccess** policy-level access. +{{% /notice %}} This workshop should take between two and three hours to complete, depending on your proficiency with the AWS services being featured. - -#### Additional considerations when running this workshop in a corporate IT environment -If you are running this workshop from a corporate IT environment, contact your Systems Administrator to ensure that you will be able to establish outbound Secure Shell (SSH) connections to an Internet host: - -* If you cannot establish SSH connections to Internet hosts (and do not have a suitable workaround), you will not be able to complete Labs 3 & 4; -* If you can establish SSH connections to Internet hosts, obtain from your Systems Administrator the source IP address CIDR block that connections will be established from. - -If you access the Internet through a transparent proxy server running in your corporate IT environment and this proxy server uses a different source address than where SSH connections come from, additional configuration of AWS Security Groups will need to be carried out. The lab guide will indicate the configuration steps required when appropriate. - diff --git a/content/amazon-ec2-spot-cicd-workshop/clea.md b/content/amazon-ec2-spot-cicd-workshop/clea.md deleted file mode 100644 index 0f41dc5c..00000000 --- a/content/amazon-ec2-spot-cicd-workshop/clea.md +++ /dev/null @@ -1,83 +0,0 @@ -+++ -title = "Workshop Cleanup" -weight = 60 -+++ -Congratulations, you have completed this workshop! Your next challenge is to remove all of the resources that were provisioned in your account so as to ensure that no additional cost can be incurred. Please note that the steps below should be implemented in order - some later steps have dependencies on earlier ones! - -## DELETE ALL OBJECTS WITHIN THE S3 BUCKET CREATED BY CLOUDFORMATION -When you attempt to delete the CloudFormation template that you deployed during the Workshop Preparation, the S3 bucket that it provisioned will not be able to be removed unless it is empty. Delete all objects in this bucket (note, you don't need to remove the bucket itself). - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **S3** console (or [click here](https://s3.console.aws.amazon.com/s3/home?region=us-east-1)); -2. Click on the name of the S3 bucket that you noted in Lab 2 (it should have a name of spotcicdworkshop-deploymentartifactss3bucket-random_chars); -3. Mark the checkbox next to the **gameoflife-web** directory, then click on the **Actions** dropdown and select **Delete**. At the confirmation screen, click on the **Delete** button. -{{% /expand%}} - -## MODIFY THE JENKINS MASTER ECS SERVICE SO THAT 0 TASKS ARE DESIRED -Before the EC2 instances that comprise your ECS cluster can be terminated, you'll need to ensure that no containers are running in the cluster. The containers hosting your build agents are likely already stopped as they only remain online for a few seconds after your build jobs have completed - but the container running Jenkins needs to be stopped. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **ECS** console (or [click here](https://eu-west-1.console.aws.amazon.com/ecs/home?region=eu-west-1#/clusters)) and click on the **SpotCICDWorkshopECSCluster** link; -2. Click on the checkbox of the service with a Service Name that begins with **SpotCICDWorkshop-ECSServiceJenkinsMaster**, then click on the **Update** button; -3. Change the Number of tasks from **1** back to **0**, then click on the **Next Step** button; -4. At the Configure network screen, click on the **Next Step** button. Repeat the same action at the Set Auto Scaling (optional) screen. Finally at the Review screen, click on the **Update Service** button. Once the service has been updated, click on the **View Service** button. -{{% /expand%}} - -## DELETE ALL AUTO SCALING GROUPS; -As was the case with EC2 instances created by Spot Fleet requests, you won't be able to terminate an EC2 instance that belongs to the Auto Scaling Group that you created without the Auto Scaling Group re-launching a new instance to replace the one that you terminated. Therefore, the Auto Scaling Group that you created in Lab 4 must be deleted. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Auto Scaling Groups** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/autoscaling/home?region=eu-west-1#AutoScalingGroups)); -2. Mark the checkbox next to the Auto Scaling Group with the name of **Spot CICD Workshop ECS Auto Scaling Group** and from the **Actions** dropdown, click **Delete**. In the confirmation, select **Yes, Delete**. -{{% /expand%}} - -## CANCEL ALL REMAINING SPOT REQUESTS -The CloudFormation template will not be able to delete subnet and VPC resources if there are EC2 resources still running. Spot fleets are an interesting case though - if you terminate an EC2 instance belonging to a fleet, the fleet will attempt to re-launch it (as shown in Lab 3). Therefore you will need to ensure that all of the Spot Fleets that you created in this workshop have been cancelled. - - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Spot Requests** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2sp/v1/spot/home?region=eu-west-1#)); -2. Select the checkboxes corresponding to the Spot Fleet request that you created during this workshop that still remain (it should be noted that if you followed every step in the Lab guides, all of these Spot Fleets should already have been removed); -3. Click the Actions dropdown at the top of the screen and select the **Cancel spot request** option. At the confirmation dialogue, ensure that the **Terminate instances** checkbox is selected and click on the **Confirm** button. -{{% /expand%}} - -## ENSURE THAT ALL EC2 INSTANCES HAVE BEEN TERMINATED -Having removed all of the Spot Fleets and Auto Scaling groups from the VPC that was created by the CloudFormation template deployed during the Workshop Preparation, you should double-check that all EC2 instances used during the workshop have been terminated. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Instances** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#Instances:sort=instanceId)); - * If there is a running instance with a Name of **Jenkins Master (On-Demand)**, mark the checkbox associated with this instance and from the **Actions** dropdown, select **Instance State** > **Terminate**. At the confirmation dialog, click on the **Yes, Terminate** button; - * If there are running instances with a name of **Jenkins Master (Spot)** or **Jenkins Build Agent**, you still have open Spot Fleet requests - repeat the **Cancel all remaining Spot Requests** section above; - * If there are running instances with a name of **Spot CICD Workshop ECS Instance**, you still have an active Auto Scaling Group - repeat the **Delete all Auto Scaling Groups** section above; - * If there are running instances with a name of **Game of Life Test Instance**, you still have the test environment that you created in Lab 2 running - it's fine for these instances to be running, though you will have to delete the CloudFormation stack associated with this test environment prior to deleting the main stack in the next section. -{{% /expand%}} - -## DELETE ALL CLOUDFORMATION STACKS -Once you've checked that all EC2 instances have been terminated and there are no objects present in the S3 bucket used for deployment artifacts, you can remove the CloudFormation template that you deployed during the Workshop Preparation. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1)); -2. If you see a Stack named **GameOfLife**, mark the checkbox associated with this stack and from the **Actions** dropdown, select the **Delete Stack** option. At the confirmation pop-up, click on the **Yes, Delete** button. Note, you must wait for this stack to be deleted before proceeding on to the next step. -3. Mark the checkbox associated with the stack named **SpotCICDWorkshop** and from the **Actions** dropdown, select the **Delete Stack** option. At the confirmation dialog, click on the **Yes, Delete** button. -{{% /expand%}} - -## DELETE ALL CLOUDWATCH LOG GROUPS RELATED TO THE SPOTCICDWORKSHOP -The Lambda functions used throughout the workshop (notably, to create and tear down the testing environment in Lab 2, and to look up the latest version of various AMIs throughout the workshop) wrote log entries to CloudWatch Logs. These should be removed. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **CloudWatch** console and click on the **Logs** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logs:)); -2. For each of the Log Groups that start with **/aws/lambda/SpotCICDWorkshop**: - 1. Select the radio box associated with the Log Group; - 2. From the **Actions** dropdown, select the **Delete log group** option. From the resulting dialog, click on the **Yes, Delete** button. -{{% /expand%}} - -## REMOVE THE AMAZON EC2 KEYPAIR CREATED DURING THE WORKSHOP PREPARATION -The final resource that needs to be removed was the first one that you created - the EC2 Key Pair that you created prior to launching the CloudFormation stack. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Key Pairs** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#KeyPairs)); -2. Mark the check box associated with the Key Pair named **Spot CICD Workshop Key Pair** and click on the **Delete** button. At the resulting pop-up, confirm this action by clicking on the **Yes** button. -{{% /expand%}} - -## THANK YOU -At this point, we would like to than you for attending this workshop. diff --git a/content/amazon-ec2-spot-cicd-workshop/conclusion.md b/content/amazon-ec2-spot-cicd-workshop/conclusion.md new file mode 100644 index 00000000..f4cd8c5c --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/conclusion.md @@ -0,0 +1,4 @@ ++++ +title = "Conclusion" +weight = 1000 ++++ diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/_index.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/_index.md new file mode 100644 index 00000000..eb40c184 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/_index.md @@ -0,0 +1,18 @@ ++++ +title = "Jenkins with Auto Scaling groups" +weight = 100 ++++ +By default, all job builds are executed on the same instance that Jenkins is running on. This results in a couple of less-than-desirable behaviours: +* When CPU-intensive builds are being executed, there may not be sufficient system resources to display the Jenkins server interface; and +* The Jenkins server is often provisioned with more resources than the server interface requires in order to allow builds to execute. When builds are not being executed, these server resources are essentially going to waste. + +To address these behaviours, Jenkins provides the capability to execute builds on external hosts (called build agents). Further, AWS provides a Jenkins plugin to allow Jenkins to scale out a fleet of EC2 instances in order to execute build jobs on. This lab will focus on implementing EC2 Spot build agents, showcasing what a batch processing workload typically looks like when using Amazon EC2 Spot instances. + +## What we'll be doing? +The purpose of these labs is to help you understand how to configure build agents using Spot instances. We'll focus on configuring a Jenkins plugin so that build agents are launched only when you need them. At the end of these labs, you'll be able to: + +* Learn how to launch Spot instances following our best practices using Auto Scaling groups +* Configure properly the EC2 Fleet Jenkins plugin +* Configure Jenkins build jobs to use Spot instances as agents +* Simulate Spot interruption events to increase fault-tolerance +* Test how resilient and fault-toleran Jenkins could be using Spot instances \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/asg.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/asg.md new file mode 100644 index 00000000..7080f746 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/asg.md @@ -0,0 +1,61 @@ ++++ +title = "Launch an Auto Scaling group" +weight = 110 ++++ +An Auto Scaling group contains a collection of Amazon EC2 Instances that are treated as a logical grouping for the purposes of automatic scaling and management. An Auto Scaling group also enables you to use Amazon EC2 Auto Scaling features such as health check replacements and scaling policies. Both maintaining the number of instances in an Auto Scaling group and automatic scaling are the core functionality of the Amazon EC2 Auto Scaling service. + +Amazon EC2 Auto Scaling helps you ensure that you have the correct number of Amazon EC2 Instances available to handle the load for your application. You can specify the minimum and maximum number of instances, and Amazon EC2 Auto Scaling ensures that your group never goes below or above this size. + +When adopting EC2 Spot Instances, our recommendation is use Auto Scaling groups since it offers a very rich API with benefits like scale-in protection, health checks, lifecycle hooks, rebalance recommendation integration, warm pools and predictive scaling, and many more functionalities that we list below. + +## Launching an Auto Scaling group +For this workshop, you need to create an Auto Scaling group that will be used by the Jenkins plugin to perform your application builds. One of the key Spot best practice is to select multiple instance types. At AWS, instance types comprise varying combinations of CPU, memory, storage, and networking capacity to give you the flexibility to choose the appropriate mix of resources for your applications. However, to make this selection simpler, AWS released **[Attribute-Based Instance Type Selection (ABS)](https://aws.amazon.com/blogs/aws/new-attribute-based-instance-type-selection-for-ec2-auto-scaling-and-ec2-fleet/)** to express workload requirements as a set of instance attributes such as: vCPU, memory, type of processor, etc. ABS translates these requirements and selects all matching instance types that meet the criteria. To select which instance to launch, the Auto Scaling group chose instances based on the allocation strategy configured. For Spot Instances we recommend to use **capacity-optimized**, which select the optimal instances that reduce the frequency of interruptions. ABS does also future-proof the Auto Scaling group configuration: *any new instance type we launch that matches the selected attributes, will be included in the list automatically*. No need to update the Auto Scaling group configuration. + +To launch the Auto Scaling group, first create a *json* configuration file. This file describes a mixed-instance-policy section with a set of overrides that drive **diversification of Spot Instance pools** using ABS. The configuration of the Auto Scaling group does refer to the Launch Template that was created in the previous steps with Cloudformation, so make sure you have the corresponding value in the `LAUNCH_TEMPLATE_ID` environment variable + +Here's the command you need to run to create the configuration file: + +```bash +cat < ~/asg-policy.json +{ + "LaunchTemplate":{ + "LaunchTemplateSpecification":{ + "LaunchTemplateId":"${LAUNCH_TEMPLATE_ID}", + "Version":"\$Latest" + }, + "Overrides":[ + { + "InstanceRequirements": { + "VCpuCount":{"Min": 2, "Max": 8}, + "MemoryMiB":{"Min": 4096} } + } + ] + }, + "InstancesDistribution":{ + "OnDemandBaseCapacity": 0, + "OnDemandPercentageAboveBaseCapacity": 0, + "SpotAllocationStrategy":"capacity-optimized" + } +} +EoF +``` + +Notice that in the override section is where you specify the instance attributes with a range of 2 to 8 vCPUs, and a minimum of 4 GBs of memory. ABS will pick a list of instance types that match the criteria like `m5.large`, `c4.large`, or `c5.large`. Then, based on the **capacity-optimized** allocation strategy, the Auto Scaling group will launch instances from pool with more spare capacity available. + +With Auto Scaling groups you can define what is the balance between Spot vs On-Demand Instances that makes sense for your workload. `OnDemandBaseCapacity` allows you to set an initial capacity of On-Demand Instances to use. After that, any new procured capacity will be a mix of Spot and On-Demand Instances as defined by the `OnDemandPercentageAboveBaseCapacity`. This time, *we've configured the Auto Scaling group to launch only Spot instances*. + +Finally, the configuration above, sets the `SpotAllocationStrategy` to `capacity-optimized`. The `capacity-optimized` allocation strategy **allocates instances from the Spot Instance pools with the optimal capacity for the number of instances that are launching**, making use of real-time capacity data and optimizing the selection of used Spot Instances. You can read about the benefits of using `capcity-optimized` in the blog post [Capacity-Optimized Spot Instance allocation in action at Mobileye and Skyscanner](https://aws.amazon.com/blogs/aws/capacity-optimized-spot-instance-allocation-in-action-at-mobileye-and-skyscanner/). + +Let’s now create the Auto Scaling group. In this case the Auto Scaling group spans across 3 Availability Zones, and sets the min-size to 0 (*to avoid running instances when there's no need*), max-size to 2 and desired-capacity to 0. You'll override some of this configuration later through Jenkins. + +{{% notice note %}} +We’re initialising the Auto Scaling group with zero instances to reduce costs. However, the impact is that you need to wait around five minutes to launch a new job while the new instance starts. If you want to launch pending jobs faster, you need set up number of minimum number of instances to the ones you'll need as a baseline for capacity of Jenkins agents. +{{% /notice %}} + +Run the following command: + +```bash +aws autoscaling create-auto-scaling-group --auto-scaling-group-name EC2SpotJenkinsASG --min-size 0 --max-size 2 --desired-capacity 0 --vpc-zone-identifier "${PRIVATE_SUBNETS}" --mixed-instances-policy file://asg-policy.json +``` + +Nothing else to do here, it's time to configure Jenkins to use this Auto Scaling group. \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/clean.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/clean.md new file mode 100644 index 00000000..c8d78338 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/clean.md @@ -0,0 +1,26 @@ ++++ +title = "Cleanup" +weight = 160 ++++ +Congratulations, you have completed the Jenkins with Auto Scaling group lab! Your next challenge is to remove all of the resources that were provisioned in your account so as to ensure that no additional cost can be incurred. Please note that the commands below should be executed in order - some later steps have dependencies on earlier ones! + +## Delete all Auto Scaling groups + +```bash +aws autoscaling delete-auto-scaling-group --auto-scaling-group-name EC2SpotJenkinsASG; +aws autoscaling delete-auto-scaling-group --auto-scaling-group-name EC2SpotJenkinsHostASG; +``` + +## Delete the CloudFormation stacks + +```bash +aws cloudformation delete-stack --stack-name SpotCICDWorkshop; +aws cloudformation delete-stack --stack-name track-spot-interruption; +aws cloudformation delete-stack --stack-name fis-spot-interruption; +``` + +## Remove theh EC2 key pair +The final resource that needs to be removed was the first one that you created - the EC2 Key Pair that you created prior to launching the CloudFormation stack. + +1. Go to the **EC2** console and click on the **Key Pairs** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#KeyPairs)); +2. Mark the check box associated with the Key Pair named **Spot CICD Workshop Key Pair** and click on the **Delete** button. At the resulting pop-up, confirm this action by clicking on the **Yes** button. diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/ec2-fleet-jenkins.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/ec2-fleet-jenkins.md new file mode 100644 index 00000000..9e0eb3e1 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/ec2-fleet-jenkins.md @@ -0,0 +1,44 @@ ++++ +title = "Configure EC2 Fleet plugin" +weight = 115 ++++ +The [EC2 Fleet Plugin](https://plugins.jenkins.io/ec2-fleet/) launches EC2 Spot or On Demand instances as worker nodes for Jenkins CI server, automatically scaling the capacity with the load. The EC2 FLeet plugin will request EC2 instances when excess jobs are detected. You can configure the plugin to use an Auto Scaling Group to launch instances instead of directly launching them by itself. This gives theh plugin all the benefits from Auto Scaling groups like allocation strategies, configure multiple instance types and availability zones, etc. Moreover, the EC2 Fleet plugin can automatically resubmit failed jobs caused by a Spot interruption. + +To start using this plugin, you need to configure it in Jenkins, so let's do it. + +## Sign-in to Jenkins +The CloudFormation template deployed during the Workshop Preparation stage deployed a Jenkins server on to an on-demand instance within your VPC and configured an Application Load Balancer (ALB) to proxy requests from the public Internet to the server. You can obtain the DNS name for the ALB from the Output tab of your CloudFormation template. Point your web browser to this DNS name and sign in using **admin** as the Username and the password that you supplied to the CloudFormation template as the password. + +1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1)); +2. Click on the checkbox associated with the **SpotCICDWorkshop** stack, then click on the **Outputs** tab toward the bottom of the screen; +3. Make a note of the DNS name for the Application Load Balancer, which is associated with the **JenkinsDNSName** key; +4. Open up a new tab in your browser and enter the DNS name in the address bar. You should be greeted with a Jenkins Sign In screen: + 1. Enter in **admin** as the Username; + 2. Enter in the password that you supplied to the CloudFormation template as the Password. + +## Configure the EC2 Fleet Jenkins plugin +The EC2 Fleet Jenkins Plugin was installed on the Jenkins server during the CloudFormation deployment - but now the plugin needs to be configured. You'll need to get the plugin to **Launch slave agents via SSH** and provide valid SSH credentials (don't forget to consider how Host Key Verification should be set when using Spot instances). + +When configuring the plugin, think about how you could force build processes to run on the spot instances (use the **spot-agents** label), and consider how you can verify that the fleet scales out when there is a backlog of build jobs waiting to be processed. + +1. From the Jenkins home screen, click on the **Manage Jenkins** link on the left side menu, and then the **Manage Nodes and Clouds** link; +2. Click on the **Configure Clouds** link on the left side menu, then click on the **Add a new cloud** dropdown, followed by the **Amazon EC2 Fleet** option; +3. **You don't need to configure any AWS Credentials** as the plugin will use the IAM Role attached to the instance; +4. Select **eu-west-1 EU (Ireland)** from the Region dropdown - the plugin will now attempt to obtain a list of Auto Scaling groups in the selected region; +6. Select the Auto Scaling group that you created earlier (`Auto Scaling Group - EC2SpotJenkinsASG`) from the **EC2 Fleet** dropdown (though it might already be selected) and then select the **Launch slave agents via SSH** option from the Launcher dropdown - this should reveal additional SSH authentication settings; +7. Click the **Add** button next to the Credentials dropdown and select the **Jenkins** option. This will pop up another **Jenkins Credentials Provider: Jenkins** sub-form. Fill out the form as follows: + 1. Change the Kind to **SSH Username with private key**; + 2. Change the Scope to **System (Jenkins and nodes only)** – you also don’t want your builds to have access to these credentials; + 3. At the Username field, enter **ec2-user**; + 4. For the Private Key, select the **Enter directly** radio button. Open the .pem file that you downloaded during the workshop setup in a text editor and copy the contents of the file to the Key field including the *BEGIN RSA PRIVATE KEY* and *END RSA PRIVATE KEY* fields; + 5. Click on the **Add** button. +8. Select the **ec2-user** option from the Credentials dropdown; +9. Given that Spot instances will have a random SSH host fingerprint, select the **Non verifying Verification Strategy** option from the Host Key Verification Strategy dropdown; +10. Mark the **Private IP** checkbox to ensure that your Jenkins Master will always communicate with the Agents via their internal VPC IP addresses (in real-world scenarios, your build agents would likely not be publicly addressable); +11. Change the Label field to be **spot-agents** - you'll shortly configure a build job to run on slave instances featuring this label; +12. Set the **Max Idle Minutes Before Scaledown** to **5**. There's no need to keep a build agent running for too much longer than it's required; +13. Change the Minimum Cluster Size from **1** to **0** (so that it can scale-in to zero instances); +14. Change the Maximum Cluster Size from **1** to **3** (so that you can test fleet scale-out); +15. Finally, click on the **Save** button. + +For now, no instances are going to be launched as there are no pending jobs to run. So, let's configure an existing Jenkins job to use Spot instances. \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/prep.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/prep.md similarity index 66% rename from content/amazon-ec2-spot-cicd-workshop/prep.md rename to content/amazon-ec2-spot-cicd-workshop/jenkins-asg/prep.md index f43b660c..c149f708 100644 --- a/content/amazon-ec2-spot-cicd-workshop/prep.md +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/prep.md @@ -1,12 +1,12 @@ +++ -title = "Workshop Preparation" +title = "Setup with CloudFormation" weight = 10 +++ Before we start presenting content on the different use cases for EC2 Spot instances through the lens of CI/CD workloads, you'll need to prepare the AWS account that you've come to this workshop with. Specifically, this workshop involves working with a large number of AWS resources as well as a deployment of Jenkins that if manually configured, would leave you little time to discover how to use EC2 Spot instances in the most effective manner. To address that, you will deploy an Amazon CloudFormation template that does a lot of the heavy lifting of provisioning these resources for you. -## CREATE A NEW EC2 KEY PAIR +## Create a new EC2 Key Pair You will need to access the SSH interfaces of some Linux EC2 instances created in this workshop. To do so in a secure manner, please create a new EC2 key pair with a name of **Spot CICD Workshop Key Pair** in the **EU (Ireland)** region (all activities for this workshop will be carried out in this region). -{{%expand "Click to reveal detailed instructions" %}} + 1. Log in to your AWS Account; 2. Switch to the **EU (Ireland)** region; 3. Provision a new EC2 Key Pair: @@ -14,31 +14,45 @@ You will need to access the SSH interfaces of some Linux EC2 instances created i 2. Click on the **Create Key Pair** button; 3. Enter **Spot CICD Workshop Key Pair** as the Key pair name and click on the **Create** button; 4. Your web browser should download a .pem file – keep this file as it will be required to access the EC2 instances that you create in this workshop. If you're using a Windows system, convert the .pem file to a puTTY .ppk file. If you're not sure how to do this, instructions are available [here](https://aws.amazon.com/premiumsupport/knowledge-center/convert-pem-file-into-ppk/). -{{% /expand%}} -## LAUNCH THE CLOUDFORMATION TEMPLATE +## Launchh the CloudFormation template So that you can concentrate on the aspects of this workshop that directly relate to Amazon EC2 Spot instances, there is a CloudFormation template that will deploy the base AWS infrastructure needed for all of the labs within the workshop - saving you from having to create things like VPCs, Security Groups, IAM policies and so forth. Download and deploy the CloudFormation template: -[amazon-ec2-spot-cicd-workshop.yaml](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop.yaml) +[amazon-ec2-spot-cicd-workshop-asg.yaml](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-asg.yaml) Be sure to give it a stack name of **SpotCICDWorkshop** and ensure that you supply appropriate parameters when prompted. -{{%expand "Click to reveal detailed instructions" %}} + 1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1)); 2. Click on the **Create Stack** button towards the top of the console; 3. At the Select Template screen, select the **Upload a template file** radio button and choose the CloudFormation template you downloaded before, then click on the **Next** button; 4. At the Specify Details screen, enter in **SpotCICDWorkshop** as the Stack name. Under the Parameters section: - 1. Identify what your current public IP address is by going to https://www.google.com.au/search?q=what%27s+my+ip+address. Enter the first three octets of this IP address into the **CurrentIP** parameter field and then add the **.0/24** suffix. For example if your IP address was 54.240.193.193, you would enter 54.240.193.0/24 into the CurrentIP field[^1]; + 1. Identify what your current public IP address is by going to https://www.google.com.au/search?q=what%27s+my+ip+address. Enter the first three octets of this IP address into the **CurrentIP** parameter field and then add the **.0/24** suffix. For example if your IP address was 54.240.193.193, you would enter 54.240.193.0/24 into the CurrentIP field; 2. Enter in a password that you would like to use for the administrator account within the Jenkins server that will be launched by the CloudFormation template; 3. Select the **Spot CICD Workshop Key Pair** option in the Keypair dropdown. 5. Click on the **Next** button; 6. At the Options screen, there is no need to make any changes to the default options – simply click on the **Next** button; 7. Finally at the Review screen, verify your settings, mark the **I acknowledge that AWS CloudFormation might create IAM resources with custom names** checkbox and then click on the **Create** button. Wait for the stack to complete provisioning, which should take a couple of minutes. -[^1]: It's good security practice to ensure that the web and SSH services being used in this workshop are not accessible to everyone on the Internet. In most cases, limiting access to the /24 CIDR block that you IP address is in provides a reasonable level of access control - but this may still be too restrictive in some corporate IT environments. If you have trouble accessing resources, additional instructions within this lab guide will guide you through what settings need to be manually changed. -{{% /expand%}} - The stack should take around five minutes to deploy. -## PROCEED TO LAB 1 -Once the CloudFormation is in the process of being deployed, you've completed all of the preparation required to start the workshop, you may proceed with [Lab 1](/amazon-ec2-spot-cicd-workshop/lab1.html). +{{% notice note %}} +It's good security practice to ensure that the web and SSH services being used in this workshop are not accessible to everyone on the Internet. In most cases, limiting access to the /24 CIDR block that you IP address is in provides a reasonable level of access control - but this may still be too restrictive in some corporate IT environments. If you have trouble accessing resources, additional instructions within this lab guide will guide you through what settings need to be manually changed. +{{% /notice %}} + +{{% notice note %}} +The CloudFormation template creates a new Launch Template to install and bootstrap Jenkins. However, you can continue using any existing Launch Template that you might already have. +{{% /notice %}} + +## Setting Up environment variables +You need to set up the following environment variables that you'll use in the workshop, to do so, run the following commands: + +```bash +export PRIVATE_SUBNETS=$(aws cloudformation describe-stacks --stack-name SpotCICDWorkshop --query "Stacks[0].Outputs[?OutputKey=='JenkinsVPCPrivateSubnets'].OutputValue" --output text); +export PUBLIC_SUBNETS=$(aws cloudformation describe-stacks --stack-name SpotCICDWorkshop --query "Stacks[0].Outputs[?OutputKey=='JenkinsVPCPublicSubnets'].OutputValue" --output text); +export LAUNCH_TEMPLATE_ID=$(aws ec2 describe-launch-templates --filters Name=launch-template-name,Values=JenkinsBuildAgentLaunchTemplate | jq -r '.LaunchTemplates[0].LaunchTemplateId'); +``` + +{{% notice note %}} +If for some reason the environment variables are cleared, you can run the previous commands again without any problem. +{{% /notice %}} diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/spot_interruption_experiment.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/spot_interruption_experiment.md new file mode 100644 index 00000000..48c393cb --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/spot_interruption_experiment.md @@ -0,0 +1,106 @@ +--- +title: "Creating the Spot Interruption Experiment" +weight: 145 +--- + +In this section, you're going to start creating the experiment to [trigger the interruption of Amazon EC2 Spot Instances using AWS Fault Injection Simulator (FIS)](https://aws.amazon.com/blogs/compute/implementing-interruption-tolerance-in-amazon-ec2-spot-with-aws-fault-injection-simulator/). When using Spot instances, you need to be prepared to be interrupted. With FIS, you can test the resiliency of your workload and validate that your application is reacting to the interruption notices that EC2 sends before terminating your instances. You can target individual Spot instances or a subset of instances from an Auto Scaling group. + +#### What do you need to get started? + +Before you start launching Spot interruptions with FIS, you need to create an experiment template. Here is where you define which resources you want to interrupt (targets), and when you want to interrupt the instance. + +You're going to use the following CloudFormation template which creates the IAM role (`FISSpotRole`) with the minimum permissions FIS needs to interrupt an instance, and the experiment template (`FISExperimentTemplate`) you're going to use to trigger a Spot interruption: + +``` +--- +AWSTemplateFormatVersion: 2010-09-09 +Description: FIS for Spot Instances +Parameters: + InstancesToInterrupt: + Description: Number of instances to interrupt + Default: 1 + Type: Number + + DurationBeforeInterruption: + Description: Number of minutes before the interruption + Default: 2 + Type: Number + +Resources: + FISSpotRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [fis.amazonaws.com] + Action: ["sts:AssumeRole"] + Path: / + Policies: + - PolicyName: root + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "ec2:DescribeInstances" + Resource: "*" + - Effect: Allow + Action: "ec2:SendSpotInstanceInterruptions" + Resource: "arn:aws:ec2:*:*:instance/*" + + FISExperimentTemplate: + Type: AWS::FIS::ExperimentTemplate + Properties: + Description: "Interrupt multiple random instances" + Targets: + SpotIntances: + ResourceTags: + Name: "jenkins-build-agent" + Filters: + - Path: State.Name + Values: + - running + ResourceType: aws:ec2:spot-instance + SelectionMode: !Join ["", ["COUNT(", !Ref InstancesToInterrupt, ")"]] + Actions: + interrupt: + ActionId: "aws:ec2:send-spot-instance-interruptions" + Description: "Interrupt multiple Spot instances" + Parameters: + durationBeforeInterruption: + !Join ["", ["PT", !Ref DurationBeforeInterruption, "M"]] + Targets: + SpotInstances: SpotIntances + StopConditions: + - Source: none + RoleArn: !GetAtt FISSpotRole.Arn + Tags: + Name: "fis-spot-interruption" + +Outputs: + FISExperimentID: + Value: !GetAtt FISExperimentTemplate.Id +``` + +Here are some important notes about the template: + +* You can configure how many instances you want to interrupt with the `InstancesToInterrupt` parameter. In the template it's defined that it's going to interrupt **three** instances. +* You can also configure how much time you want the experiment to run with the `DurationBeforeInterruption` parameter. By default, it's going to take two minutes. This means that as soon as you launch the experiment, the instance is going to receive the two-minute notification Spot interruption warning. +* The most important section is the `Targets` from the experiment template. This is where you tell FIS which instances to interrupt. In this case, we're simply going to say that we want to interrupt a instance that's `running` and has a `Name` tag with value `Jenkins Master (Spot)`. +* Notice that instances are going to be **chosen randomly** + +#### Create the EC2 Spot Interruption Experiment with FIS + +Let's continue by creating the Spot interruption experiment template using Cloudformation. You can view the CloudFormation template (**fisspotinterruption.yaml**) at GitHub [here](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/amazon-ec2-spot-cicd-workshop/fisspotinterruption.yaml). To download it, you can run the following command: + +``` +wget https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/amazon-ec2-spot-cicd-workshop/fisspotinterruption.yaml +``` + +Now, simply run the following commands to create the FIS experiment: + +``` +aws cloudformation create-stack --stack-name fis-spot-jenkins-interruption --template-body file://fisspotinterruption.yaml --capabilities CAPABILITY_NAMED_IAM +aws cloudformation wait stack-create-complete --stack-name fis-spot-jenkins-interruption +``` \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/spot_interruption_fis.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/spot_interruption_fis.md new file mode 100644 index 00000000..214ff69a --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/spot_interruption_fis.md @@ -0,0 +1,52 @@ +--- +title: "Interrupting a Spot Instance" +weight: 150 +--- + +In this section, you're going to launch a Spot Interruption using FIS and then verify that the capacity has been replenished and Jenkins is still working. This will help you to confirm the low impact of your workloads when implementing Spot effectively. + +#### Configure a retry build in Jenkins jobs +Configure it (thanks to Naginator) and schedule several jobs. + +#### Launch the Spot interruption experiment +After creating the experiment template in FIS, you can start a new experiment to interrupt one Spot instance. Run the following command: + +``` +FIS_EXP_TEMP_ID=$(aws cloudformation describe-stacks --stack-name fis-spot-jenkins-interruption --query "Stacks[0].Outputs[?OutputKey=='FISExperimentID'].OutputValue" --output text); +FIS_EXP_ID=$(aws fis start-experiment --experiment-template-id $FIS_EXP_TEMP_ID --no-cli-pager --query "experiment.id" --output text); +``` + +Wait around 30 seconds, and you should see that the experiment completes. Run the following command to confirm: + +``` +aws fis get-experiment --id $FIS_EXP_ID --no-cli-pager +``` + +At this point, FIS has triggered a Spot interruption notice, and in two minutes the instance will be terminated. + +Go to CloudWatch Logs group `/aws/events/spotinterruptions` to confirm that instance received the termination notice. + +You should see a log message like this one: + +![SpotInterruptionLog](/images/amazon-ec2-spot-cicd-workshop/spotinterruptionlog.png) + +#### Verify the actions taken by Jenkins Auto Scaling group +You are running a Jenkins instance launched by an Auto Scaling group that will launch a new instance if the desired capacity is not compliant. + +You can run the following command to see the list of instances with the date and time when they were launched. + +``` +aws ec2 describe-instances --filters\ + Name='tag:Name',Values='jenkins-build-agent'\ + Name='instance-state-name',Values='running'\ + | jq '.Reservations[].Instances[] | "Instance with ID:\(.InstanceId) launched at \(.LaunchTime)"' +``` + +You should see a list of instances with the date and time when they were launched. You'll also see the new instance launched after the interruption. + +```output +"Instance with ID:i-04c6ced30794e965b launched at 2022-09-06T14:02:49+00:00" +"Instance with ID:i-0136152e14053af81 launched at 2022-09-06T14:11:25+00:00" +``` + +#### Verify that Jenkins jobs were retried diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/test-build.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/test-build.md new file mode 100644 index 00000000..8df4dae5 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/test-build.md @@ -0,0 +1,22 @@ ++++ +title = "Configure jobs to use Spot" +weight = 120 ++++ +As alluded to in the previous section, you'll need to configure your build jobs so that they are executed on the build agents running in your Spot instances. In addition, you could configure jobs to execute concurrent builds if necessary - this will help you in testing the scale-out of your fleet. + +1. Go back to the Jenkins home screen and **repeat the following for each of the five Apache build projects** that are configured in your Jenkins instance: + 1. Click on the title of the build job and then click on the **Configure** link toward the left side of the screen; + 2. In the General section, click on the **Execute concurrent builds if necessary** checkbox and the **Restrict where this project can be run** checkbox. Next, enter **spot-agents** as the Label Expression (Note: if you select the auto-complete option instead of typing out the full label, Jenkins will add a space to the end of the label - be sure to remove any trailing spaces from the label before proceeding); + 3. Click on the **Save** button towards the bottom of the screen. + +## Test Spot Builds and Scale-out +Now it’s time to test out how Jenkins handles pushing builds to spot instances running build agents at scale. There are two things that you'll want to verify here; that your builds run successfully on the Spot instances, and that your ASG scales out when there are build jobs queued for more than a few minutes. + +1. Go Back to the Jenkins home page, click on the **Schedule a Build** icon (which looks like a play symbol) for each of the five Apache projects, starting from the **Apache PDFBox** project and working upward. This will queue up five build jobs, this will trigger the EC2 Fleet plugin to start launching build agents using Spot instances; +2. When any of the build jobs have been completed, click on the **Schedule a Build** icon corresponding to that job to re-add it back to the build queue - the intent here is to keep the build queue populated with a backlog of build jobs until your Auto Scaling group has scaled out and build jobs are executing on Spot instances; +3. After around four minutes, the **EC2 Fleet Status** reported to the left of the screen will increment the **target** count to 5, indicating that the plugin has requested a scale-out action from the plugin. After a few moments, five build instances will appear in the **Build Executor Status**, though these build agents will initially appear to be offline. Once the instances have had the chance to complete the launch and bootstrapping processes (which takes around two minutes), your Jenkins Master will deploy the build agent to them via SSH, and it will come online and process the next build jobs in the queue. Once you have concurrent builds being executed on the Spot instances, you can stop adding build jobs to the build queue; +4. After a period of around a five minutes, after your builds have completed, the Spot instances should be terminated by the plugin. + +{{% notice note %}} +If the build agents are keep showing as `offline`, make sure that you've configured correctly the SSH key pair in the credentials section when configuring the EC2 Fleet plugin. You can click on the agent to open the logs and see why it's not coming online. +{{% /notice %}} \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/tracking_spot_interruptions.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/tracking_spot_interruptions.md new file mode 100644 index 00000000..ab3ed2a6 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-asg/tracking_spot_interruptions.md @@ -0,0 +1,36 @@ +--- +title: "Tracking Spot interruptions" +weight: 140 +--- +At some point in time, you might receive a Spot interruption notice when On-Demand needs the capacity back. When this happens, your Spot instance will be provided with a two-minute notice of termination and after that time lapses, the instance will be terminated. At this point, the Auto Scaling group will observe that there are no running instances in the group and because the desired capacity is one. Then, it will launch a replacement instance from the pool with more capacity available (this is why diversification across many capacity pool is a best practice). The new instance will be launched and bootstrapped in exactly the same manner as your original instance. + +Before we start testing the resiliency of Jenkins with Spot, let's configure CloudWatch Logs to log Spot interruptions. If a Jenkins job fails, we'll be able to check if the failure correlates to a Spot interruption or not. + +{{% notice note %}} +In most cases we don't really need to track the Spot interruptions as Jenkins jobs can be retried if it fails. However, when we're starting with running our Jenkins jobs on Spot instances tracking could be useful. Organizations can use this data to correlate possible job failures or prolonged execution times, in case Spot instances were interrupted during a job execution. +{{% /notice %}} + +#### Creating the CloudFormation Stack to Track EC2 Spot Interruptions + +We've created a CloudFormation template that includes all the resources you need to track EC2 Spot Interruptions. The stack creates the following: + +* An Event Rule for tracking EC2 Spot Interruption Warnings +* A CloudWatch Log group to log interruptions and instance details +* IAM Role to allow the event rule to log into CloudWatch Logs + +You can view the CloudFormation template (**cloudwatchlogs.yaml**) at GitHub [here](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/track-spot-interruptions/cloudwatchlogs.yaml). To download it, you can run the following command: + +``` +wget https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/track-spot-interruptions/cloudwatchlogs.yaml +``` + +After downloading the CloudFormation template, run the following command in a terminal: + +``` +aws cloudformation create-stack --stack-name track-spot-interruption --template-body file://cloudwatchlogs.yaml --capabilities CAPABILITY_NAMED_IAM +aws cloudformation wait stack-create-complete --stack-name track-spot-interruption +``` + +You should see an event rule in the Amazon EventBridge console, like this: + +![Spot Interruption Event Rule](/images/tracking-spot/itn-event-rule.png) diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/_index.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/_index.md new file mode 100644 index 00000000..e4215a89 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/_index.md @@ -0,0 +1,10 @@ ++++ +title = "Jenkins on ECS" +weight = 200 ++++ +You’ve now got a scalable solution using nothing but Spot instances for your CICD systems, your build agents and your test environments – however, you still have some inefficiencies with this setup: + +* Your Jenkins master utilizes a relatively low percentage of the CPU resources on the instance types that Jenkins is running on; and +* You still have at least one Jenkins build agent running at all times; + +These inefficiencies can be addressed by moving your solution to a container environment that continues to utilize Spot instances. This lab will see you configure the ECS cluster resources that were created by the initial CloudFormation template and migrate your Jenkins master and agents to this cluster. \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/asg.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/asg.md new file mode 100644 index 00000000..7080f746 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/asg.md @@ -0,0 +1,61 @@ ++++ +title = "Launch an Auto Scaling group" +weight = 110 ++++ +An Auto Scaling group contains a collection of Amazon EC2 Instances that are treated as a logical grouping for the purposes of automatic scaling and management. An Auto Scaling group also enables you to use Amazon EC2 Auto Scaling features such as health check replacements and scaling policies. Both maintaining the number of instances in an Auto Scaling group and automatic scaling are the core functionality of the Amazon EC2 Auto Scaling service. + +Amazon EC2 Auto Scaling helps you ensure that you have the correct number of Amazon EC2 Instances available to handle the load for your application. You can specify the minimum and maximum number of instances, and Amazon EC2 Auto Scaling ensures that your group never goes below or above this size. + +When adopting EC2 Spot Instances, our recommendation is use Auto Scaling groups since it offers a very rich API with benefits like scale-in protection, health checks, lifecycle hooks, rebalance recommendation integration, warm pools and predictive scaling, and many more functionalities that we list below. + +## Launching an Auto Scaling group +For this workshop, you need to create an Auto Scaling group that will be used by the Jenkins plugin to perform your application builds. One of the key Spot best practice is to select multiple instance types. At AWS, instance types comprise varying combinations of CPU, memory, storage, and networking capacity to give you the flexibility to choose the appropriate mix of resources for your applications. However, to make this selection simpler, AWS released **[Attribute-Based Instance Type Selection (ABS)](https://aws.amazon.com/blogs/aws/new-attribute-based-instance-type-selection-for-ec2-auto-scaling-and-ec2-fleet/)** to express workload requirements as a set of instance attributes such as: vCPU, memory, type of processor, etc. ABS translates these requirements and selects all matching instance types that meet the criteria. To select which instance to launch, the Auto Scaling group chose instances based on the allocation strategy configured. For Spot Instances we recommend to use **capacity-optimized**, which select the optimal instances that reduce the frequency of interruptions. ABS does also future-proof the Auto Scaling group configuration: *any new instance type we launch that matches the selected attributes, will be included in the list automatically*. No need to update the Auto Scaling group configuration. + +To launch the Auto Scaling group, first create a *json* configuration file. This file describes a mixed-instance-policy section with a set of overrides that drive **diversification of Spot Instance pools** using ABS. The configuration of the Auto Scaling group does refer to the Launch Template that was created in the previous steps with Cloudformation, so make sure you have the corresponding value in the `LAUNCH_TEMPLATE_ID` environment variable + +Here's the command you need to run to create the configuration file: + +```bash +cat < ~/asg-policy.json +{ + "LaunchTemplate":{ + "LaunchTemplateSpecification":{ + "LaunchTemplateId":"${LAUNCH_TEMPLATE_ID}", + "Version":"\$Latest" + }, + "Overrides":[ + { + "InstanceRequirements": { + "VCpuCount":{"Min": 2, "Max": 8}, + "MemoryMiB":{"Min": 4096} } + } + ] + }, + "InstancesDistribution":{ + "OnDemandBaseCapacity": 0, + "OnDemandPercentageAboveBaseCapacity": 0, + "SpotAllocationStrategy":"capacity-optimized" + } +} +EoF +``` + +Notice that in the override section is where you specify the instance attributes with a range of 2 to 8 vCPUs, and a minimum of 4 GBs of memory. ABS will pick a list of instance types that match the criteria like `m5.large`, `c4.large`, or `c5.large`. Then, based on the **capacity-optimized** allocation strategy, the Auto Scaling group will launch instances from pool with more spare capacity available. + +With Auto Scaling groups you can define what is the balance between Spot vs On-Demand Instances that makes sense for your workload. `OnDemandBaseCapacity` allows you to set an initial capacity of On-Demand Instances to use. After that, any new procured capacity will be a mix of Spot and On-Demand Instances as defined by the `OnDemandPercentageAboveBaseCapacity`. This time, *we've configured the Auto Scaling group to launch only Spot instances*. + +Finally, the configuration above, sets the `SpotAllocationStrategy` to `capacity-optimized`. The `capacity-optimized` allocation strategy **allocates instances from the Spot Instance pools with the optimal capacity for the number of instances that are launching**, making use of real-time capacity data and optimizing the selection of used Spot Instances. You can read about the benefits of using `capcity-optimized` in the blog post [Capacity-Optimized Spot Instance allocation in action at Mobileye and Skyscanner](https://aws.amazon.com/blogs/aws/capacity-optimized-spot-instance-allocation-in-action-at-mobileye-and-skyscanner/). + +Let’s now create the Auto Scaling group. In this case the Auto Scaling group spans across 3 Availability Zones, and sets the min-size to 0 (*to avoid running instances when there's no need*), max-size to 2 and desired-capacity to 0. You'll override some of this configuration later through Jenkins. + +{{% notice note %}} +We’re initialising the Auto Scaling group with zero instances to reduce costs. However, the impact is that you need to wait around five minutes to launch a new job while the new instance starts. If you want to launch pending jobs faster, you need set up number of minimum number of instances to the ones you'll need as a baseline for capacity of Jenkins agents. +{{% /notice %}} + +Run the following command: + +```bash +aws autoscaling create-auto-scaling-group --auto-scaling-group-name EC2SpotJenkinsASG --min-size 0 --max-size 2 --desired-capacity 0 --vpc-zone-identifier "${PRIVATE_SUBNETS}" --mixed-instances-policy file://asg-policy.json +``` + +Nothing else to do here, it's time to configure Jenkins to use this Auto Scaling group. \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/clean.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/clean.md new file mode 100644 index 00000000..c8d78338 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/clean.md @@ -0,0 +1,26 @@ ++++ +title = "Cleanup" +weight = 160 ++++ +Congratulations, you have completed the Jenkins with Auto Scaling group lab! Your next challenge is to remove all of the resources that were provisioned in your account so as to ensure that no additional cost can be incurred. Please note that the commands below should be executed in order - some later steps have dependencies on earlier ones! + +## Delete all Auto Scaling groups + +```bash +aws autoscaling delete-auto-scaling-group --auto-scaling-group-name EC2SpotJenkinsASG; +aws autoscaling delete-auto-scaling-group --auto-scaling-group-name EC2SpotJenkinsHostASG; +``` + +## Delete the CloudFormation stacks + +```bash +aws cloudformation delete-stack --stack-name SpotCICDWorkshop; +aws cloudformation delete-stack --stack-name track-spot-interruption; +aws cloudformation delete-stack --stack-name fis-spot-interruption; +``` + +## Remove theh EC2 key pair +The final resource that needs to be removed was the first one that you created - the EC2 Key Pair that you created prior to launching the CloudFormation stack. + +1. Go to the **EC2** console and click on the **Key Pairs** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#KeyPairs)); +2. Mark the check box associated with the Key Pair named **Spot CICD Workshop Key Pair** and click on the **Delete** button. At the resulting pop-up, confirm this action by clicking on the **Yes** button. diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/ec2-fleet-jenkins.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/ec2-fleet-jenkins.md new file mode 100644 index 00000000..68a2fd53 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/ec2-fleet-jenkins.md @@ -0,0 +1,44 @@ ++++ +title = "Configure EC2 Fleet plugin" +weight = 115 ++++ +The [EC2 Fleet Plugin](https://plugins.jenkins.io/ec2-fleet/) launches EC2 Spot or On Demand instances as worker nodes for Jenkins CI server, automatically scaling the capacity with the load. The EC2 FLeet plugin will request EC2 instances when excess jobs are detected. You can configure the plugin to use an Auto Scaling Group to launch instances instead of directly launching them by itself. This gives theh plugin all the benefits from Auto Scaling groups like allocation strategies, configure multiple instance types and availability zones, etc. Moreover, the EC2 Fleet plugin can automatically resubmit failed jobs caused by a Spot interruption. + +To start using this plugin, you need to configure it in Jenkins, so let's do it. + +## Sign-in to Jenkins +The CloudFormation template deployed during the Workshop Preparation stage deployed a Jenkins server on to an on-demand instance within your VPC and configured an Application Load Balancer (ALB) to proxy requests from the public Internet to the server. You can obtain the DNS name for the ALB from the Output tab of your CloudFormation template. Point your web browser to this DNS name and sign in using **admin** as the Username and the password that you supplied to the CloudFormation template as the password. + +1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1)); +2. Click on the checkbox associated with the **SpotCICDWorkshop** stack, then click on the **Outputs** tab toward the bottom of the screen; +3. Make a note of the DNS name for the Application Load Balancer, which is associated with the **JenkinsDNSName** key; +4. Open up a new tab in your browser and enter the DNS name in the address bar. You should be greeted with a Jenkins Sign In screen: + 1. Enter in **admin** as the Username; + 2. Enter in the password that you supplied to the CloudFormation template as the Password. + +## Configure the EC2 Fleet Jenkins plugin +The EC2 Fleet Jenkins Plugin was installed on the Jenkins server during the CloudFormation deployment - but now the plugin needs to be configured. You'll need to get the plugin to **Launch slave agents via SSH** and provide valid SSH credentials (don't forget to consider how Host Key Verification should be set when using Spot instances). + +When configuring the plugin, think about how you could force build processes to run on the spot instances (use the **spot-agents** label), and consider how you can verify that the fleet scales out when there is a backlog of build jobs waiting to be processed. + +1. From the Jenkins home screen, click on the **Manage Jenkins** link on the left side menu, and then the **Manage Nodes and Clouds** link; +2. Click on the **Configure Clouds** link on the left side menu, then click on the **Add a new cloud** dropdown, followed by the **Amazon EC2 Fleet** option; +3. **You don't need to configure any AWS Credentials** as the plugin will use the IAM Role attached to the instance; +4. Select **eu-west-1 EU (Ireland)** from the Region dropdown - the plugin will now attempt to obtain a list of Auto Scaling groups in the selected region; +6. Select the Auto Scaling group that you created earlier (`Auto Scaling Group - EC2SpotJenkinsASG`) from the **EC2 Fleet** dropdown (though it might already be selected) and then select the **Launch slave agents via SSH** option from the Launcher dropdown - this should reveal additional SSH authentication settings; +7. Click the **Add** button next to the Credentials dropdown and select the **Jenkins** option. This will pop up another **Jenkins Credentials Provider: Jenkins** sub-form. Fill out the form as follows: + 1. Change the Kind to **SSH Username with private key**; + 2. Change the Scope to **System (Jenkins and nodes only)** – you also don’t want your builds to have access to these credentials; + 3. At the Username field, enter **ec2-user**; + 4. For the Private Key, select the **Enter directly** radio button. Open the .pem file that you downloaded during the workshop setup in a text editor and copy the contents of the file to the Key field including the *BEGIN RSA PRIVATE KEY* and *END RSA PRIVATE KEY* fields; + 5. Click on the **Add** button. +8. Select the **ec2-user** option from the Credentials dropdown; +9. Given that Spot instances will have a random SSH host fingerprint, select the **Non verifying Verification Strategy** option from the Host Key Verification Strategy dropdown; +10. Mark the **Private IP** checkbox to ensure that your Jenkins Master will always communicate with the Agents via their internal VPC IP addresses (in real-world scenarios, your build agents would likely not be publicly addressable); +11. Change the Label field to be **spot-agents** - you'll shortly configure a build job to run on slave instances featuring this label; +12. Set the **Max Idle Minutes Before Scaledown** to **5**. There's no need to keep a build agent running for too much longer than it's required; +13. Change the Minimum Cluster Size from **1** to **0** (so that it can scale-in to zero instances); +14. Change the Maximum Cluster Size from **1** to **5** (so that you can test fleet scale-out); +15. Finally, click on the **Save** button. + +For now, no instances are going to be launched as there are no pending jobs to run. So, let's configure an existing Jenkins job to use Spot instances. \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/prep.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/prep.md new file mode 100644 index 00000000..4b8b55c2 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/prep.md @@ -0,0 +1,46 @@ ++++ +title = "Setup with CloudFormation" +weight = 10 ++++ +Before we start presenting content on how to configure Jenkins with EC2 Spot instances in ECS, you'll need to prepare the AWS account that you've come to this workshop with. Specifically, this workshop involves working with a large number of AWS resources as well as a deployment of Jenkins that if manually configured, would leave you little time to discover how to use EC2 Spot instances in the most effective manner. To address that, you will deploy an Amazon CloudFormation template that does a lot of the heavy lifting of provisioning these resources for you. + +## Launchh the CloudFormation template +So that you can concentrate on the aspects of this workshop that directly relate to Amazon EC2 Spot instances, there is a CloudFormation template that will deploy the base AWS infrastructure needed for all of the labs within the workshop - saving you from having to create things like ECS Cluster, VPCs, Security Groups, IAM policies and so forth. + +Download and deploy the CloudFormation template: +[amazon-ec2-spot-cicd-workshop-asg.yaml](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-ecs.yaml) + +Be sure to give it a stack name of **SpotCICDWorkshopECS** and ensure that you supply appropriate parameters when prompted. + +1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1)); +2. Click on the **Create Stack** button towards the top of the console; +3. At the Select Template screen, select the **Upload a template file** radio button and choose the CloudFormation template you downloaded before, then click on the **Next** button; +4. At the Specify Details screen, enter in **SpotCICDWorkshopECS** as the Stack name. Under the Parameters section: + 1. Identify what your current public IP address is by going to https://www.google.com.au/search?q=what%27s+my+ip+address. Enter the first three octets of this IP address into the **CurrentIP** parameter field and then add the **.0/24** suffix. For example if your IP address was 54.240.193.193, you would enter 54.240.193.0/24 into the CurrentIP field; + 2. Enter in a password that you would like to use for the administrator account within the Jenkins server that will be launched by the CloudFormation template; +5. Click on the **Next** button; +6. At the Options screen, there is no need to make any changes to the default options – simply click on the **Next** button; +7. Finally at the Review screen, verify your settings, mark the **I acknowledge that AWS CloudFormation might create IAM resources with custom names** checkbox and then click on the **Create** button. Wait for the stack to complete provisioning, which should take a couple of minutes. + +The stack should take around five minutes to deploy. + +{{% notice note %}} +It's good security practice to ensure that the web and SSH services being used in this workshop are not accessible to everyone on the Internet. In most cases, limiting access to the /24 CIDR block that you IP address is in provides a reasonable level of access control - but this may still be too restrictive in some corporate IT environments. If you have trouble accessing resources, additional instructions within this lab guide will guide you through what settings need to be manually changed. +{{% /notice %}} + +{{% notice note %}} +The CloudFormation template creates a new Launch Template to install and bootstrap Jenkins. However, you can continue using any existing Launch Template that you might already have. +{{% /notice %}} + +## Setting Up environment variables +You need to set up the following environment variables that you'll use in the workshop, to do so, run the following commands: + +```bash +export PRIVATE_SUBNETS=$(aws cloudformation describe-stacks --stack-name SpotCICDWorkshop --query "Stacks[0].Outputs[?OutputKey=='JenkinsVPCPrivateSubnets'].OutputValue" --output text); +export PUBLIC_SUBNETS=$(aws cloudformation describe-stacks --stack-name SpotCICDWorkshop --query "Stacks[0].Outputs[?OutputKey=='JenkinsVPCPublicSubnets'].OutputValue" --output text); +export LAUNCH_TEMPLATE_ID=$(aws ec2 describe-launch-templates --filters Name=launch-template-name,Values=JenkinsBuildAgentLaunchTemplate | jq -r '.LaunchTemplates[0].LaunchTemplateId'); +``` + +{{% notice note %}} +If for some reason the environment variables are cleared, you can run the previous commands again without any problem. +{{% /notice %}} diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/spot_interruption_experiment.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/spot_interruption_experiment.md new file mode 100644 index 00000000..4ec06939 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/spot_interruption_experiment.md @@ -0,0 +1,105 @@ +--- +title: "Creating the Spot Interruption Experiment" +weight: 145 +--- + +In this section, you're going to start creating the experiment to [trigger the interruption of Amazon EC2 Spot Instances using AWS Fault Injection Simulator (FIS)](https://aws.amazon.com/blogs/compute/implementing-interruption-tolerance-in-amazon-ec2-spot-with-aws-fault-injection-simulator/). When using Spot instances, you need to be prepared to be interrupted. With FIS, you can test the resiliency of your workload and validate that your application is reacting to the interruption notices that EC2 sends before terminating your instances. You can target individual Spot instances or a subset of instances from an Auto Scaling group. + +#### What do you need to get started? + +Before you start launching Spot interruptions with FIS, you need to create an experiment template. Here is where you define which resources you want to interrupt (targets), and when you want to interrupt the instance. + +You're going to use the following CloudFormation template which creates the IAM role (`FISSpotRole`) with the minimum permissions FIS needs to interrupt an instance, and the experiment template (`FISExperimentTemplate`) you're going to use to trigger a Spot interruption: + +``` +AWSTemplateFormatVersion: 2010-09-09 +Description: FIS for Spot Instances +Parameters: + InstancesToInterrupt: + Description: Number of instances to interrupt + Default: 3 + Type: Number + + DurationBeforeInterruption: + Description: Number of minutes before the interruption + Default: 2 + Type: Number + +Resources: + + FISSpotRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [fis.amazonaws.com] + Action: ["sts:AssumeRole"] + Path: / + Policies: + - PolicyName: root + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: 'ec2:DescribeInstances' + Resource: '*' + - Effect: Allow + Action: 'ec2:SendSpotInstanceInterruptions' + Resource: 'arn:aws:ec2:*:*:instance/*' + + FISExperimentTemplate: + Type: AWS::FIS::ExperimentTemplate + Properties: + Description: "Interrupt multiple random instances" + Targets: + SpotIntances: + ResourceTags: + Name: "Jenkins Master (Spot)" + Filters: + - Path: State.Name + Values: + - running + ResourceType: aws:ec2:spot-instance + SelectionMode: !Join ["", ["COUNT(", !Ref InstancesToInterrupt, ")"]] + Actions: + interrupt: + ActionId: "aws:ec2:send-spot-instance-interruptions" + Description: "Interrupt multiple Spot instances" + Parameters: + durationBeforeInterruption: !Join ["", ["PT", !Ref DurationBeforeInterruption, "M"]] + Targets: + SpotInstances: SpotIntances + StopConditions: + - Source: none + RoleArn: !GetAtt FISSpotRole.Arn + Tags: + Name: "fis-spot-interruption" + +Outputs: + FISExperimentID: + Value: !GetAtt FISExperimentTemplate.Id +``` + +Here are some important notes about the template: + +* You can configure how many instances you want to interrupt with the `InstancesToInterrupt` parameter. In the template it's defined that it's going to interrupt **three** instances. +* You can also configure how much time you want the experiment to run with the `DurationBeforeInterruption` parameter. By default, it's going to take two minutes. This means that as soon as you launch the experiment, the instance is going to receive the two-minute notification Spot interruption warning. +* The most important section is the `Targets` from the experiment template. This is where you tell FIS which instances to interrupt. In this case, we're simply going to say that we want to interrupt a instance that's `running` and has a `Name` tag with value `Jenkins Master (Spot)`. +* Notice that instances are going to be **chosen randomly** + +#### Create the EC2 Spot Interruption Experiment with FIS + +Let's continue by creating the Spot interruption experiment template using Cloudformation. You can view the CloudFormation template (**fisspotinterruption.yaml**) at GitHub [here](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/amazon-ec2-spot-cicd-workshop/fisspotinterruption.yaml). To download it, you can run the following command: + +``` +wget https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/amazon-ec2-spot-cicd-workshop/fisspotinterruption.yaml +``` + +Now, simply run the following commands to create the FIS experiment: + +``` +aws cloudformation create-stack --stack-name fis-spot-interruption --template-body file://fisspotinterruption.yaml --capabilities CAPABILITY_NAMED_IAM +aws cloudformation wait stack-create-complete --stack-name fis-spot-interruption +``` \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/spot_interruption_fis.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/spot_interruption_fis.md new file mode 100644 index 00000000..d0db6bb2 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/spot_interruption_fis.md @@ -0,0 +1,54 @@ +--- +title: "Interrupting a Spot Instance" +weight: 150 +--- + +In this section, you're going to launch a Spot Interruption using FIS and then verify that the capacity has been replenished and Jenkins is still working. This will help you to confirm the low impact of your workloads when implementing Spot effectively. + +#### Launch the Spot Interruption Experiment +After creating the experiment template in FIS, you can start a new experiment to interrupt one Spot instance. Run the following command: + +``` +FIS_EXP_TEMP_ID=$(aws cloudformation describe-stacks --stack-name fis-spot-interruption --query "Stacks[0].Outputs[?OutputKey=='FISExperimentID'].OutputValue" --output text) +FIS_EXP_ID=$(aws fis start-experiment --experiment-template-id $FIS_EXP_TEMP_ID --no-cli-pager --query "experiment.id" --output text) +``` + +Wait around 30 seconds, and you should see that the experiment completes. Run the following command to confirm: + +``` +aws fis get-experiment --id $FIS_EXP_ID --no-cli-pager +``` + +At this point, FIS has triggered a Spot interruption notice, and in two minutes the instance will be terminated. + +Go to CloudWatch Logs group `/aws/events/spotinterruptions` to confirm that instance received the termination notice. + +You should see a log message like this one: + +![SpotInterruptionLog](/images/amazon-ec2-spot-cicd-workshop/spotinterruptionlog.png) + +#### Verify the actions taken by Jenkins Auto Scaling group + +You are running a Jenkins instance launched by an Auto Scaling group that will launch a new instance if the desired capacity is not compliant. + +You can run the following command to see the list of instances with the date and time when they were launched. + +``` +aws ec2 describe-instances --filters\ + Name='tag:Name',Values='Jenkins Master (Spot)'\ + Name='instance-state-name',Values='running'\ + | jq '.Reservations[].Instances[] | "Instance with ID:\(.InstanceId) launched at \(.LaunchTime)"' +``` + +You should see a list of instances with the date and time when they were launched. You'll also see the new instance launched after the interruption. + +```output +"Instance with ID:i-04c6ced30794e965b launched at 2022-04-06T14:02:49+00:00" +"Instance with ID:i-0136152e14053af81 launched at 2022-04-06T14:11:25+00:00" +``` + +#### Verify that Jenkins is still working + +Login to the Jenkins server again and start running some jobs as [you did previously](/amazon-ec2-spot-cicd-workshop/jenkins-asg/test-persistence.html#verify-that-the-new-spot-instance-is-running-the-jenkins-server). + +You should see that the EC2 Fleet plugin is still working and is launching Spot instances as build agents. \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/test-build.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/test-build.md new file mode 100644 index 00000000..8df4dae5 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/test-build.md @@ -0,0 +1,22 @@ ++++ +title = "Configure jobs to use Spot" +weight = 120 ++++ +As alluded to in the previous section, you'll need to configure your build jobs so that they are executed on the build agents running in your Spot instances. In addition, you could configure jobs to execute concurrent builds if necessary - this will help you in testing the scale-out of your fleet. + +1. Go back to the Jenkins home screen and **repeat the following for each of the five Apache build projects** that are configured in your Jenkins instance: + 1. Click on the title of the build job and then click on the **Configure** link toward the left side of the screen; + 2. In the General section, click on the **Execute concurrent builds if necessary** checkbox and the **Restrict where this project can be run** checkbox. Next, enter **spot-agents** as the Label Expression (Note: if you select the auto-complete option instead of typing out the full label, Jenkins will add a space to the end of the label - be sure to remove any trailing spaces from the label before proceeding); + 3. Click on the **Save** button towards the bottom of the screen. + +## Test Spot Builds and Scale-out +Now it’s time to test out how Jenkins handles pushing builds to spot instances running build agents at scale. There are two things that you'll want to verify here; that your builds run successfully on the Spot instances, and that your ASG scales out when there are build jobs queued for more than a few minutes. + +1. Go Back to the Jenkins home page, click on the **Schedule a Build** icon (which looks like a play symbol) for each of the five Apache projects, starting from the **Apache PDFBox** project and working upward. This will queue up five build jobs, this will trigger the EC2 Fleet plugin to start launching build agents using Spot instances; +2. When any of the build jobs have been completed, click on the **Schedule a Build** icon corresponding to that job to re-add it back to the build queue - the intent here is to keep the build queue populated with a backlog of build jobs until your Auto Scaling group has scaled out and build jobs are executing on Spot instances; +3. After around four minutes, the **EC2 Fleet Status** reported to the left of the screen will increment the **target** count to 5, indicating that the plugin has requested a scale-out action from the plugin. After a few moments, five build instances will appear in the **Build Executor Status**, though these build agents will initially appear to be offline. Once the instances have had the chance to complete the launch and bootstrapping processes (which takes around two minutes), your Jenkins Master will deploy the build agent to them via SSH, and it will come online and process the next build jobs in the queue. Once you have concurrent builds being executed on the Spot instances, you can stop adding build jobs to the build queue; +4. After a period of around a five minutes, after your builds have completed, the Spot instances should be terminated by the plugin. + +{{% notice note %}} +If the build agents are keep showing as `offline`, make sure that you've configured correctly the SSH key pair in the credentials section when configuring the EC2 Fleet plugin. You can click on the agent to open the logs and see why it's not coming online. +{{% /notice %}} \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/tracking_spot_interruptions.md b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/tracking_spot_interruptions.md new file mode 100644 index 00000000..ab3ed2a6 --- /dev/null +++ b/content/amazon-ec2-spot-cicd-workshop/jenkins-ecs/tracking_spot_interruptions.md @@ -0,0 +1,36 @@ +--- +title: "Tracking Spot interruptions" +weight: 140 +--- +At some point in time, you might receive a Spot interruption notice when On-Demand needs the capacity back. When this happens, your Spot instance will be provided with a two-minute notice of termination and after that time lapses, the instance will be terminated. At this point, the Auto Scaling group will observe that there are no running instances in the group and because the desired capacity is one. Then, it will launch a replacement instance from the pool with more capacity available (this is why diversification across many capacity pool is a best practice). The new instance will be launched and bootstrapped in exactly the same manner as your original instance. + +Before we start testing the resiliency of Jenkins with Spot, let's configure CloudWatch Logs to log Spot interruptions. If a Jenkins job fails, we'll be able to check if the failure correlates to a Spot interruption or not. + +{{% notice note %}} +In most cases we don't really need to track the Spot interruptions as Jenkins jobs can be retried if it fails. However, when we're starting with running our Jenkins jobs on Spot instances tracking could be useful. Organizations can use this data to correlate possible job failures or prolonged execution times, in case Spot instances were interrupted during a job execution. +{{% /notice %}} + +#### Creating the CloudFormation Stack to Track EC2 Spot Interruptions + +We've created a CloudFormation template that includes all the resources you need to track EC2 Spot Interruptions. The stack creates the following: + +* An Event Rule for tracking EC2 Spot Interruption Warnings +* A CloudWatch Log group to log interruptions and instance details +* IAM Role to allow the event rule to log into CloudWatch Logs + +You can view the CloudFormation template (**cloudwatchlogs.yaml**) at GitHub [here](https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/track-spot-interruptions/cloudwatchlogs.yaml). To download it, you can run the following command: + +``` +wget https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/workshops/track-spot-interruptions/cloudwatchlogs.yaml +``` + +After downloading the CloudFormation template, run the following command in a terminal: + +``` +aws cloudformation create-stack --stack-name track-spot-interruption --template-body file://cloudwatchlogs.yaml --capabilities CAPABILITY_NAMED_IAM +aws cloudformation wait stack-create-complete --stack-name track-spot-interruption +``` + +You should see an event rule in the Amazon EventBridge console, like this: + +![Spot Interruption Event Rule](/images/tracking-spot/itn-event-rule.png) diff --git a/content/amazon-ec2-spot-cicd-workshop/lab1.md b/content/amazon-ec2-spot-cicd-workshop/lab1.md deleted file mode 100644 index 13f88e32..00000000 --- a/content/amazon-ec2-spot-cicd-workshop/lab1.md +++ /dev/null @@ -1,98 +0,0 @@ -+++ -title = "Lab 1: Reduce the cost of builds using Amazon EC2 Spot Fleet" -weight = 20 -+++ -The Jenkins server that was launched by the CloudFormation template has a number of build projects preconfigured. However by default, all builds will be executed on the same instance that Jenkins is running on. This results in a couple of less-than-desirable behaviours: -* When CPU-intensive builds are being executed, there may not be sufficient system resources to display the Jenkins server interface; and -* The Jenkins server is often provisioned with more resources than the server interface requires in order to allow builds to execute. When builds are not being executed, these server resources are essentially going to waste. - -To address these behaviours, Jenkins provides the capability to execute builds on external hosts (called build agents). Further, AWS provides a Jenkins plugin to allow Jenkins to scale out a fleet of EC2 instances in order to execute build jobs on. This lab will focus on implementing EC2 Spot build agents, showcasing what a batch processing workload typically looks like when using Amazon EC2 Spot instances. - -## PROVISION A SPOT FLEET FOR YOUR BUILD AGENTS -Before configuring the EC2 Fleet Jenkins Plugin, create a Spot Fleet that will be used by the plugin to perform your application builds. As this is a batch processing use case, remember the best practices for this type of workload - leverage per-second billing (catered for through the use of an Amazon Linux AMI defined in the Launch Template); optimize for lowest cost; determine job completion and retry failed jobs (the former is handled by the Jenkins EC2 Fleet plugin); and be instance flexible. - -1. Go to the **EC2** console and click on the **Spot Requests** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2sp/v1/spot/home?region=eu-west-1#)); -2. Click on the **Request Spot Instances** button; -3. At the first screen of the Spot instance launch wizard: - 1. Under the Tell us your application or task need heading, ensure that the **Load balancing workloads** option selected (Note: do NOT select the Flexible workloads option as this will deploy a Spot Fleet with weightings applied to some of the EC2 instance types which will adversely impact how the plugin scales out); - 2. In the Configure your instances section, select the **JenkinsBuildAgentLaunchTemplate** template from the Launch template dropdown (if this option is not present in the dropdown, please verify that the CloudFormation template that you launched during the Workshop Preparation has deployed successfully). Change the Network to be the **Spot CICD Workshop VPC**. After making this selection, enable the check boxes for all three Availability Zones and then select the **Amazon EC2 Spot CICD Workshop Public Subnet** associated with each availability zone as the subnet to launch instances in; - 3. At the Tell us how much capacity you need section, keep the Total target capacity at **1** instance and the Optional On-Demand portion set to **0**, and then tick the **Maintain target capacity** checkbox. Once selected, leave the Interruption behavior set to **Terminate**; - 4. While the new Spot Instances wizard makes some good recommendations to you on how best to add diversity to the fleet that you're creating, the recommended instance types provide more resources than we strictly need in this workshop - so to the right of the Fleet request settings heading, clear the tick from **Apply recommendations** checkbox. Click on each of the **Remove** links associated with the all of the instance types initially defined to remove them from the fleet configuration. Then click on the **Select instance types** button and add the **t2.medium**, **t2.large**, **t3.medium** and **t3.large** instance types to the fleet definition (Hint: you may need to adjust the **vCPUs** and **Memory (GiB)** filters to reveal all of these instance types). Once the checkboxes for the required instance types have been ticked, click on the **Select** button. Once you have the four desired instance types listed in the fleet request, select the **Lowest Price** Fleet allocation strategy (since we’re interested in keeping cost to an absolute minimum for this use case); - 5. Review the Your fleet request as a glance section - it should indicate that your Fleet strength is Strong as a result of being able to draw instances from 12 instance pools, and your Estimated price should indicate that you're expecting to make a 70% saving compared to the cost of equivalent on-demand resources; - 6. Lastly, click on the **Launch** button. -4. Make a note of the Request ID of the Spot Fleet that you’ve just created. - -## CREATE A SECRET KEY AND ACCESS KEY FOR THE PLUGIN -The CloudFormation template that you deployed in Lab 1 created an IAM User called **SpotCICDWorkshopJenkins**. Jenkins will use this IAM User to control the spot fleet used for your build slaves. Generate a secret key and access key for this user. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **IAM** console and click on the **Users** option from the left frame (or [click here](https://console.aws.amazon.com/iam/home?region=eu-west-1#/users)); -2. Click on the **SpotCICDWorkshopJenkins** user; -3. Click on the **Security credentials** tab, then click on the **Create access key** button – Make a note of the Access key ID and Secret access key, then click the **Close** button. -{{% /expand%}} - -## SIGN IN TO YOUR JENKINS SERVER -The CloudFormation template deployed during the Workshop Preparation stage deployed a Jenkins server on to an on-demand instance within your VPC and configured an Application Load Balancer (ALB) to proxy requests from the public Internet to the server. You can obtain the DNS name for the ALB from the Output tab of your CloudFormation template. Point your web browser to this DNS name and sign in using **spotcicdworkshop** as the Username and the password that you supplied to the CloudFormation template as the password. -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1)); -2. Click on the checkbox associated with the **SpotCICDWorkshop** stack, then click on the **Outputs** tab toward the bottom of the screen; -3. Make a note of the DNS name for the Application Load Balancer, which is associated with the **JenkinsDNSName** key; -4. Open up a new tab in your browser and enter the DNS name in the address bar. You should be greeted with a Jenkins Sign In screen: - 1. Enter in **spotcicdworkshop** as the Username; - 2. Enter in the password that you supplied to the CloudFormation template as the Password. -{{% /expand%}} - -## CONFIGURE THE EC2 FLEET JENKINS PLUGIN -The EC2 Fleet Jenkins Plugin was installed on the Jenkins server during the CloudFormation deployment - but now the plugin needs to be configured. You'll first need to supply the IAM Access Key ID and Secret Key that you created so that the plugin can find your Spot Fleet request. You'll then need to get the plugin to **Launch slave agents via SSH** and provide valid SSH credentials (don't forget to consider how Host Key Verification should be set when using Spot instances). - -When configuring the plugin, think about how you could force build processes to run on the spot instances (use the **spot-agents** label), and consider how you can verify that the fleet scales out when there is a backlog of build jobs waiting to be processed. - -{{%expand "Click to reveal detailed instructions" %}} -1. From the Jenkins home screen, click on the **Manage Jenkins** link on the left side menu, and then the **Configure System** link; -2. Scroll all the way down to the bottom of the page and under the **Cloud** section, click on the **Add a new cloud button**, followed by the **Amazon SpotFleet** option; -3. Under the Spot Fleet Configuration section, click on the **Add** button next to the AWS Credentials [sic] dropdown, then click on the **Jenkins** option. This will pop up a new **Jenkins Credentials Provider: Jenkins** sub-form. Fill out the form as follows: - 1. Change the Kind to **AWS Credentials**; - 2. Change the Scope to **System (Jenkins and nodes only)** – you don’t want your builds to have access to these credentials! - 3. At the ID field, enter **SpotCICDWorkshopJenkins**; - 4. At the Access Key ID and Secret Access Key fields, enter in the information that you gathered earlier; - 5. Click on the **Add** button; -4. Once the sub-form disappears, select your Access Key ID from the AWS Credentials dropdown - the plugin will then issue a request to the AWS APIs and populate the list of regions; -5. Select **eu-west-1** from the Region dropdown - the plugin will now attempt to obtain a list of Spot Fleet requests made in the selected region; -6. Select the Request Id of the Spot Fleet that you created earlier from the Spot Fleet dropdown (though it might already be selected) and then select the **Launch slave agents via SSH** option from the Launcher dropdown - this should reveal additional SSH authentication settings; -7. Click the **Add** button next to the Credentials dropdown and select the **Jenkins** option. This will pop up another **Jenkins Credentials Provider: Jenkins** sub-form. Fill out the form as follows: - 1. Change the Kind to **SSH Username with private key**; - 2. Change the Scope to **System (Jenkins and nodes only)** – you also don’t want your builds to have access to these credentials; - 3. At the Username field, enter **ec2-user**; - 4. For the Private Key, select the **Enter directly** radio button. Open the .pem file that you downloaded during the workshop setup in a text editor and copy the contents of the file to the Key field including the BEGIN RSA PRIVATE KEY and END RSA PRIVATE KEY fields; - 5. Click on the **Add** button. -8. Select the ec2-user option from the Credentials dropdown; -9. Given that Spot instances will have a random SSH host fingerprint, select the **Non verifying Verification Strategy** option from the Host Key Verification Strategy dropdown; -10. Mark the **Connect Private** checkbox to ensure that your Jenkins Master will always communicate with the Agents via their internal VPC IP addresses (in real-world scenarios, your build agents would likely not be publicly addressable); -11. Change the Label field to be **spot-agents** - you'll shortly reconfigure your build job to run on slave instances featuring this label; -12. Set the Max Idle Minutes Before Scaledown to **5**. As AWS launched per-second billing in 2017, there's no need to keep a build agent running for too much longer than it's required; -13. Change the Maximum Cluster Size from **1** to **2** (so that you can test fleet scale-out); -14. Finally, click on the **Save** button. - -Within sixty-seconds, the Jenkins Slave Agent should have been installed on to the Spot instance that was launched by your Spot fleet; you should see an EC2 instance ID appear underneath the Build Executor Status section on the left side of the Jenkins user interface. Underneath that, you should see that there is a single Build Executor on this host, which is in an idle state. -{{% /expand%}} - -## RECONFIGURE YOUR BUILD JOBS TO USE THE NEW SPOT INSTANCE(S) -As alluded to in the previous section, you'll need to reconfigure your build jobs so that they are executed on the build agents running in your Spot fleet (again, use the **spot-agents** label). In addition, configure each job to execute concurrent builds if necessary - this will help you in testing the scale-out of your fleet. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go back to the Jenkins home screen and **repeat the following for each of the five Apache build projects** that are configured in your Jenkins deployment: - 1. Click on the title of the build job and then click on the **Configure** link toward the left side of the screen; - 2. In the General section, click on the **Execute concurrent builds if necessary** checkbox and the **Restrict where this project can be run** checkbox. Next, enter **spot-agents** as the Label Expression (Note: if you select the auto-complete option instead of typing out the full label, Jenkins will add a space to the end of the label - be sure to remove any trailing spaces from the label before proceeding); - 3. Click on the **Save** button towards the bottom of the screen. -{{% /expand%}} - -## TEST SPOT BUILDS AND SCALE-OUT -Now it’s time to test out how Jenkins handles pushing builds to spot instances running build agents at scale. There are two things that you'll want to verify here; that your builds run successfully on the Spot instances, and that your Spot Fleet scales out when there are build jobs queued for more than a few minutes. - -1. Back at the Jenkins home page, first click on the **ENABLE AUTO REFRESH** link that's located towards to top-right corner of the screen - this will enable a full refresh of the Jenkins user interface every 10 seconds allowing you to see regular updates to the status of each build. Next, click on the **Schedule a Build** icon (which looks like a play symbol superimposed over a clock) for each of the five Apache projects, starting from the **Apache PDFBox** project and working upward. This will queue up five build jobs, the first of which will be immediately assigned to the Spot instance to be worked on; -2. When any of the build jobs have been completed, click on the **Schedule a Build** icon corresponding to that job to re-add it back to the build queue - the intent here is to keep the build queue populated with a backlog of build jobs until your Spot Fleet has scaled out and build jobs are executing on both Spot instances; -3. After a couple of minutes (typically during the first **Apache Helix** build - around four minutes after you initiate the first build), the EC2 Fleet Status reported to the left of the screen will increment the **target** count to 2, indicating that the plugin has requested a scale-out action from the plugin. After a few moments, a second build instance will appear in the **Build Executor Status**, though this build agent will initially appear to be offline. Once the instance has had the chance to complete the launch and bootstrapping processes (which takes around two minutes), your Jenkins Master will deploy the build agent to it via SSH, and it will come online and process the next build job in the queue. Once you have concurrent builds being executed on two Spot instances, you can stop adding build jobs to the build queue; -4. After a period of around a five minutes after your builds have completed, one of the Spot instances should be terminated by the plugin - there's no need to wait for this to happen (take our word for it, but you can verify this later). - -## PROCEED TO LAB 2 -Once you've verified that builds are succeeding and that your Spot Fleet is capable of scaling out to handle queued build jobs, you may proceed with [Lab 2](/amazon-ec2-spot-cicd-workshop/lab2.html). diff --git a/content/amazon-ec2-spot-cicd-workshop/lab2/_index.md b/content/amazon-ec2-spot-cicd-workshop/lab2/_index.md deleted file mode 100644 index f84e1da7..00000000 --- a/content/amazon-ec2-spot-cicd-workshop/lab2/_index.md +++ /dev/null @@ -1,105 +0,0 @@ -+++ -title = "Lab 2: Deploy testing environments using Spot & Launch Templates" -weight = 30 -+++ -Now that you are carrying out your software builds on Spot instances, the next step is to build out a CI/CD pipeline to have your software deployed to a test environment also running on Spot instances. Pipelines within Jenkins are defined using another plugin, and you'll define pipeline steps to both deploy and terminate your testing environment. The pipeline steps that you define in this lab will call a Lambda function, which in turn deploys a CloudFormation template to brings up your testing environment, and deletes the template to tear it down. - -If you view the CloudFormation template that is used to deploy the test environment, pay close attention to the following lines: - -* Lines 183-218 contains the resource definition for the Launch Template used within the Spot Fleet. Of particular note, the User Data defined within the Launch Template updates the system with all current patches, installs the Java 8 OpenJDK and Apache Tomcat 8, deploys the build artifact generated by Jenkins and starts up the Tomcat service; -* Lines 220-250 define the Spot Fleet that is used to build out the test environment. Specifically, we are following the best practices for stateless applications using Spot: the fleet uses a diversified allocation strategy to ensure that instances are distributed across multipe availablility zones and the fleet is set to replace any instances that are unhealthy; - -## INCREASE THE NUMBER OF BUILD EXECUTORS ON YOUR JENKINS BUILD AGENTS -When executing a pipeline job in Jenkins, the pipeline consumes one of the build execution slots. Given that the the software builds themselves are executed using a build agent on a Spot instance, you should configure the Jenkins master server to also have a build executor - for pipeline execution to run on. - -{{%expand "Click to reveal detailed instructions" %}} -1. Back at the Jenkins home page, click on the **Manage Jenkins** link on the left side menu again, and then the **Configure System** link; -2. Scroll down to the bottom of the page and under the **Spot Fleet Configuration** section, increase the **Number of Executors** to 3; -3. Click on the **Save** button at the bottom of the screen; -4. In the left column under **Build Executor Status**, you should see the EC2 Instance ID for the instances that make up your build fleet. Click on the Instance ID and then click on the **Delete Agent** link on the left hand side of the screen. When prompted if you're sure about deleting the agent, click on the **Yes** button. Deleting the agent from the host will force Jenkins to re-install it, this time with multiple build executors. -{{% /expand%}} - -## RECONFIGURE THE GAME OF LIFE PROJECT -The S3 Publisher Plugin should be pre-configured for use (it will use the IAM policy attached to the EC2 Instance Profile used by the Jenkins Master or the build agents to grant access to the S3 bucket where your deployment artifacts will be uploaded to). First up, you'll need to reconfigure the **Game of Life** project to publish it's artifacts to the S3 bucket that was created by the CloudFormation stack deployed at the beginning of this workshop. - -1. Back at the Jenkins home page, go into the **Game of Life** project by clicking on its Name link. Next, click on the **Configure** link in the left side menu; -2. In the General section, click on the **Execute concurrent builds if necessary** checkbox and the **Restrict where this project can be run** checkbox. Next, enter **spot-agents** as the Label Expression; -3. Scroll down to the Post-build Actions section and to the bottom of that section, click on the **Add post-build action** button, then select the **Publish artifacts to S3 Bucket** option. A new configuration block will appear; -4. Within the new configuration block, select the **Spot CICD Workshop Artifacts** option from the S3 Profile dropdown. Click on the **Add** button next to the Files to upload label; -5. In the Source field, enter **\*\*/*.war** as the match condition for your build artifact - this will ensure that the WAR (Web Application Resource) file produce by our build process is uploaded; -6. In the Destination bucket field, enter in the name of the S3 bucket that was created by the CloudFormation template deployed in Lab 1 (the S3 bucket name can be obtained from the Outputs tab of the SpotCICDWorkshop stack in the CloudFormation console or by [clicking here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks?filter=active) - look for the value associated with the **DeploymentArtifactsS3Bucket** key); -7. In the Bucket Region dropdown, select the **eu-west-1** region; -8. Finally, mark both the **No upload on build failure** and **Publish from Slave** checkboxes - this will ensure that uploads only occur when a build is successful, and to ensure that our build agents are aware that they will also have the permissions to upload (given that they are using the same IAM role that our master server is using); -9. At the bottom of the page, click on the **Save** button. - -## INSPECT THE CI/CD PIPELINE AND LAMBDA FUNCTION THAT WILL MANIPULATE THE TEST ENVIRONMENT -Your Jenkins server has been preconfigured with a Game of Life Pipeline job that contains four stages as defined in the following Groovy script: - -```groovy -node { - stage('Build') { - // Build our project and upload artifact to S3 - build 'Game of Life' - } - stage('Deploy') { - // Invoke a Lambda function that will create our test environment by launching a Spot Fleet - invokeLambda([awsAccessKeyId: '', awsRegion: 'eu-west-1', awsSecretKey: '', functionName: 'SpotCICDWorkshop_ManageTestEnvironment', payload: '''{ - "action": "deploy", - "stackName": "GameOfLife", - "jobBaseName": "${JOB_BASE_NAME}", - "buildId": "${BUILD_ID}" - }''', synchronous: true, useInstanceCredentials: true]) - } - stage('Test') { - // A manual approval stage designed to halt the pipeline until someone indicates that the pipeline can proceed - input 'Run your tests' - } - stage('Terminate') { - // Invoke a Lambda function to tear down our test environment once testing has been completed - invokeLambda([awsAccessKeyId: '', awsRegion: 'eu-west-1', awsSecretKey: '', functionName: 'SpotCICDWorkshop_ManageTestEnvironment', payload: '''{ - "action": "terminate", - "stackName": "GameOfLife", - "jobBaseName": "${JOB_BASE_NAME}", - "buildId": "${BUILD_ID}" - }''', synchronous: true, useInstanceCredentials: true]) - } -} -``` - -* The **Build** stage should simply execute the Game of Life build job that you just reconfigured. As you'd expect from the configuration change that you made, the deployment artifact generated by this stage will be stored in an S3 bucket; -* The **Deploy** stage will need to invoke a Lambda function, which in turn will deploy a CloudFormation template. The template will create a Spot Fleet consisting of two instances using configuration defined in an EC2 Launch Template; and within this Template will be EC2 UserData that will deploy your build artifact and all of its dependencies. The template will also deploy an Application Load Balancer to front the Game of Life application, through which you will test the application; -* The **Test** stage will be a manual approval stage, designed to allow you the opportunity to test out the deployed artifact; and -* The **Terminate** stage will invoke a Lambda function that will destroy the test environment by deleting the CloudFormation stack. - -In the Groovy script, note that the same Lambda function (SpotCICDWorkshop_ManageTestEnvironment) is called for both the Deploy and Terminate stages, but a different “action” parameter value is passed on to the function. - -Next, take a look at the Lambda function to observe what it's going to do upon execution - you can view the function by going to the **Lambda** console and clicking on the **SpotCICDWorkshop\_ManageTestEnvironment** function (or by [clicking here](https://eu-west-1.console.aws.amazon.com/lambda/home?region=eu-west-1#/functions/SpotCICDWorkshop_ManageTestEnvironment)) - or you can click on the code snippet link to the left to view a version of the code that has not been customised to access resources within your account by CloudFormation. - -The **deploy** action within the function starts from line 4: - -* Lines 9-35 contain the CloudFormation parameters that are passed on to the CloudFormation template. These parameters were generated based on the outputs of the initial CloudFormation template that you deployed in the Workshop Preparation; -* Line 36 defines the IAM role that is assumed when deploying the template - again, this role was deployed as a part of the initial CloudFormation template defined in the Workshop Preparation; -* Line 37 contains the URL of the CloudFormation template that is going to be launched by the function: [https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop_game-of-life.yaml](https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop_game-of-life.yaml) -* Line 39 attempts to deploy the stack; and -* If the request to deploy the stack was successful, lines 43-56 store some metadata (including the CloudFormation Stack ID) into a DynamoDB table that was created by the initial CloudFormation template. - -The **terminate** action within the function starts from line 61: - -* Lines 63-70 queries the DynamoDB table to obtain the CloudFormation Stack ID added by the deploy action; and -* If the stack ID was able to be obtained from DynamoDB, it is used to tell CloudFormation to delete the stack with that ID on lines 74-81. - -Finally, lines 87-102 contain the boilerplate code used to load the relevant AWS SDK libraries and call the actions defined above. - -## DEPLOY, TEST, THEN TERMINATE A TEST ENVIRONMENT USING THE GAME OF LIFE PIPELINE -You should now be at the stage where your the Game of Life Pipeline can be tested. To commence this, go back to the Jenkins Home page and click on the **Game of Life Pipeline** project. Within the project, click on the **Build Now** link on the left. When the pipeline initiates, you'll get to see the execution status of each stage as the workflow progresses through the pipeline. - -Once the deploy stage has been completed successfully: - -1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1)) and wait for the deployment of the GameOfLife stack to switch to a **CREATE_COMPLETE** status; -2. When the stack has deployed, click on the checkbox next to the GameOfLife stack, then click on the Outputs tab at the bottom of the screen; -3. Copy the value associated with the **GameOfLifeDNSName** key into the address bar of a new browser tab and append the **/gameoflife** suffix before hitting the Enter key. - -Once you've verified that that the Game of Life web application loads, return to Jenkins and click on the **Test** pipeline stage and click on **Proceed** to get the pipeline to process the Terminate stage. After this final stage has executed, go back to the CloudFormation console to verify that the GameOfLife stack is being deleted. - -## PROCEED TO LAB 3 -Once your test environment has been terminated, you may proceed with [Lab 3](/amazon-ec2-spot-cicd-workshop/lab3.html). diff --git a/content/amazon-ec2-spot-cicd-workshop/lab2/clfn.md b/content/amazon-ec2-spot-cicd-workshop/lab2/clfn.md deleted file mode 100644 index ff908867..00000000 --- a/content/amazon-ec2-spot-cicd-workshop/lab2/clfn.md +++ /dev/null @@ -1,262 +0,0 @@ -+++ -title = "Code snippet: The Test Environment CloudFormation template" -weight = 10 -+++ -Below is a sanitized version of the CloudFormation template that will be used to launch your test environments using an Amazon EC2 Spot Fleet provisioned from an EC2 Launch Template: -```yaml -AWSTemplateFormatVersion: '2010-09-09' - -Description: A CloudFormation template that will deploy a test environment as a part of the Amazon EC2 Spot CI/CD Workshop. This template is provided as-is under a modified MIT license - please see https://github.com/aws-samples/amazon-ec2-spot-cicd-workshop/blob/master/LICENSE - -Parameters: - KeyPair: - Description: The Key Pair created in Step 3 of the Preparation Lab - Type: AWS::EC2::KeyPair::KeyName - - CurrentIP: - Description: The IP address supplied by the workshop participant when provisioning the Master CloudFormation template for this workshop. - Type: String - - AMILookupLambdaFunctionARN: - Description: The ARN correspoding to the AMI Lookup Lambda Function created by the Master CloudFormation template. - Type: String - - DeploymentArtifactsS3Bucket: - Description: The S3 bucket created by the Master CloudFormation template to host deployment artifacts. - Type: String - - VPC: - Description: The VPC created by the Master CloudFormation template, where the test environment will be provisioned. - Type: String - - SubnetA: - Description: Subnet A created by the Master CloudFormation template, one of three which will be used by the test environment. - Type: String - - SubnetB: - Description: Subnet B created by the Master CloudFormation template, one of three which will be used by the test environment. - Type: String - - SubnetC: - Description: Subnet C created by the Master CloudFormation template, one of three which will be used by the test environment. - Type: String - - -Resources: - - IAMRoleGameOfLife: - Type: AWS::IAM::Role - # DependsOn: None - # DependedOn: InstanceProfileGameOfLife - Properties: - AssumeRolePolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Principal: - Service: - - ec2.amazonaws.com - Action: - - sts:AssumeRole - Path: / - Policies: - - PolicyName: Test - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: s3:GetObject - Resource: !Sub "arn:aws:s3:::${DeploymentArtifactsS3Bucket}/*" - - InstanceProfileGameOfLife: - Type: AWS::IAM::InstanceProfile - DependsOn: IAMRoleGameOfLife - # DependedOn: GameOfLifeLaunchTemplate - Properties: - Path: '/' - Roles: - - !Ref IAMRoleGameOfLife - - AMILookupCustomResource: # A custom resource that provides the latest Amazon Linux AMI via AMILookupCustomResource.Id - Type: Custom::AMILookup - # DependsOn: - # DependedOn: - Properties: - Architecture: HVM64 - Region: !Ref AWS::Region - ServiceToken: !Ref AMILookupLambdaFunctionARN - - SecurityGroupGameOfLifeALB: # A Security Group that allows ingress access for HTTP on ALBs and used to access the Game of Life test environment - Type: AWS::EC2::SecurityGroup - # DependsOn: None - # DependedOn: None - Properties: - GroupName: Spot CICD Workshop Game of Life ALB Security Group - GroupDescription: A Security Group that allows ingress access for HTTP on ALBs and used to access the Game of Life test environment - SecurityGroupIngress: - - IpProtocol: tcp - FromPort: 80 - ToPort: 80 - CidrIp: 0.0.0.0/0 - VpcId: !Ref VPC - - SecurityGroupGameOfLifeEC2: # A Security Group that allows ingress access for SSH and the default port that the Game of Life test application will run on - Type: AWS::EC2::SecurityGroup - DependsOn: SecurityGroupGameOfLifeALB - # DependedOn: JenkinsOnDemandEC2Instance - Properties: - GroupName: Spot CICD Workshop Game of Life EC2 Security Group - GroupDescription: A Security Group that allows ingress access for SSH and the default port that a Jenkins Master will run on - SecurityGroupIngress: - - IpProtocol: tcp - FromPort: 22 - ToPort: 22 - CidrIp: !Ref CurrentIP - - IpProtocol: tcp - FromPort: 8080 - ToPort: 8080 - CidrIp: !Ref CurrentIP - - IpProtocol: tcp - FromPort: 8080 - ToPort: 8080 - SourceSecurityGroupId: !Ref SecurityGroupGameOfLifeALB - VpcId: !Ref VPC - - GameOfLifeALB: # This is the Application Load Balancer that resides in front of the Game of Life test environment and is responsible for port-mapping requests from TCP:80 to TCP:8080 - Type: AWS::ElasticLoadBalancingV2::LoadBalancer - DependsOn: SecurityGroupGameOfLifeALB - # DependedOn: GameOfLifeALBListener - Properties: - Name: GameOfLifeALB - Scheme: internet-facing - SecurityGroups: - - !Ref SecurityGroupGameOfLifeALB - Subnets: - - !Ref SubnetA - - !Ref SubnetB - - !Ref SubnetC - - GameOfLifeALBTargetGroup: # This is the Target Group used by the GameOfLifeALB load balancer - Type: AWS::ElasticLoadBalancingV2::TargetGroup - # DependsOn: None - # DependedOn: GameOfLifeALBListener, GameOfLifeALBListenerRule - Properties: - HealthCheckIntervalSeconds: 15 - HealthCheckPath: /gameoflife/ - HealthCheckPort: 8080 - HealthCheckProtocol: HTTP - HealthCheckTimeoutSeconds: 5 - HealthyThresholdCount: 2 - Matcher: - HttpCode: 200 - Name: GameOfLifeALBTargetGroup - Port: 8080 - Protocol: HTTP - UnhealthyThresholdCount: 4 - VpcId: !Ref VPC - - GameOfLifeALBListener: # This is the ALB Listener used to access the Game of Life test environment - Type: AWS::ElasticLoadBalancingV2::Listener - DependsOn: - - GameOfLifeALB - - GameOfLifeALBTargetGroup - # DepenededOn: None - Properties: - DefaultActions: - - Type: forward - TargetGroupArn: !Ref GameOfLifeALBTargetGroup - LoadBalancerArn: !Ref GameOfLifeALB - Port: 80 - Protocol: HTTP - - GameOfLifeALBListenerRule: # The ALB Listener rule that forwards all traffic destined for the Game of Life test environment to the appropriate Target Group - Type: AWS::ElasticLoadBalancingV2::ListenerRule - DependsOn: - - GameOfLifeALBListener - - GameOfLifeALBTargetGroup - # DependedOn: None - Properties: - Actions: - - Type: forward - TargetGroupArn: !Ref GameOfLifeALBTargetGroup - Conditions: - - Field: path-pattern - Values: - - "/*" - ListenerArn: !Ref GameOfLifeALBListener - Priority: 1 - - GameOfLifeLaunchTemplate: # The Launch Template that will be used to deploy the test environment - Type: AWS::EC2::LaunchTemplate - DependsOn: SecurityGroupGameOfLifeEC2 - # DependedOn: None - Properties: - LaunchTemplateName: GameOfLifeLaunchTemplate - LaunchTemplateData: - BlockDeviceMappings: - - DeviceName: "/dev/xvda" - Ebs: - DeleteOnTermination: 'true' - VolumeSize: 8 - VolumeType: gp2 - IamInstanceProfile: - Name: !Ref InstanceProfileGameOfLife - ImageId: !GetAtt AMILookupCustomResource.Id - InstanceType: t3.micro - KeyName: !Ref KeyPair - SecurityGroupIds: - - !Ref SecurityGroupGameOfLifeEC2 - TagSpecifications: - - ResourceType: instance - Tags: - - Key: Name - Value: Game of Life Test Instance - UserData: - Fn::Base64: !Sub | - #!/bin/bash - # Install all pending updates to the system - yum -y update - # Install Java 8 & Apache Tomcat 8 - yum -y install java-1.8.0-openjdk tomcat8 - # Download the Game of Life build artifact to Tomcat's webapps directory - aws s3 cp s3://${DeploymentArtifactsS3Bucket}/gameoflife-web/target/gameoflife.war /usr/share/tomcat8/webapps/gameoflife.war - # Start the Tomcat8 Service - service tomcat8 start - - GameOfLifeSpotFleet: # The Spot Fleet definition that will launch the EC2 instances that will comprise the test environment - Type: AWS::EC2::SpotFleet - DependsOn: GameOfLifeLaunchTemplate - # DependedOn: None - Properties: - SpotFleetRequestConfigData: - AllocationStrategy: diversified - IamFleetRole: !Sub arn:aws:iam::${AWS::AccountId}:role/aws-ec2-spot-fleet-tagging-role - LaunchTemplateConfigs: - - LaunchTemplateSpecification: - LaunchTemplateName: GameOfLifeLaunchTemplate - Version: !GetAtt GameOfLifeLaunchTemplate.LatestVersionNumber - Overrides: - - SubnetId: !Ref SubnetA - - LaunchTemplateSpecification: - LaunchTemplateName: GameOfLifeLaunchTemplate - Version: !GetAtt GameOfLifeLaunchTemplate.LatestVersionNumber - Overrides: - - SubnetId: !Ref SubnetB - - LaunchTemplateSpecification: - LaunchTemplateName: GameOfLifeLaunchTemplate - Version: !GetAtt GameOfLifeLaunchTemplate.LatestVersionNumber - Overrides: - - SubnetId: !Ref SubnetC - LoadBalancersConfig: - TargetGroupsConfig: - TargetGroups: - - Arn: !Ref GameOfLifeALBTargetGroup - ReplaceUnhealthyInstances: true - TargetCapacity: 2 - Type: maintain - -Outputs: - GameOfLifeDNSName: - Description: The DNS name of the Application Load Balancer that is used to gain access to the Game of Life testing environment. - Value: !GetAtt GameOfLifeALB.DNSName -``` \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/lab2/lamb.md b/content/amazon-ec2-spot-cicd-workshop/lab2/lamb.md deleted file mode 100644 index 3416ad00..00000000 --- a/content/amazon-ec2-spot-cicd-workshop/lab2/lamb.md +++ /dev/null @@ -1,110 +0,0 @@ -+++ -title = "Code snippet: The SpotCICDWorkshop_ManageTestEnvironment Lambda function" -weight = 20 -+++ -Below is a sanitized version of the SpotCICDWorkshop_ManageTestEnvironment Lambda function that was deployed to your account by the CloudFormation template that you deployed during the Workshop Preparation: -```javascript -'use strict'; -var AWS = require('aws-sdk'); -var actions = { - deploy: function (cfn, ddb, request_payload) { - return new Promise(function (resolve, reject) { - var cfn_params = { - StackName: request_payload.stackName, - Capabilities: [ 'CAPABILITY_IAM' ], - Parameters: [ - { - ParameterKey: 'KeyPair', - ParameterValue: '${SpotCICDWorkshop.KeyPair}' - }, { - ParameterKey: 'CurrentIP', - ParameterValue: '${SpotCICDWorkshop.Current.Ip}' - }, { - ParameterKey: 'AMILookupLambdaFunctionARN', - ParameterValue: '${SpotCICDWorkshop.AMILookupLambdaFunction.Arn}' - }, { - ParameterKey: 'DeploymentArtifactsS3Bucket', - ParameterValue: '${SpotCICDWorkshop.DeploymentArtifactsS3Bucket}' - }, { - ParameterKey: 'VPC', - ParameterValue: '${SpotCICDWorkshop.VPC}' - }, { - ParameterKey: 'SubnetA', - ParameterValue: '${SpotCICDWorkshop.SubnetPublicA}' - }, { - ParameterKey: 'SubnetB', - ParameterValue: '${SpotCICDWorkshop.SubnetPublicB}' - }, { - ParameterKey: 'SubnetC', - ParameterValue: '${SpotCICDWorkshop.SubnetPublicC}' - } - ], - RoleARN: '${SpotCICDWorkshop.IAMRoleTestEnvironmentCloudFormation.Arn}', - TemplateURL: 'https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop_game-of-life.yaml' - }; - cfn.createStack(cfn_params, function(err, cfn_data) { - if (err) { return reject(err); } - console.log('[INFO]', 'StackId: ', cfn_data.StackId); - return new Promise(function (resolve, reject) { - var ddb_params = { - Item: { - 'JobBaseName': { S: request_payload.jobBaseName }, - 'BuildID': { N: request_payload.buildId }, - 'CloudFormationStackID': { S: cfn_data.StackId } - }, - ReturnConsumedCapacity: 'TOTAL', - TableName: '${SpotCICDWorkshop.DynamoDBTestEnvironmentTable}' - }; - ddb.putItem(ddb_params, function(err, ddb_data) { - if (err) { return reject(err); } - console.log('[INFO]', 'Consumed Capacity Units: ', ddb_data.ConsumedCapacity.CapacityUnits); - return resolve(); - }); - }); - }); - }); - }, - terminate: function(cfn, ddb, request_payload) { - return new Promise(function (resolve, reject) { - var ddb_params = { - Key: { - 'JobBaseName': { S: request_payload.jobBaseName }, - 'BuildID': { N: request_payload.buildId } - }, - TableName: '${SpotCICDWorkshop.DynamoDBTestEnvironmentTable}' - }; - ddb.getItem(ddb_params, function(err, ddb_data) { - if (err) { return reject(err); } - console.log('[INFO]', 'CloudFormationStackId: ', ddb_data.Item.CloudFormationStackID.S); - return new Promise(function (resolve, reject) { - var cfn_params = { - StackName: request_payload.stackName, - RoleARN: '${SpotCICDWorkshop.IAMRoleTestEnvironmentCloudFormation.Arn}' - }; - cfn.deleteStack(cfn_params, function(err, cfn_data) { - if (err) { return reject(err); } - return resolve(); - }); - }); - }); - }); - } -}; -exports.handler = function (event, context, callback) { - var p = actions[event.action]; - if (!p) { - return callback('Unknown action'); - } - var msgAction = event.action.toUpperCase() + ' '; - var cfn = new AWS.CloudFormation(); - var ddb = new AWS.DynamoDB(); - console.log('[INFO]', 'Attempting', msgAction); - return p(cfn, ddb, event).then(function (data) { - return callback(null, data); - }).catch(function (err) { - console.log('[ERROR]', JSON.stringify(err)); - return callback(err); - }); -}; - -``` \ No newline at end of file diff --git a/content/amazon-ec2-spot-cicd-workshop/lab3.md b/content/amazon-ec2-spot-cicd-workshop/lab3.md deleted file mode 100644 index 1bab0f1a..00000000 --- a/content/amazon-ec2-spot-cicd-workshop/lab3.md +++ /dev/null @@ -1,89 +0,0 @@ -+++ -title = "Lab 3: Externalise state data to add resiliency to Jenkins" -weight = 40 -+++ -You're now using Spot instances for your code builds and for the environments that are built out for testing – but your Jenkins server is still using an on-demand instance. Jenkins itself does not natively support running in high-availability configurations because it persists all data on a local file system. If you can store this data durably somewhere else than on the local file system, you can move your Jenkins Master instance to a self-healing Spot instance. To provide persistence for this file system data, you’ll move your Jenkins data to an Elastic File System (EFS) volume and mount this volume on instance spawned by a Spot Fleet. - -## OBTAIN THE RELEVANT INFORMATION FOR CLOUDFORMATION FOR THIS LAB -As with the previous labs, the CloudFormation stack deployed during your Workshop Preparation has provisioned some of the resources required for this lab (in order to allow us to focus on the aspects of the workshop that directly apply to EC2 Spot). You will need to determine and make a note of what the **EFS Filesystem ID** is. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#)); -2. Click on the checkbox associated with the **SpotCICDWorkshop** stack; -3. From the Outputs tab of the SpotCICDWorkshop stack in the CloudFormation console, make note of values associated with the **EFSFileSystemID** key. -{{% /expand%}} - -## COPY THE CONTENTS OF JENKINS_HOME TO YOUR EFS FILE SYSTEM -In order to copy the contents of the JENKINS_HOME directory to the EFS file system (for which the ID of which you determined in the previous step), you'll need to first mount the file system on the EC2 instance currently running your Jenkins server. - -Once the file system has been mounted, stop the Jenkins service and set the file system permission of the root of your filesystem so that the jenkins user and group are owners. Finally, copy the contents of **/var/lib/jenkins** (this is the JENKINS_HOME directory) across to the root of your EFS file system. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and select the **Instances** option from the left pane (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#Instances:sort=instanceId)); -2. Select the instance with the Name tag of **Jenkins Master (On-demand)** and make a note of it's current IPv4 Pubic IP; -3. Establish an SSH session to this IP address (For instructions on how to establish an SSH connection to your EC2 instance, please refer to [this link](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html?icmpid=docs_ec2_console)) - you'll need to use the EC2 Key Pair that you generated during the Workshop Preparation to establish connectivity; -4. Mount the EFS file system that was created by the CloudFormation template at the /mnt mountpoint by entering the following command, replacing %FILE-SYSTEM-ID% with the EFSFileSystemID noted above: - - ```bash -sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 \ -$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)\ -.%FILE-SYSTEM-ID%.efs.eu-west-1.amazonaws.com:/ /mnt -``` -5. Stop the Jenkins service by entering the following command: - - ```bash -sudo service jenkins stop -``` - -6. The content of JENKINS_HOME is stored under /var/lib/jenkins – copy this content (whilst preserving permissions) to your EFS file system mount point (/mnt) using the following commands (note that this will take a couple of minutes - while it's progressing, commence the next section of this lab): - - ```bash -sudo chown jenkins:jenkins /mnt -sudo cp -rpv /var/lib/jenkins/* /mnt -``` -{{% /expand%}} - -## PROVISION AN EC2 SPOT FLEET FOR YOUR NEW JENKINS HOST -The Spot Fleet that you'll provision for your Jenkins server will be configured in a similar manner to what you did for the build agents - though this time you'll be a bit more aggressive with the bid price to ensure that you see an overall saving over what you would have spent on an on-demand t3.medium instance. Additionally, you'll configure this Spot Fleet so that the instances are associated with the Target Group used by the Application Load Balancer that you've been using to access Jenkins. - -1. Go to the **EC2** console and click on the **Spot Requests** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2sp/v1/spot/home?region=eu-west-1#)); -2. Click on the **Request Spot Instances** button; -3. At the first screen of the Spot instance launch wizard: - 1. Under the Tell us your application or task need heading, switch to the **Load balancing workloads** option; - 2. In the Configure your instances section, select the **JenkinsMasterLaunchTemplate** template from the Launch template dropdown. Change the Network to be the **Spot CICD Workshop VPC**. After making this selection, enable the check boxes for all three Availability Zones and then select the **Amazon EC2 Spot CICD Workshop Public Subnet** associated with each availability zone as the subnet to launch instances in; - 3. At the Tell us how much capacity you need section, keep the Total target capacity at **1** instance and the Optional On-Demand portion set to **0**, and then tick the **Maintain target capacity** checkbox. Once selected, leave the Interruption behavior set to **Terminate**; - 4. Again, you'll override the recommendations made by the console, so clear the tick from **Apply recommendations** checkbox. Click on the **Remove** links associated with the all of the instance types initially defined to remove them from the fleet configuration. Then click on the **Select instance types** button and add the **m3.large**, **m4.large**, **t2.medium** and **t3.medium** instance types to the fleet definition. Once the checkboxes for the required instance types have been ticked, click on the **Select** button. Once you have the four desired instance types listed in the fleet request, select the **Lowest Price** Fleet allocation strategy (since we’re interested in keeping cost to an absolute minimum for this use case, and it makes little sense to diversify a single instance across any number of instance pools); - 5. At the Additional request details section, remove the tick from the **Apply defaults** checkbox. As you want to keep the cost of running your Jenkins server below that for which you're currently paying, select the **Set your max price (per instance/hour)** option, and set the price to be the on-demand price of a t3.medium instance in the Ireland region, which is **$0.0456**. In order to ensure that the server can receive HTTP requests from the Application Load Balancer you've been using, tick the checkbox labelled **Receive traffic from one or more load balancers** and from the Target groups dropdown, select **JenkinsMasterEC2TargetGroup**. - 6. Review the Your fleet request as a glance section - it should indicate that your Fleet strength is strong as a result of being able to draw instances from 12 instance pools, and your Estimated price should indicate that you're expecting to make a 73% saving compared to the cost of equivalent on-demand resources; - 7. Lastly, click on the **Launch** button. - -## VERIFY THAT YOUR EC2 SPOT INSTANCE IS ATTACHED TO YOUR ALB TARGET GROUP -After a few moments, your Spot instance will start up and should attach itself the the Target Group being used by your Application Load Balancer. Determine if this registration has completed successfully and when it has done so, access Jenkins through your Load Balancer and fire off a build of Apache PDFBox to ensure that everything is still working as expected. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Target Groups** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#TargetGroups)); -2. Select the checkbox corresponding to the target group named **JenkinsMasterEC2TargetGroup**, then select the **Targets** tab. You should see your **Jenkins Master (On-demand)** instance with an unhealthy status (as a result of you stopping the Jenkins service on this host). Depending on how quickly you’ve got to this point, you might also see a second instance registered in the target group – the **Jenkins Master (Spot)** instance from the fleet that you just launched. If you don’t see it, it’s likely that your new spot instance isn't up and running yet – something that usually takes a minute or two to complete after placing your Spot Fleet request. Refresh the list of targets every minute or so until the **Jenkins Master (Spot)** instance appears in the list. While there are no healthy instances in the target group, your web browser should return a HTTP 502 error when you attempt to load Jenkins through your ALB; -3. When the **Jenkins Master (Spot)** instance is shown with a healthy status, you should then be able to reload Jenkins through the ALB's DNS name. Once you can do so, initiate a build of the **Apache PDFBox** project to ensure that everything is working as expected. -{{% /expand%}} - -## TEST THE SELF-HEALING ARCHITECTURE BY TERMINATING THE RUNNING EC2 SPOT INSTANCE -What happens when the market price for the capacity pool that your EC2 Spot instance is running in goes over your $0.0456 per instance-hour bid price? - -When this happens, your EC2 Spot instance will be provided with a two-minute notice of termination and after that time lapses, the instance will be terminated. At this point, Spot Fleet will observe that there are no running instances in the fleet and because the desired capacity is one, it will attempt to launch a replacement instance in a capacity pool that is still below the per instance-hour bid price that you set (this is why diversification across many capacity pool is a best practice). If such an instance is available, a replacement instance will be launched and bootstrapped in exactly the same manner as your original instance. - -Test this out by terminating the Spot instance with the **Jenkins Master (Spot)** name tag and verifying that a replacement instance comes up to take its place. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Instances** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#Instances)); -2. Search for the EC2 instance with the Name tag of **Jenkins Master (Spot)**, right-click on it and select the **Instance State** > **Terminate** option. At the confirmation pop-up, click on the **Yes, Terminate** option. Within moments of terminating this instance, your Spot Fleet registration should detect that the number of running instances is below the target that you’ve defined (that target being one instance) and therefore should launch a new, replacement spot instance. Refresh the list of EC2 instances every few seconds until you see the new instance be allocated (while you're waiting, it might also be interesting to see what's happening under the History tab of your Spot Fleet). Whilst your Jenkins service will be unavailable until this new spot instance launches and bootstraps, service should automatically be restored in a couple of minutes. -{{% /expand%}} - -## TERMINATE THE ON-DEMAND INSTANCE THAT WAS INITIALLY USED FOR YOUR JENKINS SERVER -Once you've verified that your Spot Fleet is self-healing, you no longer have any need for the On-demand instance. To prevent it from incurring unnecessary cost, it can be terminated. - -{{%expand "Click to reveal detailed instructions" %}} -1. Remain at the EC2 Instances screen and search for the EC2 instance with the Name tag of **Jenkins Master (On-demand)**. Right-click on the one instance that should come up and select the **Instance State** > **Terminate** option. At the confirmation pop-up, click on the **Yes, Terminate** option. -{{% /expand%}} - -## PROCEED TO LAB 4 -Once your on-demand Jenkins instance has been terminated, you may proceed with [Lab 4](/amazon-ec2-spot-cicd-workshop/lab4.html). diff --git a/content/amazon-ec2-spot-cicd-workshop/lab4.md b/content/amazon-ec2-spot-cicd-workshop/lab4.md deleted file mode 100644 index 32bccd8e..00000000 --- a/content/amazon-ec2-spot-cicd-workshop/lab4.md +++ /dev/null @@ -1,155 +0,0 @@ -+++ -title = "Lab 4: Using containers backed by Spot instance in Auto Scaling Groups" -weight = 50 -+++ -You’ve now got a scalable solution using nothing but Spot instances for your CICD systems, your build agents and your test environments – however, you still have some inefficiencies with this setup: - -* Your Jenkins master utilizes a relatively low percentage of the CPU resources on the instance types that Jenkins is running on; and -* You still have at least one Jenkins build agent running at all times; - -These inefficiencies can be addressed by moving your solution to a container environment that continues to utilize Spot instances. This lab will see you configure the ECS cluster resources that were created by the initial CloudFormation template and migrate your Jenkins master and agents to this cluster. - -## OBTAIN THE RELEVANT INFORMATION FOR CLOUDFORMATION FOR THIS LAB -As was the case with Lab 3, you will need a value from the Outputs tab of the CloudFormation stack that you deployed in the Workshop Preparation - this time, make a note of what the **JenkinsIAMRoleARN** is. -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **CloudFormation** console (or [click here](https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#)); -2. Click on the checkbox associated with the **SpotCICDWorkshop** stack; -3. From the Outputs tab of the SpotCICDWorkshop stack in the CloudFormation console, make note of value associated with the **JenkinsIAMUserARN** key. -{{% /expand%}} - -## INSPECT THE ECSLAUNCHTEMPLATE LAUNCH TEMPLATE -A couple of weeks prior to the 2018 re:Invent conference, AWS launched a new feature for Auto Scaling Groups - the capability to provision Auto Scaling Groups with a mixture of different instance types and different purchasing models. The ECS cluster that you provision in this lab utilises this feature, specifically launching an on-demand instance and a spot instance in the Auto Scaling Group that you create in the next section. - -By deploying the Auto Scaling Group in this manner, you can configure a placement constraint on the Jenkins server container in order to have it preferentially run on the on-demand instance and eliminating the drawbacks of having the container running on a spot instance. Setting up a placement constraint in an ECS Task Definition is relatively straightforward, but having the EC2 instance configure a custom attribute within ECS to indicate whether it is a spot instance or not is a little more complex. By inspecting the **ECSLaunchTemplate** Launch Template, you can see how this is achieved. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and select the **Launch Templates** option from the left pane (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#LaunchTemplates:sort=launchTemplateId)); -2. Select the launch template with the Launch Template Name of **ECSLaunchTemplate**. Toward the bottom of the lower pane, expand the **Advanced details** section and click on the **View user data** link; -3. Inspect the User Data defined in the Launch Template: - 1. First, the Instance ID needs is obtained by polling the instance metadata; - 2. Once the Instance ID has been obtained, the user data script issues an EC2 describe-instances call to obtain the Instance Lifecycle. If the instance that this call refers to is a Spot instance, the Instance Lifecycle will be **spot**, otherwise it will be **null**; - 3. Finally, the ECS\_INSTANCE\_ATTRIBUTES configuration directive is added to the /etc/ecs/ecs.config file, defining the **lifecycle** custom attribute. When the user data has been processed, the ECS agent will start and register this custom variable with the ECS service. -{{% /expand%}} - -## PROVISION AN AUTO SCALING GROUP FOR YOUR ECS CLUSTER -While the CloudFormation template that was deployed during the Workshop Preparation has already defined the ECS Cluster resources that you'll utilize in this workshop, you'll need to create a new Auto Scaling Group to provide the compute resources to the ECS Cluster definition. While you could provision the entire Auto Scaling Group to use EC2 Spot Instances, this walkthrough will guide you through how to configure the Auto Scaling Group to use multiple instance types and purchasing options - a feature that was launched just over a week prior to re:Invent 2018. - -1. Go to the **EC2** console and click on the **Auto Scaling groups** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/autoscaling/home?region=eu-west-1#AutoScalingGroups:)); -2. Click on the **Create Auto Scaling group** button; -3. The recently released capability to deploy EC2 instances using multiple instance types and purchase options requires the use of **Launch Templates**, so select this as the Auto Scaling Group parameters that you want to use. Select the **ECSLaunchTemplate** launch template (which was created by the CloudFormation template deployed during the Workshop Preparation), and then click on the **Next Step** button. -4. At the Configure Auto Scaling group details step of the wizard: - 1. Give the Group a name of **Spot CICD Workshop ECS Auto Scaling Group**; - 2. Change the Fleet Composition to **Combine purchase options and instances** - Selecting this option should reveal additional choices. The Launch Template selected previously does not specify what purchasing model should be used for these instances and the default option will see on-demand instances launched; - 3. At the Instance Types section, add **t3.large**, **t2.medium** and **t2.large** instance types to the Auto Scaling Group configuration; - 4. Remove the tick from the **Instances Distribution** checkbox so that you can explore the additional configuration options; - 5. As you don't have a specific price objective to meet for your ECS cluster, leave the Maximum Spot Price using default bidding. Also leave the Spot Allocation Strategy so that instances are diversified across the 2 lowest priced instances types per Availability Zone (this will typically ensure that the medium-sized instances will almost always be used when instances are launched, but the large instances are able to be used if something unexpected happens to the availability or market price of the medium instances); - 6. While you could run the entire Auto Scaling Group using Spot instances, it might be more desirable to have a portion of the Auto Scaling Group running on-demand instances, to provide a little more assurance that the container running your Jenkins server will always be running. In order to do this, set the Optional On-Demand Base such that you designate the first **1** instance as On-Demand, and then change the On-Demand Percentage Above Base to be **0%** On-Demand and 100% Spot; - 7. Change the Group size to start with **2** instances; - 8. Select the **Amazon EC2 Spot CICD Workshop VPC** from the Network dropdown; - 9. Within the Subnet field, click on and select each of the three subnets that start with **Amazon EC2 Spot CICD Workshop Public Subnet**; - 10. Click on the **Next: Configure scaling policies** button; -5. At the Configure scaling policies step of the wizard, ensure that the **Keep this group at its initial size** option. In a production environment, it's likely that you will want to configure scaling policies to ensure that your compute resources grow in line with the number of containers deployed to the cluster, but a group with no scaling policy should be sufficient for this workshop. Click on the **Next: Configure Notifications** button; -6. You will not be setting up any notifications at the Configure Notifications step of the wizard, so simply click on the **Next: Configure Tags* button; -7. At the Configure Tags step of the wizard, add a tag with a Key of **Name** and a Value of **Spot CICD Workshop ECS Instance**, then click on the **Review** button; -8. At the Review step of the wizard, click on the **Create Auto Scaling Group** button. Once the Auto Scaling Group has been launched, click on the **Close** button. - -## MODIFY PERMISSIONS ON THE EFS FILE SYSTEM -You will be deploying the official Jenkins docker image to your ECS cluster which differs slightly from the RPM bundles typically deployed to Amazon Linux systems. One of the differences is that the docker image will ALWAYS run Jenkins using a Linux uid of 1000, as opposed to the incremental uid used in RPM deployments. Because this uid differs, we'll need to ensure that the EFS file system is configured with the correct permissions to allow our ECS Jenkins Master to read the persisted data. - -To do this, stop the Jenkins service on the Spot instance that you crated in the previous lab and set the file system permission of the root of your EFS file system so that the user with uid of 1000, and the group with a gid of 1000 is the owner of all file system objects. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and select the **Instances** option from the left pane (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#Instances:sort=instanceId)); -2. Select the instance with the Name tag of **Jenkins Master (Spot)* and make a note of it's current IPv4 Pubic IP. -3. Establish an SSH session to this IP address (For instructions on how to establish an SSH connection to your EC2 instance, please refer to [this link]( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html?icmpid=docs_ec2_console)) - you'll need to use the EC2 Key Pair that you generated during the Workshop Preparation to establish connectivity. -4. Stop the Jenkins service by entering the following command: - - ```bash -sudo service jenkins stop -``` -5. The content of JENKINS_HOME is stored under /var/lib/jenkins – execute the following command to set the permissions appropriately (note that this will take a couple of minutes - while it's progressing, commence the next section of this lab): - - ```bash -sudo chown -R 1000:1000 /var/lib/jenkins -``` -{{% /expand%}} - -## MODIFY THE JENKINS MASTER APPLICATION LOAD BALANCER SO THAT REQUESTS ARE FORWARDED TO THE ECS TARGET GROUP -In order to operate properly with ECS Service Discovery (which is used later in this lab), the CloudFormation template that you deployed in the Workshop Preparation deployed two Application Load Balancer Target Groups - one for instance targets (EC2) and the other for IP address targets (ECS). Given that the Jenkins installation on your Spot instance is now stopped, now is an opportune time to change the configuration of the listener for the JenkinsMasterALB Application Load Balancer to use the **JenkinsMasterECSTargetGroup** target group. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and select the **Load Balancers** option from the left pane (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#LoadBalancers:sort=loadBalancerName)); -2. If it is not selected, select the load balancer named **JenkinsMasterALB** and click on the **Listeners** tab in the bottom pane. Then, click on the **View/edit rules** link; -3. Click on the **Reorder Rules** button (signified by an up arrow next to a down arrow), select rule **2** and click on the **Up** button to increase its priority. The rule that forwards content to the **JenkinsMasterECSTargetGroup** should now be at the top of the list - if this is the case, click on the **Save** button. -{{% /expand%}} - -## MODIFY THE JENKINSMASTER ECS SERVICE TO START A JENKINS MASTER TASK -An ECS Service called **SpotCICDWorkshop-ECSServiceJenkinsMaster** was provisioned the CloudFormation stack launched during the Workshop Preparation phase, along with the EFS volume that you're using to persist the contents of JENKINS\_HOME. Up until Lab 3, the contents of this EFS volume was empty, which would have prevented a Jenkins Task from starting successfully until the contents of JENKINS\_HOME resided on the EFS volume, and no other Jenkins installation had a lock file in place on the volume. Therefore during the CloudFormation deployment, the ECS Service definition was configured to have the desired number of tasks running set to **0**. Update the service so that **1** copy of the task is running. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **ECS** console (or [click here](https://eu-west-1.console.aws.amazon.com/ecs/home?region=eu-west-1#/clusters)) and click on the **SpotCICDWorkshopECSCluster** link; -2. Click on the checkbox of the service with a Service Name that begins with **SpotCICDWorkshop-ECSServiceJenkinsMaster**, then click on the **Update** button; -3. Change the Number of tasks from **0** to **1**, then click on the **Next Step** button; -4. At the Configure network screen, click on the **Next Step** button. Repeat the same action at the Set Auto Scaling (optional) screen. Finally at the Review screen, click on the **Update Service** button. Once the service has been updated, click on the **View Service** button. -{{% /expand%}} - -## VERIFY THE STATUS OF YOUR ALB TARGET GROUP, VERIFY ACCESS TO JENKINS AND RUN A TEST BUILD -Within a few moments of updating the service definition, a new task and container will be launched on one of the ECS cluster nodes (specifically, the on-demand instance within the cluster). The ECS service is configured to attach the running container to the JenkinsMasterECSTargetGroup target group, which is now the target group being used by the Application Load Balancer that you've been using to access Jenkins. Verify that the new container has registered with the ALB, then access Jenkins through the ALB and run a test build of the Apache PDFBox project. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Target Groups** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#TargetGroups)); -2. Select the checkbox corresponding to the target group named **JenkinsMasterECSTargetGroup**, then select the **Targets** tab. You should see one of your Spot CICD Workshop ECS Instance instances with an **initial** status (it generally takes a couple of minutes for Jenkins to completely start up and for the health check to return a **healthy** status); -3. Refresh the list of targets every minute or so and when the registered target is shown with a healthy status, you should then be able to reload Jenkins through the ALB. Once you can do so, initiate a build of the **Apache PDFBox** project to ensure that everything is working as expected. -{{% /expand%}} - -## CANCEL THE EC2 SPOT FLEET REQUEST FOR YOUR SELF-HEALING JENKINS SERVER -As you now have Jenkins running from a container, you no longer need the Spot Fleet that you created in Lab 3 - cancel this Spot Fleet request and terminate the instances belonging to the fleet. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Spot Requests** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2sp/v1/spot/home?region=eu-west-1#)); -2. Select the checkbox corresponding to the Spot Fleet request that was created in the previous lab (this should be the Spot Fleet request that has a Capacity of **1 of 1** and a Max price of **$0.0456**); -3. Click the Actions dropdown at the top of the screen and select the **Cancel spot request** option. At the confirmation dialogue, ensure that the **Terminate instances** checkbox is selected and click on the **Confirm** button. -{{% /expand%}} - -## RECONFIGURE JENKINS TO USE BUILD AGENTS RUNNING IN CONTAINERS -When you launched build agents using Spot instances, you took advantage of the fact that you could access those agents via SSH. However with build agent containers, you won't be exposing an SSH interface outside of your container. Instead, your build agent containers will initiate a connection to the Jenkins container via the Java Network Launch Protocol (JNLP), via an interface that is addressed using ECS Service Discovery. - -1. Back at the Jenkins home page, click on the **Manage Jenkins** link on the left side menu again, and then the **Configure System** link; -2. Scroll down to the Cloud section and under the Spot Fleet Configuration section, click on the **Delete Cloud** button (the spot build agents will no longer be required now that you’re starting to use resources on your ECS cluster); -3. Next, click on the **Add a new cloud** button followed by the **Amazon EC2 Container Service Cloud** option; -4. In the Amazon EC2 Container Service Cloud section: - 1. Set **SpotCICDWorkshopECSAgents** as the Name; - 2. In the Amazon ECS Credentials dropdown, select the same IAM Access key that you used when configuring your spot build agents; - 3. Select the **eu-west-1** region from the Amazon ECS Region Name dropdown; - 4. Select the ECS Cluster that has the **SpotCICDWorkshopECSCluster** suffix from the ECS Cluster dropdown; - 5. Click on the **Advanced** button, then set the Alternative Jenkins URL to **http://master.jenkins.local:8080** - this is the address (managed through ECS Service Discovery) that the build agents will use to access Jenkins via the internal VPC network; - 6. Click on the **Add** button that appears just to the right of the ECS slave templates label; - 7. Enter **ecs-agents** into the Label field; - 8. Type **ECSBuildAgent** into the Template Name field; - 9. Enter **cloudbees/jnlp-slave-with-java-build-tools** into the Docker Image field - this is the Docker image that will be used when launching your build agent containers; - 10. Set the Hard Memory Reservation to **1536** and the CPU units to **512*; - 11. Click on the **Advanced** button; - 12. In the Task Role ARN field, enter the value specified by the **JenkinsIAMRoleARN** key obtained at the beginning of this lab (it should be of the format arn:aws:iam::account\_number:role/SpotCICDWorkshop-IAMRoleJenkins-random_chars). -5. Click on the **Save** button at the bottom of the page. - -## CANCEL THE EC2 SPOT FLEET REQUEST FOR YOUR BUILD AGENTS -As you now have build agents configured to run from containers, you no longer need the Spot Fleet that you created in Lab 1 - cancel this Spot Fleet request and terminate the instances belonging to the fleet. - -{{%expand "Click to reveal detailed instructions" %}} -1. Go to the **EC2** console and click on the **Spot Requests** option from the left frame (or [click here](https://eu-west-1.console.aws.amazon.com/ec2sp/v1/spot/home?region=eu-west-1#)); -2. Select the checkbox corresponding to what should be the sole remaining Spot Fleet request; -3. Click the Actions dropdown at the top of the screen and select the **Cancel spot request** option. At the confirmation dialogue, ensure that the **Terminate instances** checkbox is selected and click on the **Confirm** button. -{{% /expand%}} - -## RECONFIGURE THE APACHE PDFBOX BUILD PROJECT TO ONLY USE YOUR ECS BUILD AGENTS, AND RUN A TEST BUILD -As things stand now, your projects in Jenkins won't be able to be built - you've removed all of the EC2 Spot build agents and have statically defined a number of build jobs to use these agents. Recall that when you set up the EC2 Spot build agents, you assigned the **spot-agents** label to your a number of the build projects to force them to be built on your Spot instances? For this lab, reconfigure the **Apache PDFBox** build project to use the **ecs-agents** label and then kick off a test build of this project. - -{{%expand "Click to reveal detailed instructions" %}} -1. Within Jenkins, click on the Jenkins logo in the top left corner of the site – this will take you back to the Jenkins home page. Click on the **Apache PDFBox** build project link, then click on the **Configure** link in the left pane of the page; -2. Under the General section, change the Label Expression so that it says **ecs-agents**; -3. At the bottom of the screen, click on the **Save** button; -4. Finally, click on the **Build Now** link on the left side of the screen to initiate a build of this project. Click on the Jenkins logo at the top left corner of the site to go back to the main screen and see how your build progresses on the ECS build agent. One final thing to note is that when running in a container, the build agent will always show up with a suspended status - due to the ephemeral way in which containers are treated by the plugin, as soon as all of the build executors have been consumed in the container, the plugin will mark the agent as suspended so that new agents are deployed in new containers for subsequent build jobs. -{{% /expand%}} - -## PROCEED TO WORKSHOP CLEANUP -Once your Jenkins infrastructure is completely running in your ECS cluster, you've completed all of the labs in this workshop. Congratulations! You may now proceed with the [Workshop Cleanup](/amazon-ec2-spot-cicd-workshop/clea.html). diff --git a/static/images/amazon-ec2-spot-cicd-workshop/spotinterruptionlog.png b/static/images/amazon-ec2-spot-cicd-workshop/spotinterruptionlog.png new file mode 100644 index 00000000..12f01ef1 Binary files /dev/null and b/static/images/amazon-ec2-spot-cicd-workshop/spotinterruptionlog.png differ diff --git a/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-asg.yaml b/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-asg.yaml new file mode 100644 index 00000000..089150db --- /dev/null +++ b/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-asg.yaml @@ -0,0 +1,629 @@ +AWSTemplateFormatVersion: "2010-09-09" + +Description: A CloudFormation template that will deploy all AWS resources that are required to run the Amazon EC2 Spot CI/CD Workshop. This template is provided as-is under a modified MIT license - please see https://github.com/aws-samples/amazon-ec2-spot-cicd-workshop/blob/master/LICENSE + +Parameters: + KeyPair: + Description: The Key Pair created earlier in the Preparation Lab + Type: AWS::EC2::KeyPair::KeyName + + CurrentIP: + Description: Your current IP address (used to limit access to SSH services on EC2 instances) + Type: String + AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\/([0-9]|[1-2][0-9]|3[0-2])$ + ConstraintDescription: must be specified in CIDR notation (e.g, 123.45.67.0/24) + Default: 0.0.0.0/0 + + JenkinsAdminPassword: + Description: The password that you would like to use for the admin account in your Jenkins server (must be at least 8 characters) + Type: String + ConstraintDescription: must be at least 8 characters in length + MinLength: 8 + NoEcho: true + + AmazonLinux2LatestAmiId: + Type: AWS::SSM::Parameter::Value + Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 + +Resources: + IAMRoleJenkins: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: 2012-10-17 + Statement: + - Effect: Allow + Principal: + Service: + - ec2.amazonaws.com + Action: + - sts:AssumeRole + Path: / + Policies: + - PolicyName: JenkinsPolicy + PolicyDocument: + Version: 2012-10-17 + Statement: + - Effect: Allow + Action: + - ec2:DescribeSpotFleetInstances + - ec2:ModifySpotFleetRequest + - ec2:CreateTags + - ec2:DescribeRegions + - ec2:DescribeInstances + - ec2:TerminateInstances + - ec2:DescribeInstanceStatus + - ec2:DescribeSpotFleetRequests + Resource: "*" + - Effect: Allow + Action: + - autoscaling:DescribeAutoScalingGroups + - autoscaling:UpdateAutoScalingGroup + Resource: "*" + - Effect: Allow + Action: + - iam:ListInstanceProfiles + - iam:ListRoles + - iam:PassRole + Resource: "*" + + InstanceProfileJenkins: + Type: AWS::IAM::InstanceProfile + DependsOn: IAMRoleJenkins + Properties: + Path: "/" + Roles: + - !Ref IAMRoleJenkins + + VPC: # This is the VPC that the CI/CD environment will be running in + Type: AWS::EC2::VPC + Properties: + CidrBlock: 192.168.0.0/21 + EnableDnsHostnames: "true" + EnableDnsSupport: "true" + InstanceTenancy: default + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop VPC + + SubnetPublicA: # The first of three subnets defined within the VPC... + Type: AWS::EC2::Subnet + DependsOn: VPC + # DependedOn: SubnetPublicARouteTableAssociation + Properties: + AvailabilityZone: !Select [0, !GetAZs ""] + CidrBlock: 192.168.0.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Subnet A + + SubnetPublicB: # ... and the second of three subnets... + Type: AWS::EC2::Subnet + DependsOn: VPC + # DependedOn: SubnetPublicBRouteTableAssociation + Properties: + AvailabilityZone: !Select [1, !GetAZs ""] + CidrBlock: 192.168.1.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Subnet B + + SubnetPublicC: # ... and the third of three subnets defined within the VPC + Type: AWS::EC2::Subnet + DependsOn: VPC + # DependedOn: SubnetPublicCRouteTableAssociation + Properties: + AvailabilityZone: !Select [2, !GetAZs ""] + CidrBlock: 192.168.2.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Subnet C + + SubnetPrivateA: # The first of three subnets defined within the VPC... + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [0, !GetAZs ""] + CidrBlock: 192.168.3.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Subnet A + - Key: Type + Value: Private + + SubnetPrivateB: # ... and the second of three subnets... + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [1, !GetAZs ""] + CidrBlock: 192.168.4.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Subnet B + - Key: Type + Value: Private + + SubnetPrivateC: # ... and the third of three subnets defined within the VPC + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [2, !GetAZs ""] + CidrBlock: 192.168.5.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Subnet C + - Key: Type + Value: Private + + InternetGateway: # Create an Internet Gateway in order to allow EC2 instances to be accessible via the Internet + Type: AWS::EC2::InternetGateway + Properties: + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Internet Gateway + + InternetGatewayAttachment: # Once both the VPC and Internet Gateway have been created, attach the Internet Gateway to the VPC + Type: AWS::EC2::VPCGatewayAttachment + DependsOn: + - InternetGateway + - VPC + Properties: + InternetGatewayId: !Ref InternetGateway + VpcId: !Ref VPC + + EIPNATGateway: + Type: AWS::EC2::EIP + DependsOn: + - InternetGatewayAttachment + - VPC + Properties: + Domain: vpc + + NATGateway: + Type: AWS::EC2::NatGateway + DependsOn: + - SubnetPublicA + Properties: + AllocationId: !GetAtt EIPNATGateway.AllocationId + SubnetId: !Ref SubnetPublicA + + RouteTablePublic: # Create a route table which will be used within the VPC + Type: AWS::EC2::RouteTable + DependsOn: VPC + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Route Table + + RouteTablePrivate: # Create a route table which will be used within the VPC + Type: AWS::EC2::RouteTable + DependsOn: VPC + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Route Table + + DefaultRoutePublic: # Add a default route to the route table, pointing to the Internet Gateway + Type: AWS::EC2::Route + DependsOn: + - RouteTablePublic + - InternetGatewayAttachment + Properties: + DestinationCidrBlock: 0.0.0.0/0 + GatewayId: !Ref InternetGateway + RouteTableId: !Ref RouteTablePublic + + DefaultRoutePrivate: # Add a default route to the route table, pointing to the NAT Gateway + Type: AWS::EC2::Route + DependsOn: + - RouteTablePrivate + - NATGateway + Properties: + DestinationCidrBlock: 0.0.0.0/0 + NatGatewayId: !Ref NATGateway + RouteTableId: !Ref RouteTablePrivate + + SubnetPublicARouteTableAssociation: # Associate the route table with the first of the three subnets in the VPC... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePublic + - SubnetPublicA + Properties: + RouteTableId: !Ref RouteTablePublic + SubnetId: !Ref SubnetPublicA + + SubnetPublicBRouteTableAssociation: # ... and the second of three subnets... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePublic + - SubnetPublicB + Properties: + RouteTableId: !Ref RouteTablePublic + SubnetId: !Ref SubnetPublicB + + SubnetPublicCRouteTableAssociation: # ... and the third of the three subnets in the VPC + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePublic + - SubnetPublicC + Properties: + RouteTableId: !Ref RouteTablePublic + SubnetId: !Ref SubnetPublicC + + SubnetPrivateARouteTableAssociation: # Associate the route table with the first of the three subnets in the VPC... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePrivate + - SubnetPrivateA + Properties: + RouteTableId: !Ref RouteTablePrivate + SubnetId: !Ref SubnetPrivateA + + SubnetPrivateBRouteTableAssociation: # ... and the second of three subnets... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePrivate + - SubnetPrivateB + Properties: + RouteTableId: !Ref RouteTablePrivate + SubnetId: !Ref SubnetPrivateB + + SubnetPrivateCRouteTableAssociation: # ... and the third of the three subnets in the VPC + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePrivate + - SubnetPrivateC + Properties: + RouteTableId: !Ref RouteTablePrivate + SubnetId: !Ref SubnetPrivateC + + SecurityGroupJenkins: # A Security Group that allows ingress access for SSH and the default port that a Jenkins Master will run on + Type: AWS::EC2::SecurityGroup + DependsOn: VPC + Properties: + GroupName: Spot CICD Workshop Jenkins Security Group + GroupDescription: A Security Group that allows ingress access for SSH and the default port that a Jenkins Master will run on + SecurityGroupIngress: + - Description: SSH access from the IP address that the workshop participant is using. + IpProtocol: tcp + FromPort: 22 + ToPort: 22 + CidrIp: !Ref CurrentIP + - Description: SSH access from hosts within the Spot CICD Workshop VPC - required for Build Agent deployments on to EC2 Spot instances. + IpProtocol: tcp + FromPort: 22 + ToPort: 22 + CidrIp: 192.168.0.0/21 + - Description: HTTP (8080) access from the IP address that the workshop participant is using. + IpProtocol: tcp + FromPort: 8080 + ToPort: 8080 + CidrIp: !Ref CurrentIP + - Description: HTTP (8080) access from hosts within the Spot CICD Workshop VPC, including the ALBs that reside in front of Jenkins, and Build Agents running on EC2 nodes. + IpProtocol: tcp + FromPort: 8080 + ToPort: 8080 + CidrIp: 192.168.0.0/21 + - Description: JNLP (5000) access from hosts within the Spot CICD Workshop VPC - required for Build Agent deployments on EC2 nodes to communicate back with the Jenkins Master container. + IpProtocol: tcp + FromPort: 5000 + ToPort: 5000 + CidrIp: 0.0.0.0/0 + - Description: JNLP (50000) access from hosts within the Spot CICD Workshop VPC - the legacy port that used to be required for Build Agent deployments on EC2 nodes to communicate back with the Jenkins Master container. + IpProtocol: tcp + FromPort: 50000 + ToPort: 50000 + CidrIp: 0.0.0.0/0 + VpcId: !Ref VPC + + SecurityGroupJenkinsALB: # A Security Group that allows ingress access for HTTP on ALBs and used to access the Jenkins Master + Type: AWS::EC2::SecurityGroup + DependsOn: VPC + Properties: + GroupName: Spot CICD Workshop Jenkins ALB Security Group + GroupDescription: A Security Group that allows ingress access for HTTP on ALBs and used to access the Jenkins Master + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 80 + ToPort: 80 + CidrIp: 0.0.0.0/0 + VpcId: !Ref VPC + + SecurityGroupEFS: + Type: AWS::EC2::SecurityGroup + DependsOn: + - SecurityGroupJenkins + - VPC + Properties: + GroupName: Spot CICD Workshop EFS Security Group + GroupDescription: A Security Group that allows access to EFS volume targets from the Jenkins Securiy Group + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 2049 + ToPort: 2049 + SourceSecurityGroupId: !Ref SecurityGroupJenkins + VpcId: !Ref VPC + + JenkinsMasterGroup: + Type: AWS::AutoScaling::AutoScalingGroup + DependsOn: + - InstanceProfileJenkins + - SecurityGroupJenkins + - SubnetPublicA + - SubnetPublicB + - SubnetPublicC + Properties: + HealthCheckType: EC2 + HealthCheckGracePeriod: 300 + MinSize: "1" + MaxSize: "1" + DesiredCapacity: "1" + VPCZoneIdentifier: + - Ref: SubnetPublicA + - Ref: SubnetPublicB + - Ref: SubnetPublicC + MixedInstancesPolicy: + InstancesDistribution: + OnDemandAllocationStrategy: lowest-price + OnDemandBaseCapacity: 0 + OnDemandPercentageAboveBaseCapacity: 100 + LaunchTemplate: + LaunchTemplateSpecification: + LaunchTemplateId: + Ref: JenkinsMasterLaunchTemplate + Version: "1" + TargetGroupARNs: + - !Ref JenkinsMasterALBTargetGroupEC2 + Tags: + - Key: Name + Value: jenkins-master + PropagateAtLaunch: true + + JenkinsMasterALB: # This is the Application Load Balancer that resides in front of your Jenkins Master instance and is responsible for port-mapping requests from TCP:80 to TCP:8080 + Type: AWS::ElasticLoadBalancingV2::LoadBalancer + DependsOn: + - SecurityGroupJenkinsALB + - SubnetPublicA + - SubnetPublicB + - SubnetPublicC + Properties: + Name: JenkinsMasterALB + Scheme: internet-facing + SecurityGroups: + - !Ref SecurityGroupJenkinsALB + Subnets: + - !Ref SubnetPublicA + - !Ref SubnetPublicB + - !Ref SubnetPublicC + + JenkinsMasterALBTargetGroupEC2: # This is the Target Group used by the JenkinsMasterALB load balancer when Jenkins is running on an EC2 instance + Type: AWS::ElasticLoadBalancingV2::TargetGroup + DependsOn: + - VPC + Properties: + HealthCheckIntervalSeconds: 15 + HealthCheckPath: /login + HealthCheckPort: 8080 + HealthCheckProtocol: HTTP + HealthCheckTimeoutSeconds: 5 + HealthyThresholdCount: 2 + Matcher: + HttpCode: 200 + Name: JenkinsMasterEC2TargetGroup + Port: 8080 + Protocol: HTTP + TargetType: instance + UnhealthyThresholdCount: 4 + VpcId: !Ref VPC + + JenkinsMasterALBListener: # This is the ALB Listener used to access the Jenkins Master + Type: AWS::ElasticLoadBalancingV2::Listener + DependsOn: + - JenkinsMasterALB + - JenkinsMasterALBTargetGroupEC2 + Properties: + DefaultActions: + - Type: forward + TargetGroupArn: !Ref JenkinsMasterALBTargetGroupEC2 + LoadBalancerArn: !Ref JenkinsMasterALB + Port: 80 + Protocol: HTTP + + JenkinsMasterALBListenerRuleEC2: # The ALB Listener rule that forwards all traffic destined for the Jenkins Master to the appropriate Target Group + Type: AWS::ElasticLoadBalancingV2::ListenerRule + DependsOn: + - JenkinsMasterALBListener + - JenkinsMasterALBTargetGroupEC2 + Properties: + Actions: + - Type: forward + TargetGroupArn: !Ref JenkinsMasterALBTargetGroupEC2 + Conditions: + - Field: path-pattern + Values: + - "/*" + ListenerArn: !Ref JenkinsMasterALBListener + Priority: 1 + + JenkinsMasterLaunchTemplate: # This is a launch template that will be used to provision Jenkins Master servers - showing how when used in conjunction with an EFS volume stateful applications can run on self-healing spot architectures. + Type: AWS::EC2::LaunchTemplate + DependsOn: + - EFSJenkinsHomeVolume + - InstanceProfileJenkins + - SecurityGroupJenkins + Properties: + LaunchTemplateName: JenkinsMasterLaunchTemplate + LaunchTemplateData: + BlockDeviceMappings: + - DeviceName: "/dev/xvda" + Ebs: + DeleteOnTermination: "true" + VolumeSize: 50 + VolumeType: gp2 + IamInstanceProfile: + Name: !Ref InstanceProfileJenkins + ImageId: !Ref AmazonLinux2LatestAmiId + InstanceType: t3.medium + KeyName: !Ref KeyPair + SecurityGroupIds: + - !Ref SecurityGroupJenkins + TagSpecifications: + - ResourceType: instance + Tags: + - Key: Name + Value: jenkins-master + UserData: + Fn::Base64: !Sub | + #!/bin/bash + # Install all pending updates to the system + yum -y update + # Install Docker + amazon-linux-extras install docker + chkconfig docker on + service docker start + usermod -a -G docker ec2-user + # Install Docker Compose + curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose + chmod +x /usr/local/bin/docker-compose + # Mount the Jenkins EFS volume + mkdir -p /jenkins_home + mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).${EFSJenkinsHomeVolume}.efs.eu-west-1.amazonaws.com:/ /jenkins_home + # Download files from GitHub + mkdir -p /usr/share/jenkins + cd /usr/share/jenkins + wget https://raw.githubusercontent.com/christianhxc/ec2-spot-workshops/cicd-upgrade/workshops/amazon-ec2-spot-cicd-workshop/container/casc.yaml + wget https://raw.githubusercontent.com/christianhxc/ec2-spot-workshops/cicd-upgrade/workshops/amazon-ec2-spot-cicd-workshop/container/docker-compose.yml + wget https://raw.githubusercontent.com/christianhxc/ec2-spot-workshops/cicd-upgrade/workshops/amazon-ec2-spot-cicd-workshop/container/Dockerfile + wget https://raw.githubusercontent.com/christianhxc/ec2-spot-workshops/cicd-upgrade/workshops/amazon-ec2-spot-cicd-workshop/container/plugins.txt + # Reset the password defined as the JenkinsAdminPassword + sed -i 's/JenkinsAdminPassword/${JenkinsAdminPassword}/' docker-compose.yml + # Configure the Jenkins Location + curl -s http://169.254.169.254/latest/meta-data/public-ipv4 | xargs -I {} sed -i 's/localhost/{}/' casc.yaml + # Start Jenkins + docker-compose pull + docker-compose up -d --no-build + + JenkinsSpotAgentLaunchTemplate: # This is a launch template that will be used to provision Jenkins build agents - showing how spot instances can be used to scale-out build jobs at low cost. + Type: AWS::EC2::LaunchTemplate + DependsOn: + - InstanceProfileJenkins + - SecurityGroupJenkins + Properties: + LaunchTemplateName: JenkinsBuildAgentLaunchTemplate + LaunchTemplateData: + BlockDeviceMappings: + - DeviceName: "/dev/xvda" + Ebs: + DeleteOnTermination: "true" + VolumeSize: 50 + VolumeType: gp2 + IamInstanceProfile: + Name: !Ref InstanceProfileJenkins + ImageId: !Ref AmazonLinux2LatestAmiId + InstanceType: t3.small + KeyName: !Ref KeyPair + SecurityGroupIds: + - !Ref SecurityGroupJenkins + TagSpecifications: + - ResourceType: instance + Tags: + - Key: Name + Value: jenkins-build-agent + UserData: + Fn::Base64: !Sub | + #!/bin/bash + # Install all pending updates to the system + yum -y update + # Configure YUM to be able to access contributed Maven RPM packages + wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo + # Update the release version in the Maven repository configuration for this mainline release of Amazon Linux + sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo + # Install Git + yum install git apache-maven -y + # Install the Java 11 SDK, Git and Maven + amazon-linux-extras install java-openjdk11 + + EFSJenkinsHomeVolume: + Type: AWS::EFS::FileSystem + Properties: + PerformanceMode: generalPurpose + + EFSMountTargetJenkinsHomeVolumeA: + Type: AWS::EFS::MountTarget + DependsOn: + - EFSJenkinsHomeVolume + - SecurityGroupEFS + - SubnetPublicA + Properties: + FileSystemId: !Ref EFSJenkinsHomeVolume + SecurityGroups: + - !Ref SecurityGroupEFS + SubnetId: !Ref SubnetPublicA + + EFSMountTargetJenkinsHomeVolumeB: + Type: AWS::EFS::MountTarget + DependsOn: + - EFSJenkinsHomeVolume + - SecurityGroupEFS + - SubnetPublicB + Properties: + FileSystemId: !Ref EFSJenkinsHomeVolume + SecurityGroups: + - !Ref SecurityGroupEFS + SubnetId: !Ref SubnetPublicB + + EFSMountTargetJenkinsHomeVolumeC: + Type: AWS::EFS::MountTarget + DependsOn: + - EFSJenkinsHomeVolume + - SecurityGroupEFS + - SubnetPublicC + Properties: + FileSystemId: !Ref EFSJenkinsHomeVolume + SecurityGroups: + - !Ref SecurityGroupEFS + SubnetId: !Ref SubnetPublicC + +Outputs: + EFSFileSystemID: + Description: The file system ID of the EFS volume that is used to persist JENKINS_HOME across EC2. + Value: !Ref EFSJenkinsHomeVolume + + JenkinsDNSName: + Description: The DNS name of the Application Load Balancer that is used to gain access to your Jenkins server. + Value: !GetAtt JenkinsMasterALB.DNSName + + JenkinsIAMRoleARN: + Description: The ARN associated with the IAM Role that was created for use by Jenkins. + Value: !GetAtt IAMRoleJenkins.Arn + + JenkinsVPCPrivateSubnets: + Description: The private subnets where Jenkins will be deployed. Use this value to configure Jenkins Spot agents. + Value: + !Join [ + ",", + [!Ref SubnetPrivateA, !Ref SubnetPrivateB, !Ref SubnetPrivateC], + ] + + JenkinsVPCPublicSubnets: + Description: The public subnets where Jenkins will be deployed. + Value: + !Join [",", [!Ref SubnetPublicA, !Ref SubnetPublicB, !Ref SubnetPublicC]] + + JenkinsMasterEC2TargetGroup: + Description: Target Group for Jenkins EC2 nodes. + Value: !Ref JenkinsMasterALBTargetGroupEC2 diff --git a/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-ecs.yaml b/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-ecs.yaml new file mode 100644 index 00000000..0e5be7ab --- /dev/null +++ b/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop-ecs.yaml @@ -0,0 +1,733 @@ +AWSTemplateFormatVersion: "2010-09-09" + +Description: A CloudFormation template that will deploy all AWS resources that are required to run the Amazon EC2 Spot CI/CD Workshop. This template is provided as-is under a modified MIT license - please see https://github.com/aws-samples/amazon-ec2-spot-cicd-workshop/blob/master/LICENSE + +Parameters: + CurrentIP: + Description: Your current IP address (used to limit access to SSH services on EC2 instances) + Type: String + AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\/([0-9]|[1-2][0-9]|3[0-2])$ + ConstraintDescription: must be specified in CIDR notation (e.g, 123.45.67.0/24) + Default: 0.0.0.0/0 + + JenkinsAdminPassword: + Description: The password that you would like to use for the admin account in your Jenkins server (must be at least 8 characters) + Type: String + ConstraintDescription: must be at least 8 characters in length + MinLength: 8 + NoEcho: true + + ECSAMI: + Description: AMI ID + Type: AWS::SSM::Parameter::Value + Default: /aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id + +Resources: + IAMRoleECS: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: 2012-10-17 + Statement: + - Effect: Allow + Principal: + Service: + - ec2.amazonaws.com + - ecs-tasks.amazonaws.com + Action: + - sts:AssumeRole + ManagedPolicyArns: + - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role + Policies: + - PolicyName: JenkinsPolicy + PolicyDocument: + Version: 2012-10-17 + Statement: + - Effect: Allow + Action: + - ec2:DescribeSpotFleetInstances + - ec2:ModifySpotFleetRequest + - ec2:CreateTags + - ec2:DescribeRegions + - ec2:DescribeInstances + - ec2:TerminateInstances + - ec2:DescribeInstanceStatus + - ec2:DescribeSpotFleetRequests + Resource: "*" + - Effect: Allow + Action: + - autoscaling:DescribeAutoScalingGroups + - autoscaling:UpdateAutoScalingGroup + Resource: "*" + - Effect: Allow + Action: + - iam:ListInstanceProfiles + - iam:ListRoles + - iam:PassRole + Resource: "*" + Path: / + + InstanceProfileECS: + Type: AWS::IAM::InstanceProfile + DependsOn: IAMRoleECS + Properties: + Path: "/" + Roles: + - !Ref IAMRoleECS + + IAMRoleECSServiceRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: 2012-10-17 + Statement: + - Effect: Allow + Principal: + Service: + - ecs.amazonaws.com + Action: + - sts:AssumeRole + ManagedPolicyArns: + - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole + Path: / + + ECSServiceLinkedRole: + Type: AWS::IAM::ServiceLinkedRole + Properties: + AWSServiceName: "ecs.amazonaws.com" + Description: "Role to enable Amazon ECS to manage your cluster." + + VPC: # This is the VPC that the CI/CD environment will be running in + Type: AWS::EC2::VPC + Properties: + CidrBlock: 192.168.0.0/21 + EnableDnsHostnames: "true" + EnableDnsSupport: "true" + InstanceTenancy: default + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop VPC + + SubnetPublicA: # The first of three subnets defined within the VPC... + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [0, !GetAZs ""] + CidrBlock: 192.168.0.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Subnet A + + SubnetPublicB: # ... and the second of three subnets... + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [1, !GetAZs ""] + CidrBlock: 192.168.1.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Subnet B + + SubnetPublicC: # ... and the third of three subnets defined within the VPC + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [2, !GetAZs ""] + CidrBlock: 192.168.2.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Subnet C + + SubnetPrivateA: # The first of three subnets defined within the VPC... + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [0, !GetAZs ""] + CidrBlock: 192.168.3.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Subnet A + - Key: Type + Value: Private + + SubnetPrivateB: # ... and the second of three subnets... + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [1, !GetAZs ""] + CidrBlock: 192.168.4.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Subnet B + - Key: Type + Value: Private + + SubnetPrivateC: # ... and the third of three subnets defined within the VPC + Type: AWS::EC2::Subnet + DependsOn: VPC + Properties: + AvailabilityZone: !Select [2, !GetAZs ""] + CidrBlock: 192.168.5.0/24 + MapPublicIpOnLaunch: "true" + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Subnet C + - Key: Type + Value: Private + + InternetGateway: # Create an Internet Gateway in order to allow EC2 instances to be accessible via the Internet + Type: AWS::EC2::InternetGateway + Properties: + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Internet Gateway + + InternetGatewayAttachment: # Once both the VPC and Internet Gateway have been created, attach the Internet Gateway to the VPC + Type: AWS::EC2::VPCGatewayAttachment + DependsOn: + - InternetGateway + - VPC + Properties: + InternetGatewayId: !Ref InternetGateway + VpcId: !Ref VPC + + EIPNATGateway: + Type: AWS::EC2::EIP + DependsOn: + - InternetGatewayAttachment + - VPC + Properties: + Domain: vpc + + NATGateway: + Type: AWS::EC2::NatGateway + DependsOn: + - SubnetPublicA + Properties: + AllocationId: !GetAtt EIPNATGateway.AllocationId + SubnetId: !Ref SubnetPublicA + + RouteTablePublic: # Create a route table which will be used within the VPC + Type: AWS::EC2::RouteTable + DependsOn: VPC + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Public Route Table + + RouteTablePrivate: # Create a route table which will be used within the VPC + Type: AWS::EC2::RouteTable + DependsOn: VPC + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: Amazon EC2 Spot CICD Workshop Private Route Table + + DefaultRoutePublic: # Add a default route to the route table, pointing to the Internet Gateway + Type: AWS::EC2::Route + DependsOn: + - RouteTablePublic + - InternetGatewayAttachment + Properties: + DestinationCidrBlock: 0.0.0.0/0 + GatewayId: !Ref InternetGateway + RouteTableId: !Ref RouteTablePublic + + DefaultRoutePrivate: # Add a default route to the route table, pointing to the NAT Gateway + Type: AWS::EC2::Route + DependsOn: + - RouteTablePrivate + - NATGateway + Properties: + DestinationCidrBlock: 0.0.0.0/0 + NatGatewayId: !Ref NATGateway + RouteTableId: !Ref RouteTablePrivate + + SubnetPublicARouteTableAssociation: # Associate the route table with the first of the three subnets in the VPC... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePublic + - SubnetPublicA + Properties: + RouteTableId: !Ref RouteTablePublic + SubnetId: !Ref SubnetPublicA + + SubnetPublicBRouteTableAssociation: # ... and the second of three subnets... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePublic + - SubnetPublicB + Properties: + RouteTableId: !Ref RouteTablePublic + SubnetId: !Ref SubnetPublicB + + SubnetPublicCRouteTableAssociation: # ... and the third of the three subnets in the VPC + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePublic + - SubnetPublicC + Properties: + RouteTableId: !Ref RouteTablePublic + SubnetId: !Ref SubnetPublicC + + SubnetPrivateARouteTableAssociation: # Associate the route table with the first of the three subnets in the VPC... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePrivate + - SubnetPrivateA + Properties: + RouteTableId: !Ref RouteTablePrivate + SubnetId: !Ref SubnetPrivateA + + SubnetPrivateBRouteTableAssociation: # ... and the second of three subnets... + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePrivate + - SubnetPrivateB + Properties: + RouteTableId: !Ref RouteTablePrivate + SubnetId: !Ref SubnetPrivateB + + SubnetPrivateCRouteTableAssociation: # ... and the third of the three subnets in the VPC + Type: AWS::EC2::SubnetRouteTableAssociation + DependsOn: + - RouteTablePrivate + - SubnetPrivateC + Properties: + RouteTableId: !Ref RouteTablePrivate + SubnetId: !Ref SubnetPrivateC + + SecurityGroupJenkins: # A Security Group that allows ingress access for SSH and the default port that a Jenkins Master will run on + Type: AWS::EC2::SecurityGroup + DependsOn: VPC + Properties: + GroupName: Spot CICD Workshop Jenkins Security Group + GroupDescription: A Security Group that allows ingress access for SSH and the default port that a Jenkins Master will run on + SecurityGroupIngress: + - Description: SSH access from the IP address that the workshop participant is using. + IpProtocol: tcp + FromPort: 22 + ToPort: 22 + CidrIp: !Ref CurrentIP + - Description: SSH access from hosts within the Spot CICD Workshop VPC - required for Build Agent deployments on to EC2 Spot instances. + IpProtocol: tcp + FromPort: 22 + ToPort: 22 + CidrIp: 192.168.0.0/21 + - Description: HTTP (8080) access from the IP address that the workshop participant is using. + IpProtocol: tcp + FromPort: 8080 + ToPort: 8080 + CidrIp: !Ref CurrentIP + - Description: HTTP (8080) access from hosts within the Spot CICD Workshop VPC, including the ALBs that reside in front of Jenkins, and Build Agents running on ECS nodes. + IpProtocol: tcp + FromPort: 8080 + ToPort: 8080 + CidrIp: 192.168.0.0/21 + - Description: JNLP (5000) access from hosts within the Spot CICD Workshop VPC - required for Build Agent deployments on ECS nodes to communicate back with the Jenkins Master container. + IpProtocol: tcp + FromPort: 5000 + ToPort: 5000 + #CidrIp: 192.168.0.0/21 + # Limited CIDR commented out as we had an issue in the first re:Invent workshop that needs further investigation + CidrIp: 0.0.0.0/0 + - Description: JNLP (50000) access from hosts within the Spot CICD Workshop VPC - the legacy port that used to be required for Build Agent deployments on ECS nodes to communicate back with the Jenkins Master container. + IpProtocol: tcp + FromPort: 50000 + ToPort: 50000 + #CidrIp: 192.168.0.0/21 + # Limited CIDR commented out as we had an issue in the first re:Invent workshop that needs further investigation + CidrIp: 0.0.0.0/0 + VpcId: !Ref VPC + + SecurityGroupJenkinsALB: # A Security Group that allows ingress access for HTTP on ALBs and used to access the Jenkins Master + Type: AWS::EC2::SecurityGroup + DependsOn: VPC + Properties: + GroupName: Spot CICD Workshop Jenkins ALB Security Group + GroupDescription: A Security Group that allows ingress access for HTTP on ALBs and used to access the Jenkins Master + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 80 + ToPort: 80 + CidrIp: 0.0.0.0/0 + VpcId: !Ref VPC + + SecurityGroupEFS: + Type: AWS::EC2::SecurityGroup + DependsOn: + - SecurityGroupJenkins + - VPC + Properties: + GroupName: Spot CICD Workshop EFS Security Group + GroupDescription: A Security Group that allows access to EFS volume targets from the Jenkins Securiy Group + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 2049 + ToPort: 2049 + SourceSecurityGroupId: !Ref SecurityGroupJenkins + VpcId: !Ref VPC + + JenkinsMasterALB: # This is the Application Load Balancer that resides in front of your Jenkins Master instance and is responsible for port-mapping requests from TCP:80 to TCP:8080 + Type: AWS::ElasticLoadBalancingV2::LoadBalancer + DependsOn: + - SecurityGroupJenkinsALB + - SubnetPublicA + - SubnetPublicB + - SubnetPublicC + Properties: + Name: JenkinsMasterALBECS + Scheme: internet-facing + SecurityGroups: + - !Ref SecurityGroupJenkinsALB + Subnets: + - !Ref SubnetPublicA + - !Ref SubnetPublicB + - !Ref SubnetPublicC + + JenkinsMasterALBTargetGroupECS: # This is the Target Group used by the JenkinsMasterALB load balancer when Jenkins is running in a container on an ECS cluster + Type: AWS::ElasticLoadBalancingV2::TargetGroup + DependsOn: + - VPC + Properties: + HealthCheckIntervalSeconds: 15 + HealthCheckPath: /login + HealthCheckPort: 8080 + HealthCheckProtocol: HTTP + HealthCheckTimeoutSeconds: 5 + HealthyThresholdCount: 2 + Matcher: + HttpCode: 200 + Name: JenkinsMasterECSTargetGroup + Port: 8080 + Protocol: HTTP + TargetType: ip + TargetGroupAttributes: + - Key: "slow_start.duration_seconds" + Value: "120" + UnhealthyThresholdCount: 4 + VpcId: !Ref VPC + + JenkinsMasterALBListener: # This is the ALB Listener used to access the Jenkins Master + Type: AWS::ElasticLoadBalancingV2::Listener + DependsOn: + - JenkinsMasterALB + - JenkinsMasterALBTargetGroupECS + Properties: + DefaultActions: + - Type: forward + TargetGroupArn: !Ref JenkinsMasterALBTargetGroupECS + LoadBalancerArn: !Ref JenkinsMasterALB + Port: 80 + Protocol: HTTP + + JenkinsMasterALBListenerRuleECS: # The ALB Listener rule that forwards all traffic destined for the Jenkins Master to the appropriate Target Group + Type: AWS::ElasticLoadBalancingV2::ListenerRule + DependsOn: + - JenkinsMasterALBListener + - JenkinsMasterALBTargetGroupECS + Properties: + Actions: + - Type: forward + TargetGroupArn: !Ref JenkinsMasterALBTargetGroupECS + Conditions: + - Field: path-pattern + Values: + - "/*" + ListenerArn: !Ref JenkinsMasterALBListener + Priority: 2 + + EFSJenkinsHomeVolume: + Type: AWS::EFS::FileSystem + Properties: + PerformanceMode: generalPurpose + + EFSMountTargetJenkinsHomeVolumeA: + Type: AWS::EFS::MountTarget + DependsOn: + - EFSJenkinsHomeVolume + - SecurityGroupEFS + - SubnetPublicA + Properties: + FileSystemId: !Ref EFSJenkinsHomeVolume + SecurityGroups: + - !Ref SecurityGroupEFS + SubnetId: !Ref SubnetPublicA + + EFSMountTargetJenkinsHomeVolumeB: + Type: AWS::EFS::MountTarget + DependsOn: + - EFSJenkinsHomeVolume + - SecurityGroupEFS + - SubnetPublicB + Properties: + FileSystemId: !Ref EFSJenkinsHomeVolume + SecurityGroups: + - !Ref SecurityGroupEFS + SubnetId: !Ref SubnetPublicB + + EFSMountTargetJenkinsHomeVolumeC: + Type: AWS::EFS::MountTarget + DependsOn: + - EFSJenkinsHomeVolume + - SecurityGroupEFS + - SubnetPublicC + Properties: + FileSystemId: !Ref EFSJenkinsHomeVolume + SecurityGroups: + - !Ref SecurityGroupEFS + SubnetId: !Ref SubnetPublicC + + ECSLaunchTemplate: # This is a launch template that will be used to provision ECS cluster nodes + Type: AWS::EC2::LaunchTemplate + DependsOn: + - EFSJenkinsHomeVolume + - InstanceProfileECS + - SecurityGroupJenkins + Properties: + LaunchTemplateName: ECSLaunchTemplate + LaunchTemplateData: + BlockDeviceMappings: + - DeviceName: "/dev/xvda" + Ebs: + DeleteOnTermination: "true" + VolumeSize: 30 + VolumeType: gp2 + IamInstanceProfile: + Name: !Ref InstanceProfileECS + ImageId: !Ref ECSAMI + InstanceType: t3.medium + SecurityGroupIds: + - !Ref SecurityGroupJenkins + TagSpecifications: + - ResourceType: instance + Tags: + - Key: Name + Value: "Jenkins ECS Cluster Instance" + UserData: + Fn::Base64: !Sub | + #!/bin/bash + # Register instance with ECS cluster + echo ECS_CLUSTER=SpotCICDWorkshopECSCluster >> /etc/ecs/ecs.config + # Install all pending updates to the system + yum -y update + # Install the nfs-utils package + yum -y install nfs-utils + # Create EFS mountpoint + mkdir /mnt/efs + mkdir /mnt/efs/jenkins_home + # Mount the Jenkins EFS volume + mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).${EFSJenkinsHomeVolume}.efs.eu-west-1.amazonaws.com:/ /mnt/efs/jenkins_home + # Deploy the AWS CLI + yum -y install unzip wget + wget -q https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -O ./awscli-bundle.zip + unzip ./awscli-bundle.zip + ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws + # Register a custom attribute for this ECS host to indicate if this is a spot instance or not + yum -y install jq + INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) + INSTANCE_LIFECYCLE=$(/usr/local/bin/aws ec2 describe-instances --region eu-west-1 --instance-ids $INSTANCE_ID | jq .Reservations | jq .[0] | jq .Instances | jq .[0] | jq .InstanceLifecycle | tr -d \") + echo ECS_INSTANCE_ATTRIBUTES={\"lifecycle\": \"$INSTANCE_LIFECYCLE\"} >> /etc/ecs/ecs.config + # If this is a spot instance, ensure that container draining occurs prior to interruption + if [ $INSTANCE_LIFECYCLE == spot ] + then + wget -q https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/interruption_check.sh -O ./interruption_check.sh + chmod +x ./interruption_check.sh + nohup ./interruption_check.sh &>/dev/null & + fi + + ASGOnDemand: + Type: AWS::AutoScaling::AutoScalingGroup + Properties: + HealthCheckType: EC2 + HealthCheckGracePeriod: 300 + MinSize: 1 + MaxSize: 1 + DesiredCapacity: 1 + VPCZoneIdentifier: + - Ref: SubnetPublicA + - Ref: SubnetPublicB + - Ref: SubnetPublicC + NewInstancesProtectedFromScaleIn: true + MixedInstancesPolicy: + InstancesDistribution: + OnDemandAllocationStrategy: lowest-price + OnDemandBaseCapacity: 0 + OnDemandPercentageAboveBaseCapacity: 100 + LaunchTemplate: + LaunchTemplateSpecification: + LaunchTemplateId: + Ref: ECSLaunchTemplate + Version: !GetAtt ECSLaunchTemplate.LatestVersionNumber + Overrides: + - InstanceRequirements: + VCpuCount: + Min: 2 + Max: 8 + MemoryMiB: + Min: 4096 + CpuManufacturers: + - intel + Tags: + - Key: Name + Value: "EC2OnDemandJenkinsECS" + PropagateAtLaunch: true + + ECSCluster: + Type: AWS::ECS::Cluster + DependsOn: ECSServiceLinkedRole + Properties: + ClusterName: SpotCICDWorkshopECSCluster + + ECSCapacityProviderOD: + Type: AWS::ECS::CapacityProvider + Properties: + AutoScalingGroupProvider: + AutoScalingGroupArn: !Ref ASGOnDemand + ManagedScaling: + MaximumScalingStepSize: 1 + MinimumScalingStepSize: 1 + Status: ENABLED + TargetCapacity: 100 + ManagedTerminationProtection: ENABLED + + ClusterCPAssociation: + Type: "AWS::ECS::ClusterCapacityProviderAssociations" + Properties: + Cluster: !Ref ECSCluster + CapacityProviders: + - !Ref ECSCapacityProviderOD + DefaultCapacityProviderStrategy: + - Base: 1 + Weight: 100 + CapacityProvider: !Ref ECSCapacityProviderOD + + ECSTaskDefinitionJenkinsMaster: + Type: AWS::ECS::TaskDefinition + DependsOn: ECSServiceLinkedRole + Properties: + ContainerDefinitions: + - Cpu: 512 + Essential: true + Image: christianhxc/amazon-ec2-spot-cicd-workshop-jenkins:2.346.3 + Memory: 1536 + MountPoints: + - SourceVolume: JENKINS_HOME + ContainerPath: /var/jenkins_home + Name: SpotCICDWorkshopJenkinsMasterContainer + PortMappings: + - ContainerPort: 5000 + HostPort: 5000 + - ContainerPort: 8080 + HostPort: 8080 + - ContainerPort: 50000 + HostPort: 50000 + Environment: + - Name: "JENKINS_ADMIN_ID" + Value: "admin" + - Name: "JENKINS_ADMIN_PASSWORD" + Value: !Ref JenkinsAdminPassword + Privileged: true + User: "root" + ExecutionRoleArn: !GetAtt IAMRoleECS.Arn + NetworkMode: awsvpc + Volumes: + - Host: + SourcePath: /mnt/efs/jenkins_home + Name: JENKINS_HOME + + ECSServiceJenkinsMaster: + Type: AWS::ECS::Service + DependsOn: + - ECSCluster + - ECSServiceLinkedRole + - ECSTaskDefinitionJenkinsMaster + - JenkinsMasterALBListenerRuleECS + - ServiceDiscoveryJenkinsMaster + Properties: + Cluster: !Ref ECSCluster + DeploymentConfiguration: + MaximumPercent: 100 + MinimumHealthyPercent: 50 + DesiredCount: 1 + HealthCheckGracePeriodSeconds: 120 + LoadBalancers: + - ContainerName: SpotCICDWorkshopJenkinsMasterContainer + ContainerPort: 8080 + TargetGroupArn: !Ref JenkinsMasterALBTargetGroupECS + NetworkConfiguration: + AwsvpcConfiguration: + SecurityGroups: + - !Ref SecurityGroupJenkins + Subnets: + - !Ref SubnetPrivateA + - !Ref SubnetPrivateB + - !Ref SubnetPrivateC + ServiceRegistries: + - ContainerName: SpotCICDWorkshopJenkinsMasterContainer + RegistryArn: !GetAtt ServiceDiscoveryJenkinsMaster.Arn + TaskDefinition: !Ref ECSTaskDefinitionJenkinsMaster + + ServiceDiscoveryJenkinsMasterNamespace: + Type: AWS::ServiceDiscovery::PrivateDnsNamespace + DependsOn: VPC + Properties: + Vpc: !Ref VPC + Name: jenkins.local + + ServiceDiscoveryJenkinsMaster: + Type: AWS::ServiceDiscovery::Service + DependsOn: ServiceDiscoveryJenkinsMasterNamespace + Properties: + Description: Jenkins Master Service + DnsConfig: + DnsRecords: + - Type: A + TTL: 60 + NamespaceId: !Ref ServiceDiscoveryJenkinsMasterNamespace + HealthCheckCustomConfig: + FailureThreshold: 1 + Name: master + +Outputs: + EFSFileSystemID: + Description: The file system ID of the EFS volume that is used to persist JENKINS_HOME across EC2 & ECS instances. + Value: !Ref EFSJenkinsHomeVolume + + JenkinsDNSName: + Description: The DNS name of the Application Load Balancer that is used to gain access to your Jenkins server. + Value: !GetAtt JenkinsMasterALB.DNSName + + JenkinsMasterSecurityGroup: + Description: Security Group for Jenkins nodes. Use this value to configure Jenkins ECS Plugin + Value: !Ref SecurityGroupJenkins + + JenkinsVPCPrivateSubnets: + Description: The private subnets where Jenkins will be deployed. Use this value to configure Jenkins Spot agents. + Value: + !Join [ + ",", + [!Ref SubnetPrivateA, !Ref SubnetPrivateB, !Ref SubnetPrivateC], + ] + + JenkinsVPCPublicSubnets: + Description: The public subnets where Jenkins will be deployed. Use this value to configure Jenkins ECS plugin + Value: + !Join [",", [!Ref SubnetPublicA, !Ref SubnetPublicB, !Ref SubnetPublicC]] + + JenkinsMasterALBTargetGroupECS: + Description: Target Group for Jenkins EC2 nodes. + Value: !Ref JenkinsMasterALBTargetGroupECS diff --git a/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop.yaml b/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop.yaml deleted file mode 100644 index 0b738405..00000000 --- a/workshops/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop.yaml +++ /dev/null @@ -1,1298 +0,0 @@ -AWSTemplateFormatVersion: "2010-09-09" - -Description: A CloudFormation template that will deploy all AWS resources that are required to run the Amazon EC2 Spot CI/CD Workshop. This template is provided as-is under a modified MIT license - please see https://github.com/aws-samples/amazon-ec2-spot-cicd-workshop/blob/master/LICENSE - -Parameters: - KeyPair: - Description: The Key Pair created earlier in the Preparation Lab - Type: AWS::EC2::KeyPair::KeyName - - CurrentIP: - Description: Your current IP address (used to limit access to SSH services on EC2 instances) - Type: String - AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\/([0-9]|[1-2][0-9]|3[0-2])$ - ConstraintDescription: must be specified in CIDR notation (e.g, 123.45.67.0/24) - Default: 0.0.0.0/0 - - JenkinsAdminPassword: - Description: The password that you would like to use for the admin account in your Jenkins server (must be at least 8 characters) - Type: String - ConstraintDescription: must be at least 8 characters in length - MinLength: 8 - NoEcho: true - -Resources: - IAMRoleAMILookupLambdaExecution: # IAM Role that allows the AMILookupLambdaFunction to look up the latest Amazon Linux AMI and write logs to CloudWatch Logs - Type: AWS::IAM::Role - # DependsOn: None - # DependedOn: AMILookupLambdaFunction - Properties: - AssumeRolePolicyDocument: - Version: 2012-10-17 - Statement: - Effect: Allow - Principal: - Service: - - lambda.amazonaws.com - Action: - - sts:AssumeRole - Path: / - Policies: - - PolicyName: AMILookupExecution - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: - - logs:CreateLogGroup - - logs:CreateLogStream - - logs:PutLogEvents - Resource: arn:aws:logs:*:*:* - - Effect: Allow - Action: ec2:DescribeImages - Resource: "*" - - IAMRoleTestEnvironmentLambdaExecution: - Type: AWS::IAM::Role - DependsOn: - - DynamoDBTestEnvironmentTable - - IAMRoleTestEnvironmentCloudFormation - # DependedOn: TestEnvironmentLambdaFuntion - Properties: - AssumeRolePolicyDocument: - Version: 2012-10-17 - Statement: - Effect: Allow - Principal: - Service: - - lambda.amazonaws.com - Action: - - sts:AssumeRole - Path: / - Policies: - - PolicyName: TestEnvironmentExecution - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: - - logs:CreateLogGroup - - logs:CreateLogStream - - logs:PutLogEvents - Resource: arn:aws:logs:*:*:* - - Effect: Allow - Action: - - dynamodb:getItem - - dynamodb:putItem - Resource: !GetAtt DynamoDBTestEnvironmentTable.Arn - - Effect: Allow - Action: - - cloudformation:createStack - - cloudformation:deleteStack - Resource: "*" - - Effect: Allow - Action: - - iam:PassRole - Resource: !GetAtt IAMRoleTestEnvironmentCloudFormation.Arn - - IAMRoleTestEnvironmentCloudFormation: - Type: AWS::IAM::Role - DependsOn: AMILookupLambdaFunction - #DependedOn: None - Properties: - AssumeRolePolicyDocument: - Version: 2012-10-17 - Statement: - Effect: Allow - Principal: - Service: - - cloudformation.amazonaws.com - Action: - - sts:AssumeRole - Path: / - Policies: - - PolicyName: TestEnvironmentDeployment - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: - - ec2:* - - elasticloadbalancing:CreateListener - - elasticloadbalancing:CreateLoadBalancer - - elasticloadbalancing:CreateRule - - elasticloadbalancing:CreateTargetGroup - - elasticloadbalancing:DeleteTargetGroup - - elasticloadbalancing:DeleteListener - - elasticloadbalancing:DeleteLoadBalancer - - elasticloadbalancing:DeleteRule - - elasticloadbalancing:DescribeListeners - - elasticloadbalancing:DescribeLoadBalancers - - elasticloadbalancing:DescribeRules - - elasticloadbalancing:DescribeTargetGroups - - elasticloadbalancingv2:CreateListener - - elasticloadbalancingv2:CreateLoadBalancer - - elasticloadbalancingv2:CreateRule - - elasticloadbalancingv2:CreateTargetGroup - - elasticloadbalancingv2:DeleteListener - - elasticloadbalancingv2:DeleteLoadBalancer - - elasticloadbalancingv2:DeleteTargetGroup - - elasticloadbalancingv2:DeleteRule - - elasticloadbalancingv2:DescribeListeners - - elasticloadbalancingv2:DescribeLoadBalancers - - elasticloadbalancingv2:DescribeRules - - elasticloadbalancingv2:DescribeTargetGroups - - iam:AddRoleToInstanceProfile - - iam:CreateRole - - iam:CreateInstanceProfile - - iam:DeleteInstanceProfile - - iam:DeleteRole - - iam:DeleteRolePolicy - - iam:ListInstanceProfiles - - iam:ListRoles - - iam:PassRole - - iam:PutRolePolicy - - iam:RemoveRoleFromInstanceProfile - Resource: "*" - - Effect: Allow - Action: - - lambda:InvokeFunction - Resource: !GetAtt AMILookupLambdaFunction.Arn - - IAMRoleJenkins: - Type: AWS::IAM::Role - DependsOn: - - DeploymentArtifactsS3Bucket - - TestEnvironmentLambdaFunction - # DependedOn: InstanceProfileJenkins - Properties: - AssumeRolePolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Principal: - Service: - - ec2.amazonaws.com - - ecs-tasks.amazonaws.com - Action: - - sts:AssumeRole - Path: / - Policies: - - PolicyName: JenkinsPolicy - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: s3:* - Resource: !Sub "arn:aws:s3:::${DeploymentArtifactsS3Bucket}/*" - - Effect: Allow - Action: lambda:invokeFunction - Resource: !GetAtt TestEnvironmentLambdaFunction.Arn - - InstanceProfileJenkins: - Type: AWS::IAM::InstanceProfile - DependsOn: IAMRoleJenkins - # DependedOn: JenkinsOnDemandEC2Instance - Properties: - Path: "/" - Roles: - - !Ref IAMRoleJenkins - - IAMUserJenkins: - Type: AWS::IAM::User - # DependsOn: None - # DependedOn: None - Properties: - Policies: - - PolicyName: SpotAgentPolicy - # Policy is documented at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet-requests.html#spot-fleet-iam-users - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: - - ec2:* - - iam:ListInstanceProfiles - - iam:ListRoles - - iam:PassRole - Resource: "*" - - PolicyName: ECSAgentPolicy - # Policy is documented at https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Container+Service+Plugin - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: - - ecs:DescribeContainerInstances - - ecs:DescribeTaskDefinition - - ecs:ListClusters - - ecs:ListTaskDefinitions - - ecs:RegisterTaskDefinition - Resource: "*" - - Effect: Allow - Action: - - ecs:ListContainerInstances - - ecs:StopTask - Resource: !GetAtt ECSCluster.Arn - - Effect: Allow - Action: - - ecs:RunTask - Resource: !Join - - "" - - - "arn:aws:ecs:" - - !Ref AWS::Region - - ":" - - !Ref AWS::AccountId - - ":task-definition/SpotCICDWorkshopECSAgents-ECSBuildAgent:*" - - Effect: Allow - Action: - - ecs:DescribeTasks - - ecs:StopTask - Resource: !Join - - "" - - - "arn:aws:ecs:" - - !Ref AWS::Region - - ":" - - !Ref AWS::AccountId - - ":task/*" - UserName: SpotCICDWorkshopJenkins - - IAMRoleECS: - Type: AWS::IAM::Role - # DependsOn: None - # DependedOn: InstanceProfileECS - Properties: - AssumeRolePolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Principal: - Service: - - ec2.amazonaws.com - Action: - - sts:AssumeRole - ManagedPolicyArns: - - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role - Policies: - - PolicyName: EC2DescribeInstances - PolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Action: ec2:DescribeInstances - Resource: "*" - Path: / - - InstanceProfileECS: - Type: AWS::IAM::InstanceProfile - DependsOn: IAMRoleECS - # DependedOn: AutoScalingECSLaunchConfiguration - Properties: - Path: "/" - Roles: - - !Ref IAMRoleECS - - IAMRoleECSServiceRole: - Type: AWS::IAM::Role - # DependsOn: - # DependedOn: ECSServiceJenkinsMaster - Properties: - AssumeRolePolicyDocument: - Version: 2012-10-17 - Statement: - - Effect: Allow - Principal: - Service: - - ecs.amazonaws.com - Action: - - sts:AssumeRole - ManagedPolicyArns: - - arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole - Path: / - - ECSServiceLinkedRole: - Type: AWS::IAM::ServiceLinkedRole - Properties: - AWSServiceName: "ecs.amazonaws.com" - Description: "Role to enable Amazon ECS to manage your cluster." - - DeploymentArtifactsS3Bucket: # This is an S3 bucket where the deployment artifacts will be uploaded to - # DependsOn: None - # DependedOn: IAMRoleJenkins - Type: AWS::S3::Bucket - - VPC: # This is the VPC that the CI/CD environment will be running in - Type: AWS::EC2::VPC - # DependsOn: None - # DependedOn: InternetGatewayAttachment, JenkinsMasterALBTargetGroupEC2, JenkinsMasterALBTargetGroupECS, RouteTable, SecurityGroupJenkins, SecurityGroupJenkinsALB, SubnetPublicA, SubnetPublicA, SubnetPublicC, - Properties: - CidrBlock: 192.168.0.0/21 - EnableDnsHostnames: "true" - EnableDnsSupport: "true" - InstanceTenancy: default - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop VPC - - SubnetPublicA: # The first of three subnets defined within the VPC... - Type: AWS::EC2::Subnet - DependsOn: VPC - # DependedOn: SubnetPublicARouteTableAssociation - Properties: - AvailabilityZone: !Select [0, !GetAZs ""] - CidrBlock: 192.168.0.0/24 - MapPublicIpOnLaunch: "true" - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Public Subnet A - - SubnetPublicB: # ... and the second of three subnets... - Type: AWS::EC2::Subnet - DependsOn: VPC - # DependedOn: SubnetPublicBRouteTableAssociation - Properties: - AvailabilityZone: !Select [1, !GetAZs ""] - CidrBlock: 192.168.1.0/24 - MapPublicIpOnLaunch: "true" - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Public Subnet B - - SubnetPublicC: # ... and the third of three subnets defined within the VPC - Type: AWS::EC2::Subnet - DependsOn: VPC - # DependedOn: SubnetPublicCRouteTableAssociation - Properties: - AvailabilityZone: !Select [2, !GetAZs ""] - CidrBlock: 192.168.2.0/24 - MapPublicIpOnLaunch: "true" - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Public Subnet C - - SubnetPrivateA: # The first of three subnets defined within the VPC... - Type: AWS::EC2::Subnet - DependsOn: VPC - # DependedOn: SubnetPrivateARouteTableAssociation - Properties: - AvailabilityZone: !Select [0, !GetAZs ""] - CidrBlock: 192.168.3.0/24 - MapPublicIpOnLaunch: "true" - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Private Subnet A - - SubnetPrivateB: # ... and the second of three subnets... - Type: AWS::EC2::Subnet - DependsOn: VPC - # DependedOn: SubnetPrivateARouteTableAssociation - Properties: - AvailabilityZone: !Select [1, !GetAZs ""] - CidrBlock: 192.168.4.0/24 - MapPublicIpOnLaunch: "true" - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Private Subnet B - - SubnetPrivateC: # ... and the third of three subnets defined within the VPC - Type: AWS::EC2::Subnet - DependsOn: VPC - # DependedOn: SubnetPrivateCRouteTableAssociation - Properties: - AvailabilityZone: !Select [2, !GetAZs ""] - CidrBlock: 192.168.5.0/24 - MapPublicIpOnLaunch: "true" - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Private Subnet C - - InternetGateway: # Create an Internet Gateway in order to allow EC2 instances to be accessible via the Internet - Type: AWS::EC2::InternetGateway - # DependsOn: None - # DependedOn: InternetGatewayAttachment - Properties: - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Internet Gateway - - InternetGatewayAttachment: # Once both the VPC and Internet Gateway have been created, attach the Internet Gateway to the VPC - Type: AWS::EC2::VPCGatewayAttachment - DependsOn: - - InternetGateway - - VPC - # DependedOn: DefaultRoute - Properties: - InternetGatewayId: !Ref InternetGateway - VpcId: !Ref VPC - - EIPNATGateway: - Type: AWS::EC2::EIP - DependsOn: - - InternetGatewayAttachment - - VPC - # DependedOn: NATGateway - Properties: - Domain: vpc - - NATGateway: - Type: AWS::EC2::NatGateway - DependsOn: - - SubnetPublicA - # DependedOn: - Properties: - AllocationId: !GetAtt EIPNATGateway.AllocationId - SubnetId: !Ref SubnetPublicA - - RouteTablePublic: # Create a route table which will be used within the VPC - Type: AWS::EC2::RouteTable - DependsOn: VPC - # DependedOn: DefaultRoutePublic, SubnetPublicARouteTableAssociation, SubnetPublicBRouteTableAssociation, SubnetPublicCRouteTableAssociation - Properties: - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Public Route Table - - RouteTablePrivate: # Create a route table which will be used within the VPC - Type: AWS::EC2::RouteTable - DependsOn: VPC - # DependedOn: DefaultRoute, SubnetPrivateARouteTableAssociation, SubnetPrivateBRouteTableAssociation, SubnetPrivateCRouteTableAssociation - Properties: - VpcId: !Ref VPC - Tags: - - Key: Name - Value: Amazon EC2 Spot CICD Workshop Private Route Table - - DefaultRoutePublic: # Add a default route to the route table, pointing to the Internet Gateway - Type: AWS::EC2::Route - DependsOn: - - RouteTablePublic - - InternetGatewayAttachment - # DependedOn: None - Properties: - DestinationCidrBlock: 0.0.0.0/0 - GatewayId: !Ref InternetGateway - RouteTableId: !Ref RouteTablePublic - - DefaultRoutePrivate: # Add a default route to the route table, pointing to the NAT Gateway - Type: AWS::EC2::Route - DependsOn: - - RouteTablePrivate - - NATGateway - # DependedOn: None - Properties: - DestinationCidrBlock: 0.0.0.0/0 - NatGatewayId: !Ref NATGateway - RouteTableId: !Ref RouteTablePrivate - - SubnetPublicARouteTableAssociation: # Associate the route table with the first of the three subnets in the VPC... - Type: AWS::EC2::SubnetRouteTableAssociation - DependsOn: - - RouteTablePublic - - SubnetPublicA - # DependedOn: None - Properties: - RouteTableId: !Ref RouteTablePublic - SubnetId: !Ref SubnetPublicA - - SubnetPublicBRouteTableAssociation: # ... and the second of three subnets... - Type: AWS::EC2::SubnetRouteTableAssociation - DependsOn: - - RouteTablePublic - - SubnetPublicB - # DependedOn: None - Properties: - RouteTableId: !Ref RouteTablePublic - SubnetId: !Ref SubnetPublicB - - SubnetPublicCRouteTableAssociation: # ... and the third of the three subnets in the VPC - Type: AWS::EC2::SubnetRouteTableAssociation - DependsOn: - - RouteTablePublic - - SubnetPublicC - # DependedOn: None - Properties: - RouteTableId: !Ref RouteTablePublic - SubnetId: !Ref SubnetPublicC - - SubnetPrivateARouteTableAssociation: # Associate the route table with the first of the three subnets in the VPC... - Type: AWS::EC2::SubnetRouteTableAssociation - DependsOn: - - RouteTablePrivate - - SubnetPrivateA - # DependedOn: None - Properties: - RouteTableId: !Ref RouteTablePrivate - SubnetId: !Ref SubnetPrivateA - - SubnetPrivateBRouteTableAssociation: # ... and the second of three subnets... - Type: AWS::EC2::SubnetRouteTableAssociation - DependsOn: - - RouteTablePrivate - - SubnetPrivateB - # DependedOn: None - Properties: - RouteTableId: !Ref RouteTablePrivate - SubnetId: !Ref SubnetPrivateB - - SubnetPrivateCRouteTableAssociation: # ... and the third of the three subnets in the VPC - Type: AWS::EC2::SubnetRouteTableAssociation - DependsOn: - - RouteTablePrivate - - SubnetPrivateC - # DependedOn: None - Properties: - RouteTableId: !Ref RouteTablePrivate - SubnetId: !Ref SubnetPrivateC - - SecurityGroupJenkins: # A Security Group that allows ingress access for SSH and the default port that a Jenkins Master will run on - Type: AWS::EC2::SecurityGroup - DependsOn: VPC - # DependedOn: JenkinsOnDemandEC2Instance - Properties: - GroupName: Spot CICD Workshop Jenkins Security Group - GroupDescription: A Security Group that allows ingress access for SSH and the default port that a Jenkins Master will run on - SecurityGroupIngress: - - Description: SSH access from the IP address that the workshop participant is using. - IpProtocol: tcp - FromPort: 22 - ToPort: 22 - CidrIp: !Ref CurrentIP - - Description: SSH access from hosts within the Spot CICD Workshop VPC - required for Build Agent deployments on to EC2 Spot instances. - IpProtocol: tcp - FromPort: 22 - ToPort: 22 - CidrIp: 192.168.0.0/21 - - Description: HTTP (8080) access from the IP address that the workshop participant is using. - IpProtocol: tcp - FromPort: 8080 - ToPort: 8080 - CidrIp: !Ref CurrentIP - - Description: HTTP (8080) access from hosts within the Spot CICD Workshop VPC, including the ALBs that reside in front of Jenkins, and Build Agents running on ECS nodes. - IpProtocol: tcp - FromPort: 8080 - ToPort: 8080 - CidrIp: 192.168.0.0/21 - - Description: JNLP (5000) access from hosts within the Spot CICD Workshop VPC - required for Build Agent deployments on ECS nodes to communicate back with the Jenkins Master container. - IpProtocol: tcp - FromPort: 5000 - ToPort: 5000 - #CidrIp: 192.168.0.0/21 - # Limited CIDR commented out as we had an issue in the first re:Invent workshop that needs further investigation - CidrIp: 0.0.0.0/0 - - Description: JNLP (50000) access from hosts within the Spot CICD Workshop VPC - the legacy port that used to be required for Build Agent deployments on ECS nodes to communicate back with the Jenkins Master container. - IpProtocol: tcp - FromPort: 50000 - ToPort: 50000 - #CidrIp: 192.168.0.0/21 - # Limited CIDR commented out as we had an issue in the first re:Invent workshop that needs further investigation - CidrIp: 0.0.0.0/0 - VpcId: !Ref VPC - - SecurityGroupJenkinsALB: # A Security Group that allows ingress access for HTTP on ALBs and used to access the Jenkins Master - Type: AWS::EC2::SecurityGroup - DependsOn: VPC - # DependedOn: JenkinsMasterALB - Properties: - GroupName: Spot CICD Workshop Jenkins ALB Security Group - GroupDescription: A Security Group that allows ingress access for HTTP on ALBs and used to access the Jenkins Master - SecurityGroupIngress: - - IpProtocol: tcp - FromPort: 80 - ToPort: 80 - CidrIp: 0.0.0.0/0 - VpcId: !Ref VPC - - SecurityGroupEFS: - Type: AWS::EC2::SecurityGroup - DependsOn: - - SecurityGroupJenkins - - VPC - # DependedOn: EFSMountTargetJenkinsHomeVolumeA, EFSMountTargetJenkinsHomeVolumeB, EFSMountTargetJenkinsHomeVolumeC - Properties: - GroupName: Spot CICD Workshop EFS Security Group - GroupDescription: A Security Group that allows access to EFS volume targets from the Jenkins Securiy Group - SecurityGroupIngress: - - IpProtocol: tcp - FromPort: 2049 - ToPort: 2049 - SourceSecurityGroupId: !Ref SecurityGroupJenkins - VpcId: !Ref VPC - - AMILookupLambdaFunction: # A Lambda function that will be used to look up the AMI for the latest Amazon Linux AMI. The source code for this Lambda function can be obtained at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-custom-resources-lambda-lookup-amiids.html. - Type: AWS::Lambda::Function - DependsOn: IAMRoleAMILookupLambdaExecution - #DependedOn: EC2AMILookupCustomResource, ECSAMILookupCustomResource - Properties: - Code: - S3Bucket: !Sub amazon-ec2-spot-cicd-workshop-${AWS::Region} - S3Key: amilookup.zip - Handler: amilookup.handler - Role: !GetAtt IAMRoleAMILookupLambdaExecution.Arn - Runtime: nodejs12.x - Timeout: 30 - - EC2AMILookupCustomResource: # A custom resource that provides the latest Amazon Linux AMI via EC2AMILookupCustomResource.Id - Type: Custom::AMILookup - DependsOn: AMILookupLambdaFunction - # DependedOn: - Properties: - Architecture: HVM64 - Region: !Ref AWS::Region - ServiceToken: !GetAtt AMILookupLambdaFunction.Arn - - ECSAMILookupCustomResource: # A custom resource that provides the latest Amazon ECS-Optimized AMI via ECSAMILookupCustomResource.Id - Type: Custom::AMILookup - DependsOn: AMILookupLambdaFunction - # DependedOn: - Properties: - Architecture: ECSHVM64 - Region: !Ref AWS::Region - ServiceToken: !GetAtt AMILookupLambdaFunction.Arn - - TestEnvironmentLambdaFunction: # A Lambda function that will be used to create and destroy buld testing environments. - DependsOn: - - AMILookupLambdaFunction - - DeploymentArtifactsS3Bucket - - DynamoDBTestEnvironmentTable - - IAMRoleTestEnvironmentLambdaExecution - - SubnetPublicA - - SubnetPublicB - - SubnetPublicC - - VPC - #DependedOn: None - Type: AWS::Lambda::Function - Properties: - FunctionName: SpotCICDWorkshop_ManageTestEnvironment - Description: Handles the deployment and termination of Test Environments witin the Spot CICD Workshop. - Handler: index.handler - MemorySize: 128 - Role: !GetAtt IAMRoleTestEnvironmentLambdaExecution.Arn - Runtime: nodejs12.x - Timeout: 60 - Code: - ZipFile: !Sub | - 'use strict'; - var AWS = require('aws-sdk'); - var actions = { - deploy: function (cfn, ddb, request_payload) { - return new Promise(function (resolve, reject) { - var cfn_params = { - StackName: request_payload.stackName, - Capabilities: [ 'CAPABILITY_IAM' ], - Parameters: [ - { - ParameterKey: 'KeyPair', - ParameterValue: '${KeyPair}' - }, { - ParameterKey: 'CurrentIP', - ParameterValue: '${CurrentIP}' - }, { - ParameterKey: 'AMILookupLambdaFunctionARN', - ParameterValue: '${AMILookupLambdaFunction.Arn}' - }, { - ParameterKey: 'DeploymentArtifactsS3Bucket', - ParameterValue: '${DeploymentArtifactsS3Bucket}' - }, { - ParameterKey: 'VPC', - ParameterValue: '${VPC}' - }, { - ParameterKey: 'SubnetA', - ParameterValue: '${SubnetPublicA}' - }, { - ParameterKey: 'SubnetB', - ParameterValue: '${SubnetPublicB}' - }, { - ParameterKey: 'SubnetC', - ParameterValue: '${SubnetPublicC}' - } - ], - RoleARN: '${IAMRoleTestEnvironmentCloudFormation.Arn}', - TemplateURL: 'https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop_game-of-life.yaml' - }; - cfn.createStack(cfn_params, function(err, cfn_data) { - if (err) { return reject(err); } - console.log('[INFO]', 'StackId: ', cfn_data.StackId); - return new Promise(function (resolve, reject) { - var ddb_params = { - Item: { - 'JobBaseName': { S: request_payload.jobBaseName }, - 'BuildID': { N: request_payload.buildId }, - 'CloudFormationStackID': { S: cfn_data.StackId } - }, - ReturnConsumedCapacity: 'TOTAL', - TableName: '${DynamoDBTestEnvironmentTable}' - }; - ddb.putItem(ddb_params, function(err, ddb_data) { - if (err) { return reject(err); } - console.log('[INFO]', 'Consumed Capacity Units: ', ddb_data.ConsumedCapacity.CapacityUnits); - return resolve(); - }); - }); - }); - }); - }, - terminate: function(cfn, ddb, request_payload) { - return new Promise(function (resolve, reject) { - var ddb_params = { - Key: { - 'JobBaseName': { S: request_payload.jobBaseName }, - 'BuildID': { N: request_payload.buildId } - }, - TableName: '${DynamoDBTestEnvironmentTable}' - }; - ddb.getItem(ddb_params, function(err, ddb_data) { - if (err) { return reject(err); } - console.log('[INFO]', 'CloudFormationStackId: ', ddb_data.Item.CloudFormationStackID.S); - return new Promise(function (resolve, reject) { - var cfn_params = { - StackName: request_payload.stackName, - RoleARN: '${IAMRoleTestEnvironmentCloudFormation.Arn}' - }; - cfn.deleteStack(cfn_params, function(err, cfn_data) { - if (err) { return reject(err); } - return resolve(); - }); - }); - }); - }); - } - }; - exports.handler = function (event, context, callback) { - var p = actions[event.action]; - if (!p) { - return callback('Unknown action'); - } - var msgAction = event.action.toUpperCase() + ' '; - var cfn = new AWS.CloudFormation(); - var ddb = new AWS.DynamoDB(); - console.log('[INFO]', 'Attempting', msgAction); - return p(cfn, ddb, event).then(function (data) { - return callback(null, data); - }).catch(function (err) { - console.log('[ERROR]', JSON.stringify(err)); - return callback(err); - }); - }; - - JenkinsOnDemandEC2Instance: # This workshop starts from a baseline where we have a Jenkins server running on an on-demand EC2 instance. This resource launches and bootstraps this server - Type: AWS::EC2::Instance - DependsOn: - - EC2AMILookupCustomResource - - InstanceProfileJenkins - - SecurityGroupJenkins - - SubnetPublicA - # DependedOn: JenkinsMasterALBTargetGroupEC2 - Properties: - BlockDeviceMappings: - - DeviceName: "/dev/xvda" - Ebs: - DeleteOnTermination: "true" - VolumeSize: 8 - VolumeType: gp2 - IamInstanceProfile: !Ref InstanceProfileJenkins - ImageId: !GetAtt EC2AMILookupCustomResource.Id - InstanceType: t3.medium - KeyName: !Ref KeyPair - SecurityGroupIds: - - !Ref SecurityGroupJenkins - SubnetId: !Ref SubnetPublicA - Tags: - - Key: Name - Value: Jenkins Master (On-demand) - UserData: - Fn::Base64: !Sub | - #!/bin/bash - # Install all pending updates to the system - yum -y update - # Configure YUM to be able to access official Jenkins RPM packages - wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo - # Import the Jenkins repository public key - rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key - # Configure YUM to be able to access contributed Maven RPM packages - wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo - # Update the release version in the Maven repository configuration for this mainline release of Amazon Linux - sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo - # Install the Java 8 SDK, Git, Jenkins and Maven - yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel git apache-maven - yum -y install jenkins-2.138.4-1.1 --nogpgcheck - # Set the default version of java to run out of the Java 8 SDK path (required by Jenkins) - update-alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java - update-alternatives --set javac /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/javac - # Restore the pre-staged copy of JENKINS_HOME used by this workshop - mv /var/lib/jenkins /var/lib/jenkins_orig - mkdir /var/lib/jenkins - chown jenkins:jenkins /var/lib/jenkins - wget -q -O /var/lib/jenkins/jenkins_home.tar.gz https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/jenkins_home.tar.gz - tar -zxf /var/lib/jenkins/jenkins_home.tar.gz -C /var/lib/jenkins/ - # Reset the password for the spotcicdworkshop user in Jenkins to the password defined as the JenkinsAdminPassword - echo -n '${JenkinsAdminPassword}{spot}' | sha256sum | awk '{print $1;}' | xargs -I {} sed -i 's/jbcrypt:$2a$10$21qSY20aYvtAFeWA0yL8AezRox5bExGclhtaFatOokyYWe7CxbRfm/spot:{}/' /var/lib/jenkins/users/spotcicdworkshop/config.xml - # Configure the Jenkins Location - curl -s http://169.254.169.254/latest/meta-data/public-ipv4 | xargs -I {} sed -i 's/203.0.113.0/{}/' /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml - # Start the Jenkins service - service jenkins start - - JenkinsMasterALB: # This is the Application Load Balancer that resides in front of your Jenkins Master instance and is responsible for port-mapping requests from TCP:80 to TCP:8080 - Type: AWS::ElasticLoadBalancingV2::LoadBalancer - DependsOn: - - SecurityGroupJenkinsALB - - SubnetPublicA - - SubnetPublicB - - SubnetPublicC - # DependedOn: JenkinsMasterALBListener - Properties: - Name: JenkinsMasterALB - Scheme: internet-facing - SecurityGroups: - - !Ref SecurityGroupJenkinsALB - Subnets: - - !Ref SubnetPublicA - - !Ref SubnetPublicB - - !Ref SubnetPublicC - - JenkinsMasterALBTargetGroupEC2: # This is the Target Group used by the JenkinsMasterALB load balancer when Jenkins is running on an EC2 instance - Type: AWS::ElasticLoadBalancingV2::TargetGroup - DependsOn: - - JenkinsOnDemandEC2Instance - - VPC - # DependedOn: JenkinsMasterALBListener, JenkinsMasterALBListenerRule - Properties: - HealthCheckIntervalSeconds: 15 - HealthCheckPath: /login - HealthCheckPort: 8080 - HealthCheckProtocol: HTTP - HealthCheckTimeoutSeconds: 5 - HealthyThresholdCount: 2 - Matcher: - HttpCode: 200 - Name: JenkinsMasterEC2TargetGroup - Port: 8080 - Protocol: HTTP - Targets: - - Id: !Ref JenkinsOnDemandEC2Instance - Port: 8080 - UnhealthyThresholdCount: 4 - VpcId: !Ref VPC - - JenkinsMasterALBTargetGroupECS: # This is the Target Group used by the JenkinsMasterALB load balancer when Jenkins is running in a container on an ECS cluster - Type: AWS::ElasticLoadBalancingV2::TargetGroup - DependsOn: - - VPC - # DependedOn: JenkinsMasterALBListener, JenkinsMasterALBListenerRule - Properties: - HealthCheckIntervalSeconds: 15 - HealthCheckPath: /login - HealthCheckPort: 8080 - HealthCheckProtocol: HTTP - HealthCheckTimeoutSeconds: 5 - HealthyThresholdCount: 2 - Matcher: - HttpCode: 200 - Name: JenkinsMasterECSTargetGroup - Port: 8080 - Protocol: HTTP - TargetType: ip - UnhealthyThresholdCount: 4 - VpcId: !Ref VPC - - JenkinsMasterALBListener: # This is the ALB Listener used to access the Jenkins Master - Type: AWS::ElasticLoadBalancingV2::Listener - DependsOn: - - JenkinsMasterALB - - JenkinsMasterALBTargetGroupEC2 - # DepenededOn: None - Properties: - DefaultActions: - - Type: forward - TargetGroupArn: !Ref JenkinsMasterALBTargetGroupEC2 - LoadBalancerArn: !Ref JenkinsMasterALB - Port: 80 - Protocol: HTTP - - JenkinsMasterALBListenerRuleEC2: # The ALB Listener rule that forwards all traffic destined for the Jenkins Master to the appropriate Target Group - Type: AWS::ElasticLoadBalancingV2::ListenerRule - DependsOn: - - JenkinsMasterALBListener - - JenkinsMasterALBTargetGroupEC2 - # DependedOn: None - Properties: - Actions: - - Type: forward - TargetGroupArn: !Ref JenkinsMasterALBTargetGroupEC2 - Conditions: - - Field: path-pattern - Values: - - "/*" - ListenerArn: !Ref JenkinsMasterALBListener - Priority: 1 - - JenkinsMasterALBListenerRuleECS: # The ALB Listener rule that forwards all traffic destined for the Jenkins Master to the appropriate Target Group - Type: AWS::ElasticLoadBalancingV2::ListenerRule - DependsOn: - - JenkinsMasterALBListener - - JenkinsMasterALBTargetGroupECS - # DependedOn: None - Properties: - Actions: - - Type: forward - TargetGroupArn: !Ref JenkinsMasterALBTargetGroupECS - Conditions: - - Field: path-pattern - Values: - - "/*" - ListenerArn: !Ref JenkinsMasterALBListener - Priority: 2 - - JenkinsSpotMasterLaunchTemplate: # This is a launch template that will be used to provision Jenkins Master servers - showing how when used in conjunction with an EFS volume stateful applications can run on self-healing spot architectures. - Type: AWS::EC2::LaunchTemplate - DependsOn: - - EC2AMILookupCustomResource - - EFSJenkinsHomeVolume - - InstanceProfileJenkins - - SecurityGroupJenkins - # DependedOn: None - Properties: - LaunchTemplateName: JenkinsMasterLaunchTemplate - LaunchTemplateData: - BlockDeviceMappings: - - DeviceName: "/dev/xvda" - Ebs: - DeleteOnTermination: "true" - VolumeSize: 8 - VolumeType: gp2 - IamInstanceProfile: - #Arn: !GetAtt InstanceProfileJenkins.Arn - Name: !Ref InstanceProfileJenkins - ImageId: !GetAtt EC2AMILookupCustomResource.Id - InstanceType: t3.medium - KeyName: !Ref KeyPair - SecurityGroupIds: - - !Ref SecurityGroupJenkins - TagSpecifications: - - ResourceType: instance - Tags: - - Key: Name - Value: Jenkins Master (Spot) - UserData: - Fn::Base64: !Sub | - #!/bin/bash - # Install all pending updates to the system - yum -y update - # Configure YUM to be able to access official Jenkins RPM packages - wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo - # Import the Jenkins repository public key - rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key - # Configure YUM to be able to access contributed Maven RPM packages - wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo - # Update the release version in the Maven repository configuration for this mainline release of Amazon Linux - sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo - # Install the Java 8 SDK, Git, Jenkins and Maven - yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel git apache-maven - yum -y install jenkins-2.138.4-1.1 --nogpgcheck - # Set the default version of java to run out of the Java 8 SDK path (required by Jenkins) - update-alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java - update-alternatives --set javac /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/javac - # Mount the Jenkins EFS volume at JENKINS_HOME - mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).${EFSJenkinsHomeVolume}.efs.eu-west-1.amazonaws.com:/ /var/lib/jenkins - # Start the Jenkins service - service jenkins start - - JenkinsSpotAgentLaunchTemplate: # This is a launch template that will be used to provision Jenkins build agents - showing how spot instances can be used to scale-out build jobs at low cost. - Type: AWS::EC2::LaunchTemplate - DependsOn: - - EC2AMILookupCustomResource - - InstanceProfileJenkins - - SecurityGroupJenkins - # DependedOn: None - Properties: - LaunchTemplateName: JenkinsBuildAgentLaunchTemplate - LaunchTemplateData: - BlockDeviceMappings: - - DeviceName: "/dev/xvda" - Ebs: - DeleteOnTermination: "true" - VolumeSize: 8 - VolumeType: gp2 - IamInstanceProfile: - #Arn: !GetAtt InstanceProfileJenkins.Arn - Name: !Ref InstanceProfileJenkins - ImageId: !GetAtt EC2AMILookupCustomResource.Id - InstanceType: t3.small - KeyName: !Ref KeyPair - SecurityGroupIds: - - !Ref SecurityGroupJenkins - TagSpecifications: - - ResourceType: instance - Tags: - - Key: Name - Value: Jenkins Build Agent - UserData: - Fn::Base64: !Sub | - #!/bin/bash - # Install all pending updates to the system - yum -y update - # Configure YUM to be able to access contributed Maven RPM packages - wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo - # Update the release version in the Maven repository configuration for this mainline release of Amazon Linux - sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo - # Install the Java 8 SDK, Git and Maven - yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel git apache-maven - # Set the default version of java to run out of the Java 8 SDK path (required by Jenkins) - update-alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java - update-alternatives --set javac /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/javac - - DynamoDBTestEnvironmentTable: - Type: AWS::DynamoDB::Table - # DependsOn: - # DependedOn: IAMRoleTestEnvironmentLambdaExecution, TestEnvironmentLambdaFunction - Properties: - AttributeDefinitions: - - AttributeName: JobBaseName - AttributeType: "S" - - AttributeName: BuildID - AttributeType: "N" - KeySchema: - - AttributeName: JobBaseName - KeyType: HASH - - AttributeName: BuildID - KeyType: RANGE - ProvisionedThroughput: - ReadCapacityUnits: 5 - WriteCapacityUnits: 5 - TableName: SpotCICDWorkshopTestEnvironmentTracking - - EFSJenkinsHomeVolume: - Type: AWS::EFS::FileSystem - # DependsOn: None - # DependedOn: AutoScalingECSLaunchConfiguration - Properties: - PerformanceMode: generalPurpose - - EFSMountTargetJenkinsHomeVolumeA: - Type: AWS::EFS::MountTarget - DependsOn: - - EFSJenkinsHomeVolume - - SecurityGroupEFS - - SubnetPublicA - # DependedOn: - Properties: - FileSystemId: !Ref EFSJenkinsHomeVolume - SecurityGroups: - - !Ref SecurityGroupEFS - SubnetId: !Ref SubnetPublicA - - EFSMountTargetJenkinsHomeVolumeB: - Type: AWS::EFS::MountTarget - DependsOn: - - EFSJenkinsHomeVolume - - SecurityGroupEFS - - SubnetPublicB - # DependedOn: - Properties: - FileSystemId: !Ref EFSJenkinsHomeVolume - SecurityGroups: - - !Ref SecurityGroupEFS - SubnetId: !Ref SubnetPublicB - - EFSMountTargetJenkinsHomeVolumeC: - Type: AWS::EFS::MountTarget - DependsOn: - - EFSJenkinsHomeVolume - - SecurityGroupEFS - - SubnetPublicC - # DependedOn: - Properties: - FileSystemId: !Ref EFSJenkinsHomeVolume - SecurityGroups: - - !Ref SecurityGroupEFS - SubnetId: !Ref SubnetPublicC - - ECSLaunchTemplate: # This is a launch template that will be used to provision ECS cluster nodes - Type: AWS::EC2::LaunchTemplate - DependsOn: - - ECSAMILookupCustomResource - - EFSJenkinsHomeVolume - - InstanceProfileECS - - SecurityGroupJenkins - # DependedOn: None - Properties: - LaunchTemplateName: ECSLaunchTemplate - LaunchTemplateData: - BlockDeviceMappings: - - DeviceName: "/dev/xvda" - Ebs: - DeleteOnTermination: "true" - VolumeSize: 8 - VolumeType: gp2 - IamInstanceProfile: - Name: !Ref InstanceProfileECS - ImageId: !GetAtt ECSAMILookupCustomResource.Id - InstanceType: t3.medium - KeyName: !Ref KeyPair - SecurityGroupIds: - - !Ref SecurityGroupJenkins - TagSpecifications: - - ResourceType: instance - Tags: - - Key: Name - Value: ECS Cluster Instance - UserData: - Fn::Base64: !Sub | - #!/bin/bash - # Register instance with ECS cluster - echo ECS_CLUSTER=SpotCICDWorkshopECSCluster >> /etc/ecs/ecs.config - # Install all pending updates to the system - yum -y update - # Install the nfs-utils package - yum -y install nfs-utils - # Create EFS mountpoint - mkdir /mnt/efs - mkdir /mnt/efs/jenkins_home - # Mount the Jenkins EFS volume - mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).${EFSJenkinsHomeVolume}.efs.eu-west-1.amazonaws.com:/ /mnt/efs/jenkins_home - # Deploy the AWS CLI - yum -y install unzip wget - wget -q https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -O ./awscli-bundle.zip - unzip ./awscli-bundle.zip - ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws - # Register a custom attribute for this ECS host to indicate if this is a spot instance or not - yum -y install jq - INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id) - INSTANCE_LIFECYCLE=$(/usr/local/bin/aws ec2 describe-instances --region eu-west-1 --instance-ids $INSTANCE_ID | jq .Reservations | jq .[0] | jq .Instances | jq .[0] | jq .InstanceLifecycle | tr -d \") - echo ECS_INSTANCE_ATTRIBUTES={\"lifecycle\": \"$INSTANCE_LIFECYCLE\"} >> /etc/ecs/ecs.config - # If this is a spot instance, ensure that container draining occurs prior to interruption - if [ $INSTANCE_LIFECYCLE == spot ] - then - wget -q https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/interruption_check.sh -O ./interruption_check.sh - chmod +x ./interruption_check.sh - nohup ./interruption_check.sh &>/dev/null & - fi - - ECSCluster: - Type: AWS::ECS::Cluster - DependsOn: ECSServiceLinkedRole - # DependedOn: ECSServiceJenkinsMaster - Properties: - ClusterName: SpotCICDWorkshopECSCluster - - ECSTaskDefinitionJenkinsMaster: - Type: AWS::ECS::TaskDefinition - DependsOn: ECSServiceLinkedRole - # DependedOn: - Properties: - ContainerDefinitions: - - Cpu: 512 - Essential: true - Image: jenkins/jenkins:lts - Memory: 1536 - MountPoints: - - SourceVolume: JENKINS_HOME - ContainerPath: /var/jenkins_home - Name: SpotCICDWorkshopJenkinsMasterContainer - PortMappings: - - ContainerPort: 5000 - HostPort: 5000 - - ContainerPort: 8080 - HostPort: 8080 - - ContainerPort: 50000 - HostPort: 50000 - ExecutionRoleArn: !GetAtt IAMRoleJenkins.Arn - NetworkMode: awsvpc - PlacementConstraints: - - Type: memberOf - Expression: attribute:lifecycle != spot - Volumes: - - Host: - SourcePath: /mnt/efs/jenkins_home - Name: JENKINS_HOME - - ECSServiceJenkinsMaster: - Type: AWS::ECS::Service - DependsOn: - - ECSCluster - - ECSServiceLinkedRole - - ECSTaskDefinitionJenkinsMaster - # - IAMRoleECSServiceRole - - JenkinsMasterALBListenerRuleECS - - ServiceDiscoveryJenkinsMaster - # # DependedOn: None - Properties: - Cluster: !Ref ECSCluster - DeploymentConfiguration: - MaximumPercent: 100 - MinimumHealthyPercent: 50 - DesiredCount: 0 - HealthCheckGracePeriodSeconds: 120 - LoadBalancers: - - ContainerName: SpotCICDWorkshopJenkinsMasterContainer - ContainerPort: 8080 - TargetGroupArn: !Ref JenkinsMasterALBTargetGroupECS - NetworkConfiguration: - AwsvpcConfiguration: - SecurityGroups: - - !Ref SecurityGroupJenkins - Subnets: - - !Ref SubnetPrivateA - - !Ref SubnetPrivateB - - !Ref SubnetPrivateC - # Role: !GetAtt IAMRoleECSServiceRole.Arn - # ServiceName: JenkinsMaster - ServiceRegistries: - - ContainerName: SpotCICDWorkshopJenkinsMasterContainer - RegistryArn: !GetAtt ServiceDiscoveryJenkinsMaster.Arn - TaskDefinition: !Ref ECSTaskDefinitionJenkinsMaster - - ServiceDiscoveryJenkinsMasterNamespace: - Type: AWS::ServiceDiscovery::PrivateDnsNamespace - DependsOn: VPC - # DependedOn: None - Properties: - Vpc: !Ref VPC - Name: jenkins.local - - ServiceDiscoveryJenkinsMaster: - Type: AWS::ServiceDiscovery::Service - DependsOn: ServiceDiscoveryJenkinsMasterNamespace - # DependedOn: ECSServiceJenkinsMaster - Properties: - Description: Jenkins Master Service - DnsConfig: - DnsRecords: - - Type: A - TTL: 60 - NamespaceId: !Ref ServiceDiscoveryJenkinsMasterNamespace - HealthCheckCustomConfig: - FailureThreshold: 1 - Name: master - -Outputs: - DeploymentArtifactsS3Bucket: - Description: The name of the S3 bucket where deployment artifacts are uploaded to. - Value: !Ref DeploymentArtifactsS3Bucket - - EFSFileSystemID: - Description: The file system ID of the EFS volume that is used to persist JENKINS_HOME across EC2 & ECS instances. - Value: !Ref EFSJenkinsHomeVolume - - JenkinsDNSName: - Description: The DNS name of the Application Load Balancer that is used to gain access to your Jenkins server. - Value: !GetAtt JenkinsMasterALB.DNSName - - JenkinsIAMRoleARN: - Description: The ARN associated with the IAM Role that was created for use by Jenkins. - Value: !GetAtt IAMRoleJenkins.Arn - - JenkinsMasterSecurityGroup: - Description: Security Group for Jenkins nodes. Use this value to configure Jenkins ECS Plugin - Value: !Ref SecurityGroupJenkins - - JenkinsVPCPublicSubnets: - Description: The public subnets where Jenkins will be deployed. Use this value to configure Jenkins ECS plugin - Value: - !Join [",", [!Ref SubnetPublicA, !Ref SubnetPublicB, !Ref SubnetPublicC]] diff --git a/workshops/amazon-ec2-spot-cicd-workshop/container/Dockerfile b/workshops/amazon-ec2-spot-cicd-workshop/container/Dockerfile new file mode 100644 index 00000000..f53d692e --- /dev/null +++ b/workshops/amazon-ec2-spot-cicd-workshop/container/Dockerfile @@ -0,0 +1,15 @@ +FROM jenkins/jenkins:2.361.2-lts-jdk11 + +ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false +ENV CASC_JENKINS_CONFIG /usr/share/jenkins/ref/casc.yaml + +COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt +RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt + +COPY casc.yaml /usr/share/jenkins/ref/casc.yaml + +WORKDIR /jenkins_seed +RUN curl https://raw.githubusercontent.com/christianhxc/ec2-spot-workshops/cicd-upgrade/workshops/amazon-ec2-spot-cicd-workshop/jobs/seed.tar.gz -o seed.tar.gz \ + && tar -xvf seed.tar.gz \ + && mkdir /usr/share/jenkins/ref/jobs \ + && mv jobs/* /usr/share/jenkins/ref/jobs/ \ No newline at end of file diff --git a/workshops/amazon-ec2-spot-cicd-workshop/container/casc.yaml b/workshops/amazon-ec2-spot-cicd-workshop/container/casc.yaml new file mode 100644 index 00000000..ae36ae02 --- /dev/null +++ b/workshops/amazon-ec2-spot-cicd-workshop/container/casc.yaml @@ -0,0 +1,24 @@ +jenkins: + securityRealm: + local: + allowsSignup: false + users: + - id: ${JENKINS_ADMIN_ID} + password: ${JENKINS_ADMIN_PASSWORD} + authorizationStrategy: + globalMatrix: + permissions: + - "USER:Overall/Administer:admin" + - "GROUP:Overall/Read:authenticated" + remotingSecurity: + enabled: true + +security: + queueItemAuthenticator: + authenticators: + - global: + strategy: triggeringUsersAuthorizationStrategy + +unclassified: + location: + url: http://localhost:8080/ diff --git a/workshops/amazon-ec2-spot-cicd-workshop/container/docker-compose.yml b/workshops/amazon-ec2-spot-cicd-workshop/container/docker-compose.yml new file mode 100644 index 00000000..c7dbe5c8 --- /dev/null +++ b/workshops/amazon-ec2-spot-cicd-workshop/container/docker-compose.yml @@ -0,0 +1,16 @@ +version: '3.2' + +services: + jenkins: + image: christianhxc/amazon-ec2-spot-cicd-workshop-jenkins:2.361.2 + privileged: true + user: root + build: . + environment: + JENKINS_ADMIN_ID: admin + JENKINS_ADMIN_PASSWORD: JenkinsAdminPassword + volumes: + - /jenkins_home:/var/jenkins_home + ports: + - 8080:8080 + restart: always diff --git a/workshops/amazon-ec2-spot-cicd-workshop/container/plugins.txt b/workshops/amazon-ec2-spot-cicd-workshop/container/plugins.txt new file mode 100644 index 00000000..397f1ea6 --- /dev/null +++ b/workshops/amazon-ec2-spot-cicd-workshop/container/plugins.txt @@ -0,0 +1,23 @@ +ant:475.vf34069fef73c +antisamy-markup-formatter:2.7 +build-timeout:1.21 +cloudbees-folder:6.729.v2b_9d1a_74d673 +configuration-as-code:1512.vb_79d418d5fc8 +credentials-binding:523.vd859a_4b_122e6 +email-ext:2.90 +git:4.11.5 +github-branch-source:1677.v731f745ea_0cf +gradle:1.39.4 +ldap:2.10 +mailer:435.v79ef3972b_5c7 +matrix-auth:3.1.5 +pam-auth:1.8 +pipeline-github-lib:38.v445716ea_edda_ +pipeline-stage-view:2.24 +ssh-slaves:1.834.v622da_57f702c +timestamper:1.18 +workflow-aggregator:590.v6a_d052e5a_a_b_5 +ws-cleanup:0.42 +authorize-project:1.4.0 +ec2-fleet:2.5.1 +naginator:1.18.1 \ No newline at end of file diff --git a/workshops/amazon-ec2-spot-cicd-workshop/fisspotinterruption.yaml b/workshops/amazon-ec2-spot-cicd-workshop/fisspotinterruption.yaml new file mode 100644 index 00000000..d26aa8bf --- /dev/null +++ b/workshops/amazon-ec2-spot-cicd-workshop/fisspotinterruption.yaml @@ -0,0 +1,69 @@ +--- +AWSTemplateFormatVersion: 2010-09-09 +Description: FIS for Spot Instances +Parameters: + InstancesToInterrupt: + Description: Number of instances to interrupt + Default: 3 + Type: Number + + DurationBeforeInterruption: + Description: Number of minutes before the interruption + Default: 2 + Type: Number + +Resources: + FISSpotRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Effect: Allow + Principal: + Service: [fis.amazonaws.com] + Action: ["sts:AssumeRole"] + Path: / + Policies: + - PolicyName: root + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: "ec2:DescribeInstances" + Resource: "*" + - Effect: Allow + Action: "ec2:SendSpotInstanceInterruptions" + Resource: "arn:aws:ec2:*:*:instance/*" + + FISExperimentTemplate: + Type: AWS::FIS::ExperimentTemplate + Properties: + Description: "Interrupt multiple random instances" + Targets: + SpotIntances: + ResourceTags: + Name: "jenkins-build-agent" + Filters: + - Path: State.Name + Values: + - running + ResourceType: aws:ec2:spot-instance + SelectionMode: !Join ["", ["COUNT(", !Ref InstancesToInterrupt, ")"]] + Actions: + interrupt: + ActionId: "aws:ec2:send-spot-instance-interruptions" + Description: "Interrupt multiple Spot instances" + Parameters: + durationBeforeInterruption: + !Join ["", ["PT", !Ref DurationBeforeInterruption, "M"]] + Targets: + SpotInstances: SpotIntances + StopConditions: + - Source: none + RoleArn: !GetAtt FISSpotRole.Arn + Tags: + Name: "fis-spot-jenkins-interruption" + +Outputs: + FISExperimentID: + Value: !GetAtt FISExperimentTemplate.Id diff --git a/workshops/amazon-ec2-spot-cicd-workshop/jobs/seed.tar.gz b/workshops/amazon-ec2-spot-cicd-workshop/jobs/seed.tar.gz new file mode 100644 index 00000000..8a4669b4 Binary files /dev/null and b/workshops/amazon-ec2-spot-cicd-workshop/jobs/seed.tar.gz differ