Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamping the CI/CD Workshop #203

Open
wants to merge 31 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
49a1acf
Modified the CF to configure Jenkins using Docker
chrismld Jul 21, 2022
66830b5
Fixed problems with the CF for the CI/CD Workshop
chrismld Jul 22, 2022
9ba5f2a
Updated the instructions for theh CI/CD Workshop lab 1
chrismld Jul 27, 2022
2e886c4
Updated the instructions for theh CI/CD Workshop lab 1
chrismld Jul 27, 2022
7cf828b
Fixed problems with the CF for the CI/CD Workshop
chrismld Jul 28, 2022
2b973f8
Fixed problems with the CF for the CI/CD Workshop
chrismld Jul 28, 2022
d51ffd8
Fixed problems with the CF for the CI/CD Workshop
chrismld Jul 28, 2022
fce5f5b
Fixed problems with the CF for the CI/CD Workshop
chrismld Jul 28, 2022
16f4d60
Re-structure labs for CI/CD Workshop
chrismld Aug 1, 2022
dfed6ed
Re-structure labs for CI/CD Workshop
chrismld Aug 1, 2022
0b782de
Split instructions for Jenkins on ASG for CI/CD Workshop
chrismld Aug 1, 2022
92fc055
Re-structure labs for CI/CD Workshop
chrismld Aug 1, 2022
ce075f2
Updated the persistence lab for the CI/CD Workshop
chrismld Aug 1, 2022
4be6b22
Updated the persistence lab for the CI/CD Workshop
chrismld Aug 2, 2022
064bb50
Re-structure labs for CI/CD Workshop
chrismld Aug 3, 2022
e1ee431
Merge branch 'awslabs:master' into cicd-upgrade
chrismld Aug 3, 2022
87860de
Add ABS to the CI/CD Workshop
chrismld Aug 9, 2022
8d00493
Merge branch 'awslabs:master' into cicd-upgrade
chrismld Aug 9, 2022
48dd933
Add FIS to the CI/CD Workshop
chrismld Aug 10, 2022
2517283
Add FIS to the CI/CD Workshop
chrismld Aug 10, 2022
0926d46
Upgraded the CNF for the ECS lab in the CI/CD Workshop
chrismld Aug 16, 2022
2580a1b
Using the Docker Hub container image
chrismld Sep 5, 2022
897672d
Using the Docker Hub container image
chrismld Sep 5, 2022
f7ae1ed
Using the Docker Hub container image
chrismld Sep 5, 2022
8feb6de
Using the Docker Hub container image
chrismld Sep 5, 2022
99a9809
Updating the CI/CD CFN to use ASG
chrismld Sep 6, 2022
77330d8
Remove the persistence lab for the CI/CD workshop
chrismld Sep 6, 2022
6f085d5
Breaking down the ECS labs from the CI/CD workshop
chrismld Sep 6, 2022
f697b8d
Added a note of why the ASG starts with zero instances in the CI/CD w…
chrismld Sep 6, 2022
8be11ac
Updated to the latest Jenkins version
chrismld Oct 19, 2022
a9e96dd
Fixed problems when installing the Jenkins agent
chrismld Nov 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 9 additions & 20 deletions content/amazon-ec2-spot-cicd-workshop/_index.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,22 @@
---
title: "CI/CD and Test Workloads with EC2 Spot Instances"
menuTitle: "CI/CD and Test Workloads"
title: "CI/CD Workloads with EC2 Spot Instances"
menuTitle: "CI/CD Workloads"
date: 2019-02-19T02:02:35
weight: 80
pre: "<b>8. </b>"
---

## Overview
During this workshop, you'll get hands-on with Amazon EC2 Spot and discover architectural best practices through the lens of DevOps and CI/CD. You'll deploy Jenkins build agents and test environments on Spot instances at a fraction of the cost of on-demand instances. You'll also implement mechanisms to ensure that your CI/CD tooling recovers from spot market events by decoupling application state from your compute resources. Finally, you'll migrate your CI/CD environment to a containered environment to eke out maximum performance and cost efficiency. In addition to covering the ins and outs of Spot, we'll share some of the Spot-based mechanisms used by customers to reduce the cost of their test and production workloads.
In this workshop, you'll get hands-on with Spot instances and discover architectural best practices through the lens of DevOps and CI/CD. We'll dive dive deep on how to deploy tools like Jenkins and use Spot instances as build agents. You'll also implement mechanisms to ensure that your CI/CD tooling recovers from Spot interruptions by simulating failures and decoupling application state from your compute resources. Moreover, you'll migrate your CI/CD environment to a containerized environments using ECS to eke out maximum performance and cost efficiency. In addition to covering the best practices to use Spot, we'll share some of the Spot-based mechanisms used by customers to optimize their infrastructure resources.

## Workshop Details
This workshop will be broken down into a series of labs that flow on from each other (that is, you must complete each lab in order before proceeding with the next). The lab exercises that will be covered are:
## Workshop Labs
This workshop will be broken down into a series of labs using differenct CI/CD tools and AWS services, topics covered are:

* Workshop preparation: Deploy pre-requisite resources through Amazon CloudFormation;
* Lab 1: Reduce the cost of builds using Amazon EC2 Spot Fleet;
* Lab 2: Deploy testing environments using Amazon EC2 Spot, Amazon CloudFormation & Amazon EC2 Launch Templates;
* Lab 3: Externalize state data to add resiliency and reduce cost for your CI/CD tooling;
* Lab 4: Using containers backed by Auto Scaling Groups comprised of both on-demand and Spot instances;
* Workshop clean up.
* [Jenkins with Auto Scaling groups](/amazon-ec2-spot-cicd-workshop/jenkins-asg.html)
* [Jenkins with ECS](/amazon-ec2-spot-cicd-workshop/jenkins-ecs.html)

{{% notice note %}}
As a reminder, you should have a laptop device (Windows/OSX/Linux are supported - tablets are not appropriate) with the current version of Google Chrome or Mozilla Firefox installed. You should also have a clean AWS account, with **AdministratorAccess** policy-level access.
{{% /notice %}}

This workshop should take between two and three hours to complete, depending on your proficiency with the AWS services being featured.

#### Additional considerations when running this workshop in a corporate IT environment
If you are running this workshop from a corporate IT environment, contact your Systems Administrator to ensure that you will be able to establish outbound Secure Shell (SSH) connections to an Internet host:

* If you cannot establish SSH connections to Internet hosts (and do not have a suitable workaround), you will not be able to complete Labs 3 & 4;
* If you can establish SSH connections to Internet hosts, obtain from your Systems Administrator the source IP address CIDR block that connections will be established from.

If you access the Internet through a transparent proxy server running in your corporate IT environment and this proxy server uses a different source address than where SSH connections come from, additional configuration of AWS Security Groups will need to be carried out. The lab guide will indicate the configuration steps required when appropriate.

83 changes: 0 additions & 83 deletions content/amazon-ec2-spot-cicd-workshop/clea.md

This file was deleted.

4 changes: 4 additions & 0 deletions content/amazon-ec2-spot-cicd-workshop/conclusion.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
+++
title = "Conclusion"
weight = 1000
+++
18 changes: 18 additions & 0 deletions content/amazon-ec2-spot-cicd-workshop/jenkins-asg/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
+++
title = "Jenkins with Auto Scaling groups"
weight = 100
+++
By default, all job builds are executed on the same instance that Jenkins is running on. This results in a couple of less-than-desirable behaviours:
* When CPU-intensive builds are being executed, there may not be sufficient system resources to display the Jenkins server interface; and
* The Jenkins server is often provisioned with more resources than the server interface requires in order to allow builds to execute. When builds are not being executed, these server resources are essentially going to waste.

To address these behaviours, Jenkins provides the capability to execute builds on external hosts (called build agents). Further, AWS provides a Jenkins plugin to allow Jenkins to scale out a fleet of EC2 instances in order to execute build jobs on. This lab will focus on implementing EC2 Spot build agents, showcasing what a batch processing workload typically looks like when using Amazon EC2 Spot instances.

## What we'll be doing?
The purpose of these labs is to help you understand how to configure build agents using Spot instances. We'll focus on configuring a Jenkins plugin so that build agents are launched only when you need them. At the end of these labs, you'll be able to:

* Learn how to launch Spot instances following our best practices using Auto Scaling groups
* Configure properly the EC2 Fleet Jenkins plugin
* Configure Jenkins build jobs to use Spot instances as agents
* Simulate Spot interruption events to increase fault-tolerance
* Test how resilient and fault-toleran Jenkins could be using Spot instances
Loading