feat: Adds a commented call to the jenkins-agent submodule (terraform…

…-google-modules#95)

0-bootstrap/README.md

@@ -6,7 +6,7 @@ The purpose of this step is to bootstrap a GCP organization, creating all the re
 
 1. A GCP [Organization](https://cloud.google.com/resource-manager/docs/creating-managing-organization)
 1. A GCP [Billing Account](https://cloud.google.com/billing/docs/how-to/manage-billing-account)
-1. Cloud Identity / Gsuite groups for organization and billing admins
+1. Cloud Identity / G Suite groups for organization and billing admins
 1. Membership in the `group_org_admins` group for user running terraform
 1. Grant the roles mentioned in bootstrap [README.md](https://github.com/terraform-google-modules/terraform-google-bootstrap#permissions), as well as `roles/resourcemanager.folderCreator` for the user running the step.
 
@@ -32,7 +32,7 @@ Further details of permissions required and resources created, can be found in t
 | group\_billing\_admins | Google Group for GCP Billing Administrators | string | n/a | yes |
 | group\_org\_admins | Google Group for GCP Organization Administrators | string | n/a | yes |
 | org\_id | GCP Organization ID | string | n/a | yes |
-| org\_project\_creators | Additional list of members to have project creator role accross the organization. Prefix of group: user: or serviceAccount: is required. | list(string) | `<list>` | no |
+| org\_project\_creators | Additional list of members to have project creator role across the organization. Prefix of group: user: or serviceAccount: is required. | list(string) | `<list>` | no |
 | parent\_folder | Optional - if using a folder for testing. | string | `""` | no |
 | skip\_gcloud\_download | Whether to skip downloading gcloud (assumes gcloud is already available outside the module) | bool | `"true"` | no |
 
@@ -54,6 +54,12 @@ Further details of permissions required and resources created, can be found in t
 
 ## Requirements
 
+If you are using `jenkins_bootstrap`, please see the [README](./modules/jenkins-agent/README.md) for the requirements.
+
+## Instructions
+
+If you are using `jenkins_bootstrap`, please follow the instructions on how to run the bootstrap step with the `jenkins_bootstrap` sub-module described in the [README](./modules/jenkins-agent/README.md), which include implementing VPN, configuring your Jenkins Master among other steps.
+
 ### Software
 
 -   [gcloud sdk](https://cloud.google.com/sdk/install) >= 206.0.0

0-bootstrap/main.tf

@@ -89,6 +89,7 @@ module "seed_bootstrap" {
   ]
 }
 
+// Comment-out the cloudbuild_bootstrap module and its outputs if you want to use Jenkins instead of Cloud Build
 module "cloudbuild_bootstrap" {
   source                    = "terraform-google-modules/bootstrap/google//modules/cloudbuild"
   version                   = "~> 1.2"
@@ -119,3 +120,23 @@ module "cloudbuild_bootstrap" {
     "prod"
   ]
 }
+
+//// Un-comment the jenkins_bootstrap module and its outputs if you want to use Jenkins instead of Cloud Build
+//module "jenkins_bootstrap" {
+//  source                                  = "./modules/jenkins-agent"
+//  org_id                                  = var.org_id
+//  folder_id                               = google_folder.seed.id
+//  billing_account                         = var.billing_account
+//  group_org_admins                        = var.group_org_admins
+//  default_region                          = var.default_region
+//  terraform_sa_email                      = module.seed_bootstrap.terraform_sa_email
+//  terraform_sa_name                       = module.seed_bootstrap.terraform_sa_name
+//  terraform_state_bucket                  = module.seed_bootstrap.gcs_bucket_tfstate
+//  sa_enable_impersonation                 = true
+//  jenkins_master_ip_addresses             = var.jenkins_master_ip_addresses
+//  jenkins_agent_gce_subnetwork_cidr_range = var.jenkins_agent_gce_subnetwork_cidr_range
+//  jenkins_agent_gce_private_ip_address    = var.jenkins_agent_gce_private_ip_address
+//  nat_bgp_asn                             = var.nat_bgp_asn
+//  jenkins_agent_sa_email                  = var.jenkins_agent_sa_email
+//  jenkins_agent_gce_ssh_pub_key           = var.jenkins_agent_gce_ssh_pub_key
+//}

0-bootstrap/modules/jenkins-agent/files/jenkins_gce_startup_script.sh

@@ -15,20 +15,20 @@
 
 #!/bin/sh
 
-echo "**** Startup Step 1/6: Update apt-get repositories. ****"
+echo "**** Startup Step 1/9: Update apt-get repositories. ****"
 apt-get update
 
-echo "**** Startup Step 2/6: Install Java. Needed to accept jobs from Jenkins Master. ****"
+echo "**** Startup Step 2/9: Install Java. Needed to accept jobs from Jenkins Master. ****"
 apt-get install -y default-jdk
 
-echo "**** Startup Step 3/6: Install tools needed to run pipeline commands. ****"
+echo "**** Startup Step 3/9: Install tools needed to run pipeline commands. ****"
 apt-get install -y git jq unzip google-cloud-sdk google-cloud-sdk
 
-echo "**** Startup Step 4/6: Create a directory to locate Terraform binaries. ****"
+echo "**** Startup Step 4/9: Create a directory to locate Terraform binaries. ****"
 # shellcheck disable=SC2154
 mkdir -p "${tpl_TERRAFORM_DIR}" && cd "${tpl_TERRAFORM_DIR}" || exit
 
-echo "**** Startup Step 5/6: Download, verify and unzip Terraform binaries. ****"
+echo "**** Startup Step 5/9: Download, verify and unzip Terraform binaries. ****"
 # shellcheck disable=SC2154
 wget "https://releases.hashicorp.com/terraform/${tpl_TERRAFORM_VERSION}/terraform_${tpl_TERRAFORM_VERSION}_linux_amd64.zip" && \
     echo "${tpl_TERRAFORM_VERSION_SHA256SUM} terraform_${tpl_TERRAFORM_VERSION}_linux_amd64.zip" > terraform_SHA256SUMS && \
@@ -41,7 +41,44 @@ wget "https://releases.hashicorp.com/terraform/${tpl_TERRAFORM_VERSION}/terrafor
     apt-get clean && \
     rm -rf /var/lib/apt/lists/*
 
-echo "**** Startup Step 6/6: Download and install the Terraform validator ****"
+echo "**** Startup Step 6/9: Download and install the Terraform validator ****"
 gsutil cp gs://terraform-validator/releases/2019-04-04/terraform-validator-linux-amd64 .
 chmod 755 "${tpl_TERRAFORM_DIR}terraform-validator-linux-amd64"
 mv "${tpl_TERRAFORM_DIR}terraform-validator-linux-amd64" "${tpl_TERRAFORM_DIR}terraform-validator"
+
+echo "**** Startup Step 7/9: Set the Linux PATH to point to the Terraform directory. ****"
+export PATH=$PATH:${tpl_TERRAFORM_DIR}
+
+echo "**** Startup Step 8/9: Create jenkins user. ****"
+# Home directory for the jenkins user
+JENKINS_HOME_DIR="/home/jenkins"
+
+# The "Remote Jenkins directory" is used to store Jenkins Agent logs
+JENKINS_AGENT_REMOTE_DIR="$JENKINS_HOME_DIR/jenkins_agent_dir"
+
+# Create  and set the jenkins user as owner
+mkdir -p "$JENKINS_AGENT_REMOTE_DIR" && \
+  chmod 766 "$JENKINS_AGENT_REMOTE_DIR" && \
+
+useradd -d /home/jenkins jenkins
+chown -R jenkins:jenkins /home/jenkins/
+
+echo "**** Startup Step 9/9: Add public SSH key to the list of authorized keys. ****"
+SSHD_CONFIG_DIR="/etc/ssh"
+
+# Setting up the sshd_config file
+sed -i $SSHD_CONFIG_DIR/sshd_config \
+        -e 's/#PubkeyAuthentication.*/PubkeyAuthentication yes/' \
+        -e 's/#AuthorizedKeysFile.*/AuthorizedKeysFile    \/etc\/ssh\/authorized_keys/'
+
+# The Jenkins Agent needs the Master public key. This can be in your github repo
+# shellcheck disable=SC2154
+cat > $SSHD_CONFIG_DIR/authorized_keys <<-EOT
+${tpl_SSH_PUB_KEY}
+EOT
+
+# Configure secure permissions on SSHD_CONFIG_DIR
+chmod 755 $SSHD_CONFIG_DIR && \
+ chmod 655 $SSHD_CONFIG_DIR/authorized_keys
+
+systemctl restart sshd

0-bootstrap/modules/jenkins-agent/main.tf

@@ -59,6 +59,7 @@ data "template_file" "jenkins_agent_gce_startup_script" {
     tpl_TERRAFORM_DIR               = "/usr/local/bin/"
     tpl_TERRAFORM_VERSION           = var.terraform_version
     tpl_TERRAFORM_VERSION_SHA256SUM = var.terraform_version_sha256sum
+    tpl_SSH_PUB_KEY                 = var.jenkins_agent_gce_ssh_pub_key
   }
 }
 
@@ -110,14 +111,14 @@ resource "google_compute_instance" "jenkins_agent_gce_instance" {
 *******************************************/
 
 resource "google_compute_firewall" "fw_allow_ssh_into_jenkins_agent" {
-  project       = module.cicd_project.project_id
-  name          = "fw-${google_compute_network.jenkins_agents.name}-1000-i-a-all-all-tcp-22"
-  description   = "Allow the Jenkins Master (Client) to connect to the Jenkins Agents (Servers) using SSH."
-  network       = google_compute_network.jenkins_agents.name
-  source_ranges = var.jenkins_master_ip_addresses
-  target_tags   = local.jenkins_gce_fw_tags
-  priority      = 1000
-
+  project        = module.cicd_project.project_id
+  name           = "fw-${google_compute_network.jenkins_agents.name}-1000-i-a-all-all-tcp-22"
+  description    = "Allow the Jenkins Master (Client) to connect to the Jenkins Agents (Servers) using SSH."
+  network        = google_compute_network.jenkins_agents.name
+  source_ranges  = var.jenkins_master_ip_addresses
+  target_tags    = local.jenkins_gce_fw_tags
+  priority       = 1000
+  enable_logging = true
   allow {
     protocol = "tcp"
     ports    = ["22"]
@@ -165,7 +166,7 @@ resource "google_compute_router" "nat_router_region1" {
 }
 
 resource "google_compute_address" "nat_external_addresses1" {
-  name    = "ca-${google_compute_network.jenkins_agents.name}-${var.default_region}"
+  name    = "cn-${google_compute_network.jenkins_agents.name}-${var.default_region}"
   project = module.cicd_project.project_id
   region  = var.default_region
 }

0-bootstrap/modules/jenkins-agent/variables.tf

@@ -73,7 +73,6 @@ variable "jenkins_agent_gce_ssh_user" {
 variable "jenkins_agent_gce_ssh_pub_key" {
   description = "SSH public key needed by the Jenkins Agent GCE Instance. The Jenkins Master holds the SSH private key. The correct format is `'ssh-rsa [KEY_VALUE] [USERNAME]'`"
   type        = string
-  default     = ""
 }
 
 variable "jenkins_agent_sa_email" {
@@ -83,7 +82,7 @@ variable "jenkins_agent_sa_email" {
 }
 
 variable "jenkins_master_ip_addresses" {
-  description = "A list of IP Addresses and masks of the Jenkins Master in the form ['0.0.0.0/0']. Needed to create a FW rule that allows communication with the Jenkins Agent GCE Instance."
+  description = "A list of CIDR IP ranges of the Jenkins Master in the form ['0.0.0.0/0']. Usually only one IP in the form '0.0.0.0/32'. Needed to create a FW rule that allows communication with the Jenkins Agent GCE Instance."
   type        = list(string)
 }
 

0-bootstrap/outputs.tf

@@ -34,6 +34,10 @@ output "gcs_bucket_tfstate" {
   value       = module.seed_bootstrap.gcs_bucket_tfstate
 }
 
+/* ----------------------------------------
+    Specific to cloudbuild_module
+   ---------------------------------------- */
+// Comment-out the cloudbuild_bootstrap module and its outputs if you want to use Jenkins instead of Cloud Build
 output "cloudbuild_project_id" {
   description = "Project where CloudBuild configuration and terraform container image will reside."
   value       = module.cloudbuild_bootstrap.cloudbuild_project_id
@@ -59,3 +63,31 @@ output "kms_crypto_key" {
   value       = module.cloudbuild_bootstrap.kms_crypto_key
 }
 
+/* ----------------------------------------
+    Specific to jenkins_bootstrap module
+   ---------------------------------------- */
+//// Un-comment the jenkins_bootstrap module and its outputs if you want to use Jenkins instead of Cloud Build
+//output "cicd_project_id" {
+//  description = "Project where the cicd pipeline (Jenkins Agents and terraform builder container image) reside."
+//  value       = module.jenkins_bootstrap.cicd_project_id
+//}
+//
+//output "jenkins_agent_gce_instance_id" {
+//  description = "Jenkins Agent GCE Instance id."
+//  value       = module.jenkins_bootstrap.jenkins_agent_gce_instance_id
+//}
+//
+//output "jenkins_agent_sa_email" {
+//  description = "Email for privileged custom service account for Jenkins Agent GCE instance."
+//  value       = module.jenkins_bootstrap.jenkins_agent_sa_email
+//}
+//
+//output "jenkins_agent_sa_name" {
+//  description = "Fully qualified name for privileged custom service account for Jenkins Agent GCE instance."
+//  value       = module.jenkins_bootstrap.jenkins_agent_sa_name
+//}
+//
+//output "gcs_bucket_jenkins_artifacts" {
+//  description = "Bucket used to store Jenkins artifacts in Jenkins project."
+//  value       = module.jenkins_bootstrap.gcs_bucket_jenkins_artifacts
+//}

0-bootstrap/variables.tf

@@ -47,7 +47,7 @@ variable "parent_folder" {
 }
 
 variable "org_project_creators" {
-  description = "Additional list of members to have project creator role accross the organization. Prefix of group: user: or serviceAccount: is required."
+  description = "Additional list of members to have project creator role across the organization. Prefix of group: user: or serviceAccount: is required."
   type        = list(string)
   default     = []
 }
@@ -57,3 +57,38 @@ variable "skip_gcloud_download" {
   type        = bool
   default     = true
 }
+
+/////* ----------------------------------------
+////    Specific to jenkins_bootstrap module
+////   ---------------------------------------- */
+//// Un-comment the jenkins_bootstrap module and its outputs if you want to use Jenkins instead of Cloud Build
+//variable "jenkins_agent_gce_subnetwork_cidr_range" {
+//  description = "The subnetwork to which the Jenkins Agent will be connected to (in CIDR range 0.0.0.0/0)"
+//  type        = string
+//}
+//
+//variable "jenkins_agent_gce_private_ip_address" {
+//  description = "The private IP Address of the Jenkins Agent. This IP Address must be in the CIDR range of `jenkins_agent_gce_subnetwork_cidr_range` and be reachable through the VPN that exists between on-prem (Jenkins Master) and GCP (CICD Project, where the Jenkins Agent is located)."
+//  type        = string
+//}
+//
+//variable "jenkins_agent_gce_ssh_pub_key" {
+//  description = "SSH public key needed by the Jenkins Agent GCE Instance. The Jenkins Master holds the SSH private key. The correct format is `'ssh-rsa [KEY_VALUE] [USERNAME]'`"
+//  type        = string
+//}
+//
+//variable "jenkins_agent_sa_email" {
+//  description = "Email for Jenkins Agent service account."
+//  type        = string
+//  default     = "jenkins-agent-gce"
+//}
+//
+//variable "jenkins_master_ip_addresses" {
+//  description = "A list of CIDR IP ranges of the Jenkins Master in the form ['0.0.0.0/0']. Usually only one IP in the form '0.0.0.0/32'. Needed to create a FW rule that allows communication with the Jenkins Agent GCE Instance."
+//  type        = list(string)
+//}
+//
+//variable "nat_bgp_asn" {
+//  type        = number
+//  description = "BGP ASN for NAT cloud route. This is needed to allow the Jenkins Agent to download packages and updates from the internet without having an external IP address."
+//}

README.md

@@ -9,8 +9,28 @@ Each of these Terraform projects are to be layered on top of each other, running
 
 ### [0. bootstrap](./0-bootstrap/)
 
-This stage executes the [CFT Bootstrap module](https://github.com/terraform-google-modules/terraform-google-bootstrap) which bootstraps an existing GCP organization, creating all the required GCP resources & permissions to start using the Cloud Foundation Toolkit (CFT).
-This includes; projects, service accounts and a Terraform state bucket. After executing this step, you will have the following structure:
+This stage executes the [CFT Bootstrap module](https://github.com/terraform-google-modules/terraform-google-bootstrap) which bootstraps an existing GCP organization, creating all the required GCP resources & permissions to start using the Cloud Foundation Toolkit (CFT). You can use either of these two tools for your CICD pipelines: Cloud Build (by default) or Jenkins. If you want to use Jenkins instead of Cloud Build, please:
+   - Comment-out the `cloudbuild_bootstrap` module in `main.tf`, its variables in `variables.tf` and its outputs in `outputs.tf`
+   - Un-comment the `jenkins_bootstrap` module in `main.tf`, its variables in `variables.tf` and its outputs in `outputs.tf`
+   - Follow the instructions on how to run the bootstrap step with the `jenkins_bootstrap` sub-module described in the [README](./0-bootstrap/modules/jenkins-agent/README.md), which include implementing VPN, configuring your Jenkins Master among other steps.
+
+The bootstrap step includes:
+- The `cft-seed` project, which contains:
+  - Terraform state bucket
+  - Custom Service Account used by Terraform to create new resources in GCP
+- The `cft-cloudbuild` project (`prj-cicd` if using Jenkins), which contains:
+  - A CICD pipeline implemented with either Cloud Build or Jenkins
+  - If using Cloud Build:
+    - Cloud Source Repository
+  - If using Jenkins:
+    - Custom Service Account to run Jenkins Agents GCE instances
+    - VPN connection with on-prem (or where ever your Jenkins Master is located)
+
+It is a best practice to have two separate projects here (`cft-seed` and `prj-cicd`) for separation of concerns. On one hand, `cft-seed` stores terraform state and has the Service Account able to create / modify infrastructure. On the other hand, the deployment of that infrastructure is coordinated by a tool of your choice (either Cloud Build or Jenkins), which is implemented in `prj-cicd`.
+
+If using Cloud Build, its default service account `@cloudbuild.gserviceaccount.com` is granted access to generate tokens over the Terraform custom service account. If using Jenkins, the custom service account used by the GCE instance is granted the access.
+
+After executing this step, you will have the following structure:
 
 ```
 example-organization/