update

0-bootstrap/Dockerfile

@@ -15,7 +15,7 @@
 FROM gcr.io/cloud-builders/gcloud-slim
 
 # Use ARG so that values can be overriden by user/cloudbuild
-ARG TERRAFORM_VERSION=1.3.0
+ARG TERRAFORM_VERSION=1.5.7
 
 ENV ENV_TERRAFORM_VERSION=$TERRAFORM_VERSION
 

0-bootstrap/README-GitHub.md

@@ -15,7 +15,7 @@ To run the instructions described in this document, install the following:
 - [Google Cloud SDK](https://cloud.google.com/sdk/install) version 393.0.0 or later
     - [terraform-tools](https://cloud.google.com/docs/terraform/policy-validation/validate-policies#install) component
 - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) version 2.28.0 or later
-- [Terraform](https://www.terraform.io/downloads.html) version 1.3.0  or later
+- [Terraform](https://www.terraform.io/downloads.html) version 1.5.7  or later
 
 Also make sure that you have the following:
 

0-bootstrap/README-Jenkins.md

@@ -192,7 +192,7 @@ You arrived to these instructions because you are using the `jenkins_bootstrap`
 ### II. Create the SEED and CI/CD projects using Terraform
 
 - Required information:
-  - Terraform version 1.3.0 - See [Requirements](#requirements) section for more details.
+  - Terraform version 1.5.7 - See [Requirements](#requirements) section for more details.
   - The `terraform.tfvars` file with all the necessary values.
 
 1. Get the appropriate credentials: run the following command with an account that has the [necessary permissions](./modules/jenkins-agent/README.md#permissions).
@@ -205,7 +205,7 @@ You arrived to these instructions because you are using the `jenkins_bootstrap`
 
 1. Run terraform commands.
    - After the credentials are configured, we will create the `prj-b-seed` project (which contains the GCS state bucket and Terraform custom service account) and the `prj-b-cicd` project (which contains the Jenkins Agent, its custom service account and where we will add VPN configuration)
-   - **Use Terraform 1.3.0** to run the terraform script with the commands below
+   - **Use Terraform 1.5.7** to run the terraform script with the commands below
 
    ```bash
    terraform init

0-bootstrap/README-Terraform-Cloud.md

@@ -17,7 +17,7 @@ To run the instructions described in this document, install the following:
 - [Google Cloud SDK](https://cloud.google.com/sdk/install) version 393.0.0 or later
     - [terraform-tools](https://cloud.google.com/docs/terraform/policy-validation/validate-policies#install) component
 - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) version 2.28.0 or later
-- [Terraform](https://www.terraform.io/downloads.html) version 1.3.0  or later
+- [Terraform](https://www.terraform.io/downloads.html) version 1.5.7  or later
 - [jq](https://jqlang.github.io/jq/download/) version 1.6.0 or later
 
 Also make sure that you have the following:

0-bootstrap/README.md

@@ -60,10 +60,10 @@ To run the commands described in this document, install the following:
 
 - [Google Cloud SDK](https://cloud.google.com/sdk/install) version 393.0.0 or later
 - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) version 2.28.0 or later
-- [Terraform](https://www.terraform.io/downloads.html) version 1.3.0
+- [Terraform](https://www.terraform.io/downloads.html) version 1.5.7
 - [jq](https://jqlang.github.io/jq/download/) version 1.6.0 or later
 
-**Note:** Make sure that you use version 1.3.0 of Terraform throughout this series. Otherwise, you might experience Terraform state snapshot lock errors.
+**Note:** Make sure that you use version 1.5.7 of Terraform throughout this series. Otherwise, you might experience Terraform state snapshot lock errors.
 
 Also make sure that you've done the following:
 

0-bootstrap/cb.tf

@@ -16,7 +16,7 @@
 
 locals {
   // terraform version image configuration
-  terraform_version = "1.3.10"
+  terraform_version = "1.5.7"
   // The version of the terraform docker image to be used in the workspace builds
   docker_tag_version_terraform = "v1"
 

0-bootstrap/modules/jenkins-agent/README.md

@@ -77,7 +77,7 @@ module "jenkins_bootstrap" {
 | storage\_bucket\_prefix | Name prefix to use for storage buckets. | `string` | `"bkt"` | no |
 | terraform\_sa\_names | Fully-qualified name of the Terraform Service Accounts. It must be supplied by the Seed Project | `map(string)` | n/a | yes |
 | terraform\_state\_bucket | Default state bucket, used in Cloud Build substitutions. It must be supplied by the Seed Project | `string` | n/a | yes |
-| terraform\_version | Default terraform version. | `string` | `"1.3.0"` | no |
+| terraform\_version | Default terraform version. | `string` | `"1.5.7"` | no |
 | terraform\_version\_sha256sum | sha256sum for default terraform version. | `string` | `"380ca822883176af928c80e5771d1c0ac9d69b13c6d746e6202482aedde7d457"` | no |
 | tunnel0\_bgp\_peer\_address | BGP peer address for tunnel 0 | `string` | n/a | yes |
 | tunnel0\_bgp\_session\_range | BGP session range for tunnel 0 | `string` | n/a | yes |
@@ -103,8 +103,8 @@ module "jenkins_bootstrap" {
 ### Software
 
 - [gcloud sdk](https://cloud.google.com/sdk/install) >= 393.0.0
-- [Terraform](https://www.terraform.io/downloads.html) = 1.3.0
-  - The scripts in this codebase use Terraform v1.3.0. You should use the same version in the manual steps to avoid [Terraform State Snapshot Lock](https://github.com/hashicorp/terraform/issues/23290) errors caused by differences in terraform versions.
+- [Terraform](https://www.terraform.io/downloads.html) = 1.5.7
+  - The scripts in this codebase use Terraform v1.5.7. You should use the same version in the manual steps to avoid [Terraform State Snapshot Lock](https://github.com/hashicorp/terraform/issues/23290) errors caused by differences in terraform versions.
 
 ### Infrastructure
 

0-bootstrap/modules/jenkins-agent/variables.tf

@@ -215,7 +215,7 @@ variable "folder_id" {
 variable "terraform_version" {
   description = "Default terraform version."
   type        = string
-  default     = "1.3.0"
+  default     = "1.5.7"
 }
 
 variable "terraform_version_sha256sum" {

2-environments/README.md

@@ -241,17 +241,17 @@ You will be doing this procedure for each environment (`development`, `non-produ
     export GCP_ENVIRONMENTS_PATH=INSERT_YOUR_PATH_HERE
     ```
 
-    Make sure your git is checked out to the `non-production` branch by running `git checkout nonproduction` on `GCP_ENVIRONMENTS_PATH`.
+    Make sure your git is checked out to the `non-production` branch by running `git checkout non-production` on `GCP_ENVIRONMENTS_PATH`.
 
     ```bash
-    (cd $GCP_ENVIRONMENTS_PATH && git checkout nonproduction)
+    (cd $GCP_ENVIRONMENTS_PATH && git checkout non-production)
     ```
 
 2. Retrieve the bucket name and project id from terraform outputs.
 
     ```bash
-    export ENV_LOG_BUCKET_NAME=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/nonproduction" output -raw env_log_bucket_name)
-    export ENV_LOG_PROJECT_ID=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/nonproduction" output -raw env_log_project_id)
+    export ENV_LOG_BUCKET_NAME=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/non-production" output -raw env_log_bucket_name)
+    export ENV_LOG_PROJECT_ID=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/non-production" output -raw env_log_project_id)
     ```
 
 3. Validate the variable values.
@@ -355,7 +355,7 @@ Proceed with these steps only if `Option 1` is not chosen.
 
 After making these modifications, you can follow the README.md procedure for `2-environment` step on foundation, make sure you **change the organization policy after running the steps on foundation**.
 
-1. You can now move to the instructions in the network step. To use the [Dual Shared VPC](https://cloud.google.com/architecture/security-foundations/networking#vpcsharedvpc-id7-1-shared-vpc-) network mode go to [3-networks-dual-svpc](../3-networks-dual-svpc/README.md), or go to [3-networks-hub-and-spoke](../3-networks-hub-and-spoke/README.md) to use the [Hub and Spoke](https://cloud.google.com/architecture/security-foundations/networking#hub-and-spoke) network mode.
+1. You can now move to the instructions in the network step. To use the [Dual Shared VPC](https://cloud.google.com/architecture/security-foundations/networking#vpcsharedvpc-id7-1-shared-vpc-) network mode go to [3-networks-dual-svpc](../3-networks-dual-svpc/README.md).
 
 ### Deploying with Jenkins
 

3-networks-dual-svpc/README.md

@@ -73,9 +73,9 @@ The purpose of this step is to:
    echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"
    ```
 
-1. For the manual step described in this document, you need [Terraform](https://www.terraform.io/downloads.html) version 1.3.0 or later to be installed.
+1. For the manual step described in this document, you need [Terraform](https://www.terraform.io/downloads.html) version 1.5.7 or later to be installed.
 
-**Note:** Make sure that you use version 1.3.0 or later of Terraform throughout this series. Otherwise, you might experience Terraform state snapshot lock errors.
+**Note:** Make sure that you use version 1.5.7 or later of Terraform throughout this series. Otherwise, you might experience Terraform state snapshot lock errors.
 
 ### Troubleshooting
 
@@ -188,6 +188,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    echo "remote_state_bucket = ${backend_bucket}"
 
    sed -i "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./common.auto.tfvars
+   for i in `find -name 'backend.tf'`; do sed -i "s/UPDATE_ME/${backend_bucket}/" $i; done
    ```
    **Note:** Make sure that you update the `perimeter_additional_members` variable with your e-mail in order to be able to view/access resources in the project protected by the VPC service controls.
 
@@ -199,7 +200,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    ```
 
 1. You must manually plan and apply the `shared` environment (only once) since the `development`, `non-production` and `production` environments depend on it.
+
 1. To use the `validate` option of the `tf-wrapper.sh` script, please follow the [instructions](https://cloud.google.com/docs/terraform/policy-validation/validate-policies#install) to install the terraform-tools component.
+
 1. Use `terraform output` to get the Cloud Build project ID and the networks step Terraform Service Account from 0-bootstrap output. An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set using the Terraform Service Account to enable impersonation.
 
    ```bash
@@ -210,12 +213,6 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}
    ```
 
-1. Log into gcloud using service account impersonation and then set your configuration:
-   ```bash
-   gcloud auth application-default login --impersonate-service-account=${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}
-   gcloud config set auth/impersonate_service_account ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}
-   ```
-
 1. Run `init` and `plan` and review output for environment shared.
 
    ```bash
@@ -235,11 +232,6 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    ./tf-wrapper.sh apply shared
    ```
 
-1. Unset your gcloud configuration to remove impersonation:
-   ```bash
-   gcloud config unset auth/impersonate_service_account
-   ```
-
 1. Push your plan branch to trigger a plan for all environments. Because the
    _plan_ branch is not a [named environment branch](../docs/FAQ.md#what-is-a-named-branch), pushing your _plan_
    branch triggers _terraform plan_ but not _terraform apply_. Review the plan output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID
@@ -274,6 +266,13 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    git push origin non-production
    ```
 
+1. Before executing the next step, unset the `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` environment variable.
+
+   ```bash
+   unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT
+   ```
+
+
 1. You can now move to the instructions in the [4-projects](../4-projects/README.md) step.
 
 ### Deploying with Jenkins

4-projects/README.md

@@ -72,9 +72,9 @@ Other Workspaces can also be created to isolate deployments if needed.
 1. 2-environments executed successfully.
 1. 3-networks executed successfully.
 
-1. For the manual step described in this document, you need [Terraform](https://www.terraform.io/downloads.html) version 1.3.0 or later to be installed.
+1. For the manual step described in this document, you need [Terraform](https://www.terraform.io/downloads.html) version 1.5.7 or later to be installed.
 
-   **Note:** Make sure that you use version 1.3.0 or later of Terraform throughout this series. Otherwise, you might experience Terraform state snapshot lock errors.
+   **Note:** Make sure that you use version 1.5.7 or later of Terraform throughout this series. Otherwise, you might experience Terraform state snapshot lock errors.
 
    **Note 2:** As mentioned in 0-bootstrap [README note 2](../0-bootstrap/README.md#deploying-with-cloud-build) at the end of Cloud Build deploy section, make sure that you have requested at least 50 additional projects for the **projects step service account**, otherwise you may face a project quota exceeded error message during the following steps and you will need to apply the fix from [this entry](../docs/TROUBLESHOOTING.md#attempt-to-run-4-projects-step-without-enough-project-quota) of the Troubleshooting guide in order to continue.
 
@@ -132,7 +132,12 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw gcs_bucket_tfstate)
    echo "remote_state_bucket = ${remote_state_bucket}"
 
+   export projects_gcs_bucket_tfstate=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_gcs_bucket_tfstate)
+   echo "projects_gcs_bucket_tfstate = ${projects_gcs_bucket_tfstate}"
+
+
    sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars
+   for i in `find -name 'backend.tf'`; do sed -i "s/UPDATE_PROJECTS_BACKEND/${projects_gcs_bucket_tfstate}/" $i; done
    ```
 
 1. Commit changes.
@@ -156,12 +161,6 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}
    ```
 
-1. Log into gcloud using service account impersonation and then set your configuration:
-   ```bash
-   gcloud auth application-default login --impersonate-service-account=${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}
-   gcloud config set auth/impersonate_service_account ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}
-   ```
-
 1. Run `init` and `plan` and review output for environment shared.
 
    ```bash
@@ -181,11 +180,6 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get
    ./tf-wrapper.sh apply shared
    ```
 
-1. Unset your gcloud configuration to remove impersonation:
-   ```bash
-   gcloud config unset auth/impersonate_service_account
-   ```
-
 1. Push your plan branch to trigger a plan for all environments. Because the
    _plan_ branch is not a [named environment branch](../docs/FAQ.md#what-is-a-named-branch)), pushing your _plan_
    branch triggers _terraform plan_ but not _terraform apply_. Review the plan output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

5-app-infra/README.md

@@ -360,6 +360,14 @@ The pipeline also listens for changes made to `plan`, `development`, `non-produc
    for i in `find -name 'backend.tf'`; do sed -i "s/UPDATE_APP_INFRA_BUCKET/${backend_bucket}/" $i; done
    ```
 
+1. Update the `log_bucket` variable with the value of the `logs_export_storage_bucket_name`.
+
+   ```bash
+   export log_bucket=$(terraform -chdir="../gcp-org/envs/shared" output -raw logs_export_storage_bucket_name)
+   echo "log_bucket = ${log_bucket}"
+   sed -i "s/REPLACE_LOG_BUCKET/${log_bucket}/" ./common.auto.tfvars
+   ```
+
 1. Commit changes.
 
    ```bash

5-app-infra/modules/publish_artifacts/main.tf

@@ -20,6 +20,13 @@ resource "google_project_service_identity" "artifact_registry_agent" {
   service = "artifactregistry.googleapis.com"
 }
 
+resource "google_project_service_identity" "storage_agent" {
+  provider = google-beta
+
+  project = var.project_id
+  service = "storage.googleapis.com"
+}
+
 resource "google_kms_crypto_key_iam_member" "artifact-kms-key-binding" {
   crypto_key_id = var.kms_crypto_key
   role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
@@ -109,6 +116,8 @@ resource "google_kms_crypto_key_iam_member" "storage_agent" {
   crypto_key_id = var.kms_crypto_key
   role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
   member        = "serviceAccount:service-${data.google_project.project.number}@gs-project-accounts.iam.gserviceaccount.com"
+
+  depends_on = [ google_project_service_identity.storage_agent ]
   #member = "serviceAccount:${google_project_service_identity.storage.email}"
 }
 

5-app-infra/projects/service-catalog/ml_business_unit/shared/variables.tf

@@ -23,3 +23,8 @@ variable "remote_state_bucket" {
   description = "Backend bucket to load remote state information from previous steps."
   type        = string
 }
+
+variable "log_bucket" {
+  description = "Log bucket to be used by Service Catalog Bucket"
+  type        = string
+}

Dockerfile-dev

@@ -1,6 +1,6 @@
 FROM alpine:3.18.4
 # Use ARG so that values can be overriden by user/cloudbuild
-ARG TERRAFORM_VERSION=1.3.0
+ARG TERRAFORM_VERSION=1.5.7
 ARG GCLOUD_VERSION=455.0.0
 
 ENV ENV_TERRAFORM_VERSION=$TERRAFORM_VERSION

docs/TROUBLESHOOTING.md

@@ -87,37 +87,37 @@ This could be due to init.defaultBranch being set to something other than
 When running the build for the branch `production` in step 3-networks in your **Foundation CI/CD Pipeline** the build fails with:
 
 ```
-state snapshot was created by Terraform v1.x.x, which is newer than current v1.3.0; upgrade to Terraform v1.x.x or greater to work with this state
+state snapshot was created by Terraform v1.x.x, which is newer than current v1.5.7; upgrade to Terraform v1.x.x or greater to work with this state
 ```
 
 **Cause:**
 
-The manual deploy step for the shared environment in [3-networks](../3-networks#deploying-with-cloud-build) was executed with a Terraform version newer than version v1.3.0 used in the **Foundation CI/CD Pipeline**.
+The manual deploy step for the shared environment in [3-networks](../3-networks#deploying-with-cloud-build) was executed with a Terraform version newer than version v1.5.7 used in the **Foundation CI/CD Pipeline**.
 
 **Solution:**
 
 You have two options:
 
 #### Downgrade your local Terraform version
 
-You will need to re-run the deploy of the 3-networks shared environment with Terraform v1.3.0.
+You will need to re-run the deploy of the 3-networks shared environment with Terraform v1.5.7.
 
 Steps:
 
 - Go to folder `gcp-networks/envs/shared/`.
 - Update `backend.tf` with your bucket name from the 0-bootstrap step.
 - Run `terraform destroy` in the folder using the Terraform v1.x.x version.
 - Delete the Terraform state file in `gs://YOUR-TF-STATE-BUCKET/terraform/networks/envs/shared/default.tfstate`. This bucket is in your **Seed Project**.
-- Install Terraform v1.3.0.
-- Re-run the manual deploy of 3-networks shared environment using Terraform v1.3.0.
+- Install Terraform v1.5.7.
+- Re-run the manual deploy of 3-networks shared environment using Terraform v1.5.7.
 
 #### Upgrade your 0-bootstrap runner image Terraform version
 
 Replace `1.x.x` with the actual version of your local Terraform version in the following instructions:
 
 - Go to folder `0-bootstrap`.
 - Edit the local `terraform_version` in the Terraform [cb.tf](../0-bootstrap/cb.tf) file:
-  - Upgrade local `terraform_version` from `"1.3.0"` to `"1.x.x"`
+  - Upgrade local `terraform_version` from `"1.5.7"` to `"1.x.x"`
 - Run `terraform init`.
 - Run `terraform plan` and review the output.
 - Run `terraform apply`.