diff --git a/docs/upgrading_to_v21.0.md b/docs/upgrading_to_v21.0.md index 199bc067f6..b3e0b16c12 100644 --- a/docs/upgrading_to_v21.0.md +++ b/docs/upgrading_to_v21.0.md @@ -1,5 +1,4 @@ # Upgrading to v21.0 - The v21.0 release of *kubernetes-engine* is a backwards incompatible release. @@ -14,3 +13,135 @@ The [Terraform Kubernetes Engine Module](https://github.com/terraform-google-mod ### Kubernetes Provider upgrade The Terraform Kubernetes Engine module now requires version 2.10 or higher of the Kubernetes Provider. + +### Hub module rewrite +The old [Hub submodule](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/v20.0.0/modules/hub) +has been renamed to `hub-legacy` and deprecated. It is replaced with a new [fleet membership](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/fleet-membership) +module to handle registering GKE clusters to [fleets](https://cloud.google.com/anthos/multicluster-management/fleets) using the native API. + +The new module relies exclusively on native Terraform resources and should therefore be more robust. + +### Migrating +For GKE clusters, you should update your configuration as follows: + +```diff + module "register" { +- source = "terraform-google-modules/kubernetes-engine/google//modules/hub" +- version = "~> 20.0" ++ source = "terraform-google-modules/kubernetes-engine/google//modules/fleet-membership" ++ version = "~> 21.0" + + project_id = "my-project-id" + cluster_name = "my-cluster-name" +- gke_hub_membership_name = "gke-membership" ++ membership_name = "gke-hub-membership" + location = module.gke.location +- cluster_endpoint = module.gke.endpoint +- gke_hub_sa_name = "sa-for-kind-cluster-membership" +- use_kubeconfig = true +- labels = "testlabel=usekubecontext" +- module_depends_on = [module.gke] +} +``` + +You also need to follow these migration steps: + +1. Remove the old module from your state: + + ``` + terraform state rm module.register + ``` + +2. Remove the cluster from the fleet: + + ``` + gcloud container fleet memberships delete gke-hub-membership-name + ``` + +3. Apply the new configuration to re-register the cluster: + + ``` + terraform apply + ``` + +#### Legacy module +**The native API only supports registering GKE clusters**. Therefore, the old hub module is preserved as `hub-legacy`. + +You can continue using it by updating your configuration to point to the new location. + +```diff + module "register" { +- source = "terraform-google-modules/kubernetes-engine/google//modules/hub" +- version = "~> 20.0" ++ source = "terraform-google-modules/kubernetes-engine/google//modules/hub-legacy" ++ version = "~> 21.0" + + project_id = "my-project-id" + cluster_name = "my-cluster-name" + location = module.gke.location + cluster_endpoint = module.gke.endpoint + } +``` + +### Anthos Config Management (ACM) and Config Sync Module Rewrite +Together with the rewrite of the Hub module, the ACM module also has been rewritten to use native resources. + +You will need to follow these migration steps: + +1. Update your configuration to use the new module: + + ```diff + module "acm" { + source = "terraform-google-modules/kubernetes-engine/google//modules/acm" + - version = "~> 20.0" + + version = "~> 21.0" + + project_id = "my-project-id" + cluster_name = "simple-zonal-cluster" + location = "us-central1-a" + - cluster_endpoint = module.auth.host + + sync_repo = "git@github.com:GoogleCloudPlatform/csp-config-management.git" + sync_branch = "1.0.0" + policy_dir = "foo-corp" + + secret_type = "ssh" + } + ``` + +1. Make sure you have the `kubernetes` provider configured: + + ```hcl + provider "kubernetes" { + cluster_ca_certificate = module.auth.cluster_ca_certificate + host = module.auth.host + token = module.auth.token + } + ``` + +1. Remove the old module from your state: + + ``` + terraform state rm module.acm + ``` + +2. Import the old `git-creds` secret into Terraform: + + ``` + terraform import 'module.acm.module.acm_operator.kubernetes_secret_v1.creds' 'config-management-system/git-creds' + ``` + +3. Apply the new configuration to re-register ACM and confirm everything is working: + + ``` + terraform apply + ``` + +#### Feature Activation + +Only the first cluster in a fleet should activate the ACM fleet feature. +Other clusters should disable feature activation by setting `enable_fleet_feature = false`. + +#### Config Sync Module Removed +The dedicated Config Sync submodule has been removed. +To use Config Sync, just invoke the ACM module with `enable_policy_controller = false`. diff --git a/examples/simple_zonal_with_acm/README.md b/examples/simple_zonal_with_acm/README.md index 8befc2372e..2607c45b8b 100644 --- a/examples/simple_zonal_with_acm/README.md +++ b/examples/simple_zonal_with_acm/README.md @@ -4,23 +4,38 @@ This example illustrates how to create a simple cluster and install [Anthos Conf It incorporates the standard cluster module and the [ACM install module](../../modules/acm). +## Verifying Success + +After applying the Terraform configuration, you can run the following commands to verify that your cluster has synced correctly: + +1. Check ACM install status: + + ``` + gcloud config set project $(terraform output --raw project_id) + gcloud alpha container hub config-management status + ``` + +2. Connect to the cluster: + + ``` + gcloud container clusters get-credentials $(terraform output --raw cluster_name) --zone=$(terraform output --raw location) + ``` + +3. Confirm the `shipping-dev` namespace was created: + + ``` + kubectl describe ns shipping-dev + ``` + ## Inputs | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| -| acm\_policy\_dir | Subfolder containing configs in ACM Git repo | `string` | `"foo-corp"` | no | -| acm\_sync\_branch | Anthos config management Git branch | `string` | `"1.0.0"` | no | -| acm\_sync\_repo | Anthos config management Git repo | `string` | `"git@github.com:GoogleCloudPlatform/csp-config-management.git"` | no | | cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no | -| ip\_range\_pods | The secondary ip range to use for pods | `any` | n/a | yes | -| ip\_range\_services | The secondary ip range to use for services | `any` | n/a | yes | -| network | The VPC network to host the cluster in | `any` | n/a | yes | -| operator\_path | Path to the operator yaml config. If unset, will download from GCS releases. | `string` | `null` | no | | project\_id | The project ID to host the cluster in | `any` | n/a | yes | -| region | The region to host the cluster in | `any` | n/a | yes | -| subnetwork | The subnetwork to host the cluster in | `any` | n/a | yes | -| zones | The zone to host the cluster in (required if is a zonal cluster) | `list(string)` | n/a | yes | +| region | The region to host the cluster in | `string` | `"us-central1"` | no | +| zone | The zone to host the cluster in | `string` | `"us-central1-a"` | no | ## Outputs @@ -36,7 +51,7 @@ It incorporates the standard cluster module and the [ACM install module](../../m | location | n/a | | master\_kubernetes\_version | The master Kubernetes version | | network | n/a | -| project\_id | n/a | +| project\_id | Standard test outputs | | region | n/a | | service\_account | The default service account used for running nodes. | | subnetwork | n/a | diff --git a/examples/simple_zonal_with_acm/acm.tf b/examples/simple_zonal_with_acm/acm.tf index 0af8f08b5d..b1ea3225bc 100644 --- a/examples/simple_zonal_with_acm/acm.tf +++ b/examples/simple_zonal_with_acm/acm.tf @@ -15,13 +15,14 @@ */ module "acm" { - source = "../../modules/acm" - project_id = var.project_id - location = module.gke.location - cluster_name = module.gke.name - sync_repo = var.acm_sync_repo - sync_branch = var.acm_sync_branch - policy_dir = var.acm_policy_dir - cluster_endpoint = module.gke.endpoint - operator_path = var.operator_path + source = "../../modules/acm" + project_id = var.project_id + location = module.gke.location + cluster_name = module.gke.name + + sync_repo = "git@github.com:GoogleCloudPlatform/csp-config-management.git" + sync_branch = "1.0.0" + policy_dir = "foo-corp" + + secret_type = "ssh" } diff --git a/examples/simple_zonal_with_acm/main.tf b/examples/simple_zonal_with_acm/main.tf index 540c3a6bb3..31d374a601 100644 --- a/examples/simple_zonal_with_acm/main.tf +++ b/examples/simple_zonal_with_acm/main.tf @@ -18,6 +18,10 @@ locals { cluster_type = "simple-zonal" } +provider "google" { + region = var.region +} + data "google_client_config" "default" {} provider "kubernetes" { @@ -27,17 +31,20 @@ provider "kubernetes" { } module "gke" { - source = "../../" - project_id = var.project_id - name = "${local.cluster_type}-cluster${var.cluster_name_suffix}" - regional = false - region = var.region - zones = var.zones - network = var.network - subnetwork = var.subnetwork - ip_range_pods = var.ip_range_pods - ip_range_services = var.ip_range_services - service_account = "create" + source = "../../" + project_id = var.project_id + regional = false + region = var.region + zones = [var.zone] + + name = "${local.cluster_type}-cluster${var.cluster_name_suffix}" + + network = google_compute_network.main.name + subnetwork = google_compute_subnetwork.main.name + ip_range_pods = google_compute_subnetwork.main.secondary_ip_range[0].range_name + ip_range_services = google_compute_subnetwork.main.secondary_ip_range[1].range_name + + service_account = "create" node_pools = [ { name = "acm-node-pool" diff --git a/test/fixtures/simple_zonal/network.tf b/examples/simple_zonal_with_acm/network.tf similarity index 92% rename from test/fixtures/simple_zonal/network.tf rename to examples/simple_zonal_with_acm/network.tf index e0bf46c2f2..0f2a3d3e84 100644 --- a/test/fixtures/simple_zonal/network.tf +++ b/examples/simple_zonal_with_acm/network.tf @@ -1,5 +1,5 @@ /** - * Copyright 2018 Google LLC + * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -20,16 +20,14 @@ resource "random_string" "suffix" { upper = false } -provider "google" { - project = var.project_ids[1] -} - resource "google_compute_network" "main" { + project = var.project_id name = "cft-gke-test-${random_string.suffix.result}" auto_create_subnetworks = false } resource "google_compute_subnetwork" "main" { + project = var.project_id name = "cft-gke-test-${random_string.suffix.result}" ip_cidr_range = "10.0.0.0/17" region = var.region diff --git a/examples/simple_zonal_with_acm/outputs.tf b/examples/simple_zonal_with_acm/outputs.tf index b785e5a791..014eb7b4d9 100644 --- a/examples/simple_zonal_with_acm/outputs.tf +++ b/examples/simple_zonal_with_acm/outputs.tf @@ -25,7 +25,8 @@ output "client_token" { } output "ca_certificate" { - value = module.gke.ca_certificate + value = module.gke.ca_certificate + sensitive = true } output "service_account" { @@ -38,3 +39,48 @@ output "acm_git_creds_public" { value = module.acm.git_creds_public } +# Standard test outputs +output "project_id" { + value = var.project_id +} + +output "region" { + value = module.gke.region +} + +output "cluster_name" { + description = "Cluster name" + value = module.gke.name +} + +output "network" { + value = google_compute_network.main.name +} + +output "subnetwork" { + value = google_compute_subnetwork.main.name +} + +output "location" { + value = module.gke.location +} + +output "ip_range_pods" { + description = "The secondary IP range used for pods" + value = google_compute_subnetwork.main.secondary_ip_range[0].range_name +} + +output "ip_range_services" { + description = "The secondary IP range used for services" + value = google_compute_subnetwork.main.secondary_ip_range[1].range_name +} + +output "zones" { + description = "List of zones in which the cluster resides" + value = module.gke.zones +} + +output "master_kubernetes_version" { + description = "The master Kubernetes version" + value = module.gke.master_version +} diff --git a/examples/simple_zonal_with_acm/test_outputs.tf b/examples/simple_zonal_with_acm/test_outputs.tf deleted file mode 120000 index 17b34213ba..0000000000 --- a/examples/simple_zonal_with_acm/test_outputs.tf +++ /dev/null @@ -1 +0,0 @@ -../../test/fixtures/all_examples/test_outputs.tf \ No newline at end of file diff --git a/examples/simple_zonal_with_acm/variables.tf b/examples/simple_zonal_with_acm/variables.tf index f4661d5a97..c02931ccd9 100644 --- a/examples/simple_zonal_with_acm/variables.tf +++ b/examples/simple_zonal_with_acm/variables.tf @@ -25,49 +25,11 @@ variable "cluster_name_suffix" { variable "region" { description = "The region to host the cluster in" + default = "us-central1" } -variable "zones" { - type = list(string) - description = "The zone to host the cluster in (required if is a zonal cluster)" -} - -variable "network" { - description = "The VPC network to host the cluster in" -} - -variable "subnetwork" { - description = "The subnetwork to host the cluster in" -} - -variable "ip_range_pods" { - description = "The secondary ip range to use for pods" -} - -variable "ip_range_services" { - description = "The secondary ip range to use for services" -} - -variable "acm_sync_repo" { - description = "Anthos config management Git repo" - type = string - default = "git@github.com:GoogleCloudPlatform/csp-config-management.git" -} - -variable "acm_sync_branch" { - description = "Anthos config management Git branch" - type = string - default = "1.0.0" -} - -variable "acm_policy_dir" { - description = "Subfolder containing configs in ACM Git repo" - type = string - default = "foo-corp" -} - -variable "operator_path" { - description = "Path to the operator yaml config. If unset, will download from GCS releases." +variable "zone" { type = string - default = null + description = "The zone to host the cluster in" + default = "us-central1-a" } diff --git a/examples/simple_zonal_with_hub/README.md b/examples/simple_zonal_with_hub/README.md index 9f7fe079ab..ce0a513ac0 100644 --- a/examples/simple_zonal_with_hub/README.md +++ b/examples/simple_zonal_with_hub/README.md @@ -1,8 +1,10 @@ # Simple Zonal Cluster -This example illustrates how to create a simple cluster and register it with [Anthos](https://cloud.google.com/anthos/multicluster-management/environs) +This example illustrates how to create a simple cluster and register it with [Anthos](https://cloud.google.com/anthos/multicluster-management/connect/registering-a-cluster#gcloud). -It incorporates the standard cluster module and the [Hub registration module](../../modules/hub). +After registering the cluster, it uses that registration to install [Config Sync](https://cloud.google.com/anthos-config-management/docs/config-sync-overview). + +It incorporates the standard cluster module, the [registration module](../../modules/fleet-membership), and the [ACM module](../../modules/acm). ## Inputs @@ -10,13 +12,9 @@ It incorporates the standard cluster module and the [Hub registration module](.. | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| | cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no | -| ip\_range\_pods | The secondary ip range to use for pods | `string` | `""` | no | -| ip\_range\_services | The secondary ip range to use for services | `string` | `""` | no | -| network | The VPC network to host the cluster in | `string` | `"default"` | no | | project\_id | The project ID to host the cluster in | `any` | n/a | yes | -| region | The region to host the cluster in | `any` | n/a | yes | -| subnetwork | The subnetwork to host the cluster in | `string` | `"default"` | no | -| zones | The zone to host the cluster in (required if is a zonal cluster) | `list(string)` | n/a | yes | +| region | The region to host the cluster in | `string` | `"us-central1"` | no | +| zone | The zone to host the cluster in | `string` | `"us-central1-a"` | no | ## Outputs @@ -31,7 +29,7 @@ It incorporates the standard cluster module and the [Hub registration module](.. | location | n/a | | master\_kubernetes\_version | The master Kubernetes version | | network | n/a | -| project\_id | n/a | +| project\_id | Standard test outputs | | region | n/a | | service\_account | The default service account used for running nodes. | | subnetwork | n/a | diff --git a/examples/simple_zonal_with_hub/hub.tf b/examples/simple_zonal_with_hub/hub.tf index a0ce6cf505..08b912c4ff 100644 --- a/examples/simple_zonal_with_hub/hub.tf +++ b/examples/simple_zonal_with_hub/hub.tf @@ -15,9 +15,10 @@ */ module "hub" { - source = "../../modules/hub" - project_id = var.project_id - location = module.gke.location - cluster_name = module.gke.name - cluster_endpoint = module.gke.endpoint + source = "../../modules/fleet-membership" + project_id = var.project_id + location = module.gke.location + cluster_name = module.gke.name + + depends_on = [module.gke] } diff --git a/examples/simple_zonal_with_hub/main.tf b/examples/simple_zonal_with_hub/main.tf index 8c1d039143..09e0bd59c5 100644 --- a/examples/simple_zonal_with_hub/main.tf +++ b/examples/simple_zonal_with_hub/main.tf @@ -14,8 +14,8 @@ * limitations under the License. */ -locals { - cluster_type = "simple-zonal" +provider "google" { + region = var.region } data "google_client_config" "default" {} @@ -27,15 +27,27 @@ provider "kubernetes" { } module "gke" { - source = "../../" - project_id = var.project_id - name = "${local.cluster_type}-cluster${var.cluster_name_suffix}" - regional = false - region = var.region - zones = var.zones - network = var.network - subnetwork = var.subnetwork - ip_range_pods = var.ip_range_pods - ip_range_services = var.ip_range_services - service_account = "create" + source = "../../" + project_id = var.project_id + regional = false + region = var.region + zones = [var.zone] + + name = "config-sync-cluster${var.cluster_name_suffix}" + + network = google_compute_network.main.name + subnetwork = google_compute_subnetwork.main.name + ip_range_pods = google_compute_subnetwork.main.secondary_ip_range[0].range_name + ip_range_services = google_compute_subnetwork.main.secondary_ip_range[1].range_name + + service_account = "create" + node_pools = [ + { + name = "node-pool" + autoscaling = false + auto_upgrade = true + node_count = 4 + machine_type = "e2-standard-4" + }, + ] } diff --git a/examples/simple_zonal_with_hub/network.tf b/examples/simple_zonal_with_hub/network.tf new file mode 100644 index 0000000000..0f2a3d3e84 --- /dev/null +++ b/examples/simple_zonal_with_hub/network.tf @@ -0,0 +1,45 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +resource "random_string" "suffix" { + length = 4 + special = false + upper = false +} + +resource "google_compute_network" "main" { + project = var.project_id + name = "cft-gke-test-${random_string.suffix.result}" + auto_create_subnetworks = false +} + +resource "google_compute_subnetwork" "main" { + project = var.project_id + name = "cft-gke-test-${random_string.suffix.result}" + ip_cidr_range = "10.0.0.0/17" + region = var.region + network = google_compute_network.main.self_link + + secondary_ip_range { + range_name = "cft-gke-test-pods-${random_string.suffix.result}" + ip_cidr_range = "192.168.0.0/18" + } + + secondary_ip_range { + range_name = "cft-gke-test-services-${random_string.suffix.result}" + ip_cidr_range = "192.168.64.0/18" + } +} diff --git a/examples/simple_zonal_with_hub/outputs.tf b/examples/simple_zonal_with_hub/outputs.tf index 0d770aa809..d953d1b3db 100644 --- a/examples/simple_zonal_with_hub/outputs.tf +++ b/examples/simple_zonal_with_hub/outputs.tf @@ -25,10 +25,57 @@ output "client_token" { } output "ca_certificate" { - value = module.gke.ca_certificate + value = module.gke.ca_certificate + sensitive = true } output "service_account" { description = "The default service account used for running nodes." value = module.gke.service_account } + +# Standard test outputs +output "project_id" { + value = var.project_id +} + +output "region" { + value = module.gke.region +} + +output "cluster_name" { + description = "Cluster name" + value = module.gke.name +} + +output "network" { + value = google_compute_network.main.name +} + +output "subnetwork" { + value = google_compute_subnetwork.main.name +} + +output "location" { + value = module.gke.location +} + +output "ip_range_pods" { + description = "The secondary IP range used for pods" + value = google_compute_subnetwork.main.secondary_ip_range[0].range_name +} + +output "ip_range_services" { + description = "The secondary IP range used for services" + value = google_compute_subnetwork.main.secondary_ip_range[1].range_name +} + +output "zones" { + description = "List of zones in which the cluster resides" + value = module.gke.zones +} + +output "master_kubernetes_version" { + description = "The master Kubernetes version" + value = module.gke.master_version +} diff --git a/examples/simple_zonal_with_hub/test_outputs.tf b/examples/simple_zonal_with_hub/test_outputs.tf deleted file mode 100755 index e64c40e477..0000000000 --- a/examples/simple_zonal_with_hub/test_outputs.tf +++ /dev/null @@ -1,63 +0,0 @@ -/** - * Copyright 2018 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -// These outputs are used to test the module with kitchen-terraform -// They do not need to be included in real-world uses of this module - -output "project_id" { - value = var.project_id -} - -output "region" { - value = module.gke.region -} - -output "cluster_name" { - description = "Cluster name" - value = module.gke.name -} - -output "network" { - value = var.network -} - -output "subnetwork" { - value = var.subnetwork -} - -output "location" { - value = module.gke.location -} - -output "ip_range_pods" { - description = "The secondary IP range used for pods" - value = var.ip_range_pods -} - -output "ip_range_services" { - description = "The secondary IP range used for services" - value = var.ip_range_services -} - -output "zones" { - description = "List of zones in which the cluster resides" - value = module.gke.zones -} - -output "master_kubernetes_version" { - description = "The master Kubernetes version" - value = module.gke.master_version -} diff --git a/examples/simple_zonal_with_hub/variables.tf b/examples/simple_zonal_with_hub/variables.tf index 1416853db2..c02931ccd9 100644 --- a/examples/simple_zonal_with_hub/variables.tf +++ b/examples/simple_zonal_with_hub/variables.tf @@ -25,29 +25,11 @@ variable "cluster_name_suffix" { variable "region" { description = "The region to host the cluster in" + default = "us-central1" } -variable "zones" { - type = list(string) - description = "The zone to host the cluster in (required if is a zonal cluster)" -} - -variable "network" { - description = "The VPC network to host the cluster in" - default = "default" -} - -variable "subnetwork" { - description = "The subnetwork to host the cluster in" - default = "default" -} - -variable "ip_range_pods" { - description = "The secondary ip range to use for pods" - default = "" -} - -variable "ip_range_services" { - description = "The secondary ip range to use for services" - default = "" +variable "zone" { + type = string + description = "The zone to host the cluster in" + default = "us-central1-a" } diff --git a/examples/simple_zonal_with_hub_kubeconfig/README.md b/examples/simple_zonal_with_hub_kubeconfig/README.md index 409b473322..6228f9c827 100644 --- a/examples/simple_zonal_with_hub_kubeconfig/README.md +++ b/examples/simple_zonal_with_hub_kubeconfig/README.md @@ -2,7 +2,7 @@ This example illustrates how to register a non-GKE Kubernetes Cluster with [Anthos](https://cloud.google.com/anthos/multicluster-management/environs) a.k.a Attached cluster. -It creates a [kind](https://kind.sigs.k8s.io/) cluster, sets current kubecontext to the cluster and registers the cluster using the [Hub registration module](../../modules/hub). +It creates a [kind](https://kind.sigs.k8s.io/) cluster, sets current kubecontext to the cluster and registers the cluster using the [Hub registration module](../../modules/hub-legacy). ## Inputs diff --git a/examples/simple_zonal_with_hub_kubeconfig/hub.tf b/examples/simple_zonal_with_hub_kubeconfig/hub.tf index 035c25d4f0..2fe50463af 100644 --- a/examples/simple_zonal_with_hub_kubeconfig/hub.tf +++ b/examples/simple_zonal_with_hub_kubeconfig/hub.tf @@ -15,7 +15,7 @@ */ module "hub" { - source = "../../modules/hub" + source = "../../modules/hub-legacy" project_id = var.project_id location = "remote" cluster_name = kind_cluster.test-cluster.name diff --git a/modules/acm/README.md b/modules/acm/README.md index b996d286a8..2b127d8cb0 100644 --- a/modules/acm/README.md +++ b/modules/acm/README.md @@ -3,10 +3,15 @@ This module installs [Anthos Config Management](https://cloud.google.com/anthos-config-management/docs/) (ACM) in a Kubernetes cluster. Specifically, this module automates the following steps for [installing ACM](https://cloud.google.com/anthos-config-management/docs/how-to/installing): -1. Enabling the ACM feature on hub. -2. Installing the ACM Operator on your cluster. +1. Enabling the ACM feature on the fleet. +2. Registering the cluster to the fleet 3. Optionally, generating an SSH key for accessing Git and providing it to the Operator -4. Configuring the Operator to connect to your ACM repository +4. Configuring the ACM feature on your cluster + +## Fleet feature +Only the first cluster in a fleet should activate the ACM fleet feature. + +Other clusters should disable feature activation by setting `enable_fleet_feature = false`. ## Usage @@ -36,35 +41,29 @@ To deploy this config: * [GitHub](https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/) * [Gitlab](https://docs.gitlab.com/ee/ssh/) -## Whitelisting -Note that installing Anthos Config Management [requires](https://cloud.google.com/anthos-config-management/docs/how-to/installing#local_environment) an active Anthos license. -By default, this module will attempt to download the ACM operator from Google directly—meaning your Terraform service account needs to be whitelisted for ACM access. If this is an issue, you can predownload the operator yourself then set the `operator_path` variable to point to the file location. - ## Inputs | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| -| cluster\_endpoint | Kubernetes cluster endpoint. | `string` | n/a | yes | +| cluster\_membership\_id | The cluster membership ID. If unset, one will be autogenerated. | `string` | `""` | no | | cluster\_name | GCP cluster Name used to reach cluster and which becomes the cluster name in the Config Sync kubernetes custom resource. | `string` | n/a | yes | | create\_ssh\_key | Controls whether a key will be generated for Git authentication | `bool` | `true` | no | +| enable\_fleet\_feature | Whether to enable the ACM feature on the fleet. | `bool` | `true` | no | +| enable\_fleet\_registration | Whether to create a new membership. | `bool` | `true` | no | | enable\_log\_denies | Whether to enable logging of all denies and dryrun failures for ACM Policy Controller. | `bool` | `false` | no | -| enable\_multi\_repo | Whether to use ACM Config Sync [multi-repo mode](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/multi-repo). | `bool` | `false` | no | | enable\_policy\_controller | Whether to enable the ACM Policy Controller on the cluster | `bool` | `true` | no | | hierarchy\_controller | Configurations for Hierarchy Controller. See [Hierarchy Controller docs](https://cloud.google.com/anthos-config-management/docs/how-to/installing-hierarchy-controller) for more details | `map(any)` | `null` | no | | install\_template\_library | Whether to install the default Policy Controller template library | `bool` | `true` | no | | location | GCP location used to reach cluster. | `string` | n/a | yes | -| operator\_path | Path to the operator yaml config. If unset, will download from GCS releases. | `string` | `null` | no | | policy\_dir | Subfolder containing configs in ACM Git repo. If un-set, uses Config Management default. | `string` | `""` | no | | project\_id | GCP project\_id used to reach cluster. | `string` | n/a | yes | | secret\_type | git authentication secret type, is passed through to ConfigManagement spec.git.secretType. Overriden to value 'ssh' if `create_ssh_key` is true | `string` | `"ssh"` | no | -| service\_account\_key\_file | Path to service account key file to auth as for running `gcloud container clusters get-credentials`. | `string` | `""` | no | | source\_format | Configures a non-hierarchical repo if set to 'unstructured'. Uses [ACM defaults](https://cloud.google.com/anthos-config-management/docs/how-to/installing#configuring-config-management-operator) when unset. | `string` | `""` | no | | ssh\_auth\_key | Key for Git authentication. Overrides 'create\_ssh\_key' variable. Can be set using 'file(path/to/file)'-function. | `string` | `null` | no | | sync\_branch | ACM repo Git branch. If un-set, uses Config Management default. | `string` | `""` | no | | sync\_repo | ACM Git repo address | `string` | n/a | yes | | sync\_revision | ACM repo Git revision. If un-set, uses Config Management default. | `string` | `""` | no | -| use\_existing\_context | Use existing kubecontext to auth kube-api. | `bool` | `false` | no | ## Outputs diff --git a/modules/acm/creds.tf b/modules/acm/creds.tf new file mode 100644 index 0000000000..73968cc7a3 --- /dev/null +++ b/modules/acm/creds.tf @@ -0,0 +1,43 @@ +/** + * Copyright 2018 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +resource "tls_private_key" "k8sop_creds" { + count = var.create_ssh_key ? 1 : 0 + algorithm = "RSA" + rsa_bits = 4096 +} + +# Wait for the ACM operator to create the namespace +resource "time_sleep" "wait_acm" { + count = (var.create_ssh_key == true || var.ssh_auth_key != null) ? 1 : 0 + depends_on = [google_gke_hub_feature_membership.main] + + create_duration = "30s" +} + +resource "kubernetes_secret_v1" "creds" { + count = (var.create_ssh_key == true || var.ssh_auth_key != null) ? 1 : 0 + depends_on = [time_sleep.wait_acm] + + metadata { + name = "git-creds" + namespace = "config-management-system" + } + + data = { + "${local.k8sop_creds_secret_key}" = local.private_key + } +} diff --git a/modules/acm/feature.tf b/modules/acm/feature.tf new file mode 100644 index 0000000000..e80df8368d --- /dev/null +++ b/modules/acm/feature.tf @@ -0,0 +1,73 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +resource "google_gke_hub_feature" "acm" { + count = var.enable_fleet_feature ? 1 : 0 + provider = google-beta + + name = "configmanagement" + project = var.project_id + location = "global" +} + +resource "google_gke_hub_feature_membership" "main" { + provider = google-beta + depends_on = [ + google_gke_hub_feature.acm + ] + + location = "global" + feature = "configmanagement" + + membership = module.registration.cluster_membership_id + project = var.project_id + + configmanagement { + version = "1.11.0" + + config_sync { + source_format = var.source_format != "" ? var.source_format : null + + git { + sync_repo = var.sync_repo + policy_dir = var.policy_dir != "" ? var.policy_dir : null + sync_branch = var.sync_branch != "" ? var.sync_branch : null + sync_rev = var.sync_revision != "" ? var.sync_revision : null + secret_type = var.secret_type + } + } + + dynamic "policy_controller" { + for_each = var.enable_policy_controller ? [{ enabled = true }] : [] + + content { + enabled = true + template_library_installed = var.install_template_library + log_denies_enabled = var.enable_log_denies + } + } + + dynamic "hierarchy_controller" { + for_each = var.hierarchy_controller == null ? [] : [var.hierarchy_controller] + + content { + enabled = true + enable_hierarchical_resource_quota = each.value.enable_hierarchical_resource_quota + enable_pod_tree_labels = each.value.enable_pod_tree_labels + } + } + } +} diff --git a/modules/acm/main.tf b/modules/acm/main.tf index fb6a913ac0..6ffe06f166 100644 --- a/modules/acm/main.tf +++ b/modules/acm/main.tf @@ -14,50 +14,17 @@ * limitations under the License. */ -module "enable_acm" { - source = "terraform-google-modules/gcloud/google" - version = "~> 3.1" - - platform = "linux" - upgrade = true - additional_components = ["alpha"] - - service_account_key_file = var.service_account_key_file - create_cmd_entrypoint = "gcloud" - create_cmd_body = "alpha container hub config-management enable --project ${var.project_id}" - destroy_cmd_entrypoint = "gcloud" - destroy_cmd_body = "alpha container hub config-management disable --force --project ${var.project_id}" +locals { + private_key = var.create_ssh_key && var.ssh_auth_key == null ? tls_private_key.k8sop_creds[0].private_key_pem : var.ssh_auth_key + k8sop_creds_secret_key = var.secret_type == "cookiefile" ? "cookie_file" : var.secret_type } -module "acm_operator" { - - source = "../k8s-operator-crd-support" - - cluster_name = var.cluster_name - project_id = var.project_id - location = var.location - operator_path = var.operator_path - enable_multi_repo = var.enable_multi_repo - sync_repo = var.sync_repo - sync_branch = var.sync_branch - sync_revision = var.sync_revision - policy_dir = var.policy_dir - cluster_endpoint = var.cluster_endpoint - create_ssh_key = var.create_ssh_key - secret_type = var.secret_type - ssh_auth_key = var.ssh_auth_key - enable_policy_controller = var.enable_policy_controller - install_template_library = var.install_template_library - source_format = var.source_format - hierarchy_controller = var.hierarchy_controller - enable_log_denies = var.enable_log_denies - service_account_key_file = var.service_account_key_file - use_existing_context = var.use_existing_context - - operator_latest_manifest_url = "gs://config-management-release/released/latest/config-management-operator.yaml" - operator_cr_template_path = "${path.module}/templates/acm-config.yml.tpl" - operator_credential_namespace = "config-management-system" - operator_credential_name = "git-creds" +module "registration" { + source = "../fleet-membership" - rootsync_cr_template_path = "${path.module}/templates/root-sync.yml.tpl" + cluster_name = var.cluster_name + project_id = var.project_id + location = var.location + enable_fleet_registration = var.enable_fleet_registration + membership_name = var.cluster_membership_id } diff --git a/modules/acm/outputs.tf b/modules/acm/outputs.tf index 6bb2a808e3..316bea10cd 100644 --- a/modules/acm/outputs.tf +++ b/modules/acm/outputs.tf @@ -16,10 +16,13 @@ output "git_creds_public" { description = "Public key of SSH keypair to allow the Anthos Config Management Operator to authenticate to your Git repository." - value = module.acm_operator.git_creds_public + value = var.create_ssh_key ? coalesce(tls_private_key.k8sop_creds.*.public_key_openssh...) : null } output "wait" { description = "An output to use when you want to depend on cmd finishing" - value = module.acm_operator.wait + value = google_gke_hub_feature_membership.main.membership + depends_on = [ + google_gke_hub_feature_membership.main + ] } diff --git a/modules/acm/templates/acm-config.yml.tpl b/modules/acm/templates/acm-config.yml.tpl deleted file mode 100644 index e8ebeb966e..0000000000 --- a/modules/acm/templates/acm-config.yml.tpl +++ /dev/null @@ -1,23 +0,0 @@ -apiVersion: configmanagement.gke.io/v1 -kind: ConfigManagement -metadata: - name: config-management -spec: - # clusterName is required and must be unique among all managed clusters - clusterName: ${cluster_name} - policyController: - enabled: ${enable_policy_controller} - templateLibraryInstalled: ${install_template_library} - logDeniesEnabled: ${enable_log_denies} -%{ if enable_multi_repo ~} - enableMultiRepo: true -%{ else ~} - git: - syncRepo: ${sync_repo} - secretType: ${secret_type} - ${policy_dir_node} - ${sync_branch_node} - ${sync_revision_node} - ${source_format_node} -%{ endif ~} - ${hierarchy_controller_map_node} diff --git a/modules/acm/templates/root-sync.yml.tpl b/modules/acm/templates/root-sync.yml.tpl deleted file mode 100644 index c2ba6ec520..0000000000 --- a/modules/acm/templates/root-sync.yml.tpl +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: configsync.gke.io/v1beta1 -kind: RootSync -metadata: - name: root-sync - namespace: config-management-system -spec: - ${source_format_node} - git: - repo: ${sync_repo} - auth: ${secret_type} - ${policy_dir_node} - ${sync_branch_node} - ${sync_revision_node} - ${secret_ref_node} diff --git a/modules/acm/variables.tf b/modules/acm/variables.tf index dd11bacccf..040e1468a3 100644 --- a/modules/acm/variables.tf +++ b/modules/acm/variables.tf @@ -29,18 +29,25 @@ variable "location" { type = string } -variable "operator_path" { - description = "Path to the operator yaml config. If unset, will download from GCS releases." - type = string - default = null +variable "enable_fleet_feature" { + description = "Whether to enable the ACM feature on the fleet." + type = bool + default = true } -variable "enable_multi_repo" { - description = "Whether to use ACM Config Sync [multi-repo mode](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/multi-repo)." +variable "enable_fleet_registration" { + description = "Whether to create a new membership." type = bool - default = false + default = true } +variable "cluster_membership_id" { + description = "The cluster membership ID. If unset, one will be autogenerated." + type = string + default = "" +} + +# Config Sync variables variable "sync_repo" { description = "ACM Git repo address" type = string @@ -58,15 +65,23 @@ variable "sync_revision" { default = "" } +variable "source_format" { + description = "Configures a non-hierarchical repo if set to 'unstructured'. Uses [ACM defaults](https://cloud.google.com/anthos-config-management/docs/how-to/installing#configuring-config-management-operator) when unset." + type = string + default = "" +} + variable "policy_dir" { description = "Subfolder containing configs in ACM Git repo. If un-set, uses Config Management default." type = string default = "" } -variable "cluster_endpoint" { - description = "Kubernetes cluster endpoint." +# Config Sync Auth config +variable "secret_type" { + description = "git authentication secret type, is passed through to ConfigManagement spec.git.secretType. Overriden to value 'ssh' if `create_ssh_key` is true" type = string + default = "ssh" } variable "create_ssh_key" { @@ -75,18 +90,13 @@ variable "create_ssh_key" { default = true } -variable "secret_type" { - description = "git authentication secret type, is passed through to ConfigManagement spec.git.secretType. Overriden to value 'ssh' if `create_ssh_key` is true" - type = string - default = "ssh" -} - variable "ssh_auth_key" { description = "Key for Git authentication. Overrides 'create_ssh_key' variable. Can be set using 'file(path/to/file)'-function." type = string default = null } +# Policy Controller config variable "enable_policy_controller" { description = "Whether to enable the ACM Policy Controller on the cluster" type = bool @@ -99,31 +109,15 @@ variable "install_template_library" { default = true } -variable "source_format" { - description = "Configures a non-hierarchical repo if set to 'unstructured'. Uses [ACM defaults](https://cloud.google.com/anthos-config-management/docs/how-to/installing#configuring-config-management-operator) when unset." - type = string - default = "" -} - -variable "hierarchy_controller" { - description = "Configurations for Hierarchy Controller. See [Hierarchy Controller docs](https://cloud.google.com/anthos-config-management/docs/how-to/installing-hierarchy-controller) for more details" - type = map(any) - default = null -} - variable "enable_log_denies" { description = "Whether to enable logging of all denies and dryrun failures for ACM Policy Controller." type = bool default = false } -variable "service_account_key_file" { - description = "Path to service account key file to auth as for running `gcloud container clusters get-credentials`." - default = "" -} - -variable "use_existing_context" { - description = "Use existing kubecontext to auth kube-api." - type = bool - default = false +# Hierarchy Controller config +variable "hierarchy_controller" { + description = "Configurations for Hierarchy Controller. See [Hierarchy Controller docs](https://cloud.google.com/anthos-config-management/docs/how-to/installing-hierarchy-controller) for more details" + type = map(any) + default = null } diff --git a/modules/acm/versions.tf b/modules/acm/versions.tf new file mode 100644 index 0000000000..3175b67d98 --- /dev/null +++ b/modules/acm/versions.tf @@ -0,0 +1,55 @@ + +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + required_version = ">= 0.13.0" + + provider_meta "google" { + module_name = "blueprints/terraform/terraform-google-kubernetes-engine:acm/v15.0.2" + } + + required_providers { + google = { + source = "hashicorp/google-beta" + version = ">= 4.19.0, < 5.0" + } + google-beta = { + source = "hashicorp/google-beta" + version = ">= 4.19.0, < 5.0" + } + kubernetes = { + source = "hashicorp/kubernetes" + version = "~> 2.11" + } + tls = { + source = "hashicorp/tls" + version = ">= 3.3" + } + null = { + source = "hashicorp/null" + version = ">= 3.1" + } + random = { + source = "hashicorp/random" + version = ">= 3.1" + } + time = { + source = "hashicorp/time" + version = ">= 0.7" + } + } +} diff --git a/modules/config-sync/.gitignore b/modules/config-sync/.gitignore deleted file mode 100644 index ca3790e410..0000000000 --- a/modules/config-sync/.gitignore +++ /dev/null @@ -1,5 +0,0 @@ -# This fill will be always downloaded by terraform local-exec command from gc bucket -config-management-operator.yaml -/terraform.tfvars -/apply.out -/local.tfvars diff --git a/modules/config-sync/README.md b/modules/config-sync/README.md deleted file mode 100644 index 2a2c91bf15..0000000000 --- a/modules/config-sync/README.md +++ /dev/null @@ -1,72 +0,0 @@ -# Terraform Kubernetes Engine Config Sync Submodule - -This module installs [Config Sync](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync) in a Kubernetes cluster. - -Specifically, this module automates the following steps for [installing Config -Sync](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/installing): -1. Installing the Config Sync Operator manifest onto your cluster. -2. Using an existing or generating a new SSH key for accessing Git and providing it to the Operator -3. Configuring the Operator to connect to your git repository - -## Usage - -The following is an example minimal usage. Please see the -[variables.tf](variables.tf) file for more details and expected values and -types. - -```tf -module "config_sync" { - source = "terraform-google-modules/kubernetes-engine/google//modules/config-sync" - - project_id = "my-project-id" - cluster_name = "my-cluster-name" - location = module.gke.location - cluster_endpoint = module.gke.endpoint - - sync_repo = "git@github.com:GoogleCloudPlatform/csp-config-management.git" - sync_branch = "1.0.0" - policy_dir = "foo-corp" -} -``` - -To deploy this config: -1. Run `terraform apply` -2. Inspect the `git_creds_public` [output](#outputs) to retrieve the public key - used for accessing Git. Whitelist this key for access to your Git - repo. Instructions for some popular Git hosting providers are included for - convenience: - - * [Cloud Souce Repositories](https://cloud.google.com/source-repositories/docs/authentication#ssh) - * [Bitbucket](https://confluence.atlassian.com/bitbucket/set-up-an-ssh-key-728138079.html) - * [GitHub](https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/) - * [Gitlab](https://docs.gitlab.com/ee/ssh/) - - - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| cluster\_endpoint | Kubernetes cluster endpoint. | `string` | n/a | yes | -| cluster\_name | GCP cluster name used to reach cluster and which becomes the cluster name in the Config Sync kubernetes custom resource. | `string` | n/a | yes | -| create\_ssh\_key | Controls whether a key will be generated for Git authentication | `bool` | `true` | no | -| enable\_multi\_repo | Whether to use ACM Config Sync [multi-repo mode](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/multi-repo). | `bool` | `false` | no | -| hierarchy\_controller | Configurations for Hierarchy Controller. See [Hierarchy Controller docs](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/installing-hierarchy-controller) for more details. | `map(any)` | `null` | no | -| location | GCP location used to reach cluster. | `string` | n/a | yes | -| operator\_path | Path to the operator yaml config. If unset, will download from GCS releases. | `string` | `null` | no | -| policy\_dir | Subfolder containing configs in ACM Git repo. If un-set, uses Config Management default. | `string` | `""` | no | -| project\_id | GCP project\_id used to reach cluster. | `string` | n/a | yes | -| secret\_type | credential secret type, passed through to ConfigManagement spec.git.secretType. Overriden to value 'ssh' if `create_ssh_key` is true | `string` | n/a | yes | -| source\_format | Configures a non-hierarchical repo if set to 'unstructured'. Uses [Config Sync defaults](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/installing#configuring-config-management-operator) when unset. | `string` | `""` | no | -| ssh\_auth\_key | Key for Git authentication. Overrides 'create\_ssh\_key' variable. Can be set using 'file(path/to/file)'-function. | `string` | `null` | no | -| sync\_branch | ACM repo Git branch. If un-set, uses Config Management default. | `string` | `""` | no | -| sync\_repo | ACM Git repo address | `string` | n/a | yes | -| sync\_revision | ACM repo Git revision. If un-set, uses Config Management default. | `string` | `""` | no | - -## Outputs - -| Name | Description | -|------|-------------| -| git\_creds\_public | Public key of SSH keypair to allow the Config Sync Operator to authenticate to your Git repository. | - - diff --git a/modules/config-sync/main.tf b/modules/config-sync/main.tf deleted file mode 100644 index 052cacb3b3..0000000000 --- a/modules/config-sync/main.tf +++ /dev/null @@ -1,43 +0,0 @@ -/** - * Copyright 2018 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -module "configsync_operator" { - - source = "../k8s-operator-crd-support" - - cluster_name = var.cluster_name - project_id = var.project_id - location = var.location - operator_path = var.operator_path - enable_multi_repo = var.enable_multi_repo - sync_repo = var.sync_repo - sync_branch = var.sync_branch - sync_revision = var.sync_revision - policy_dir = var.policy_dir - cluster_endpoint = var.cluster_endpoint - create_ssh_key = var.create_ssh_key - secret_type = var.secret_type - ssh_auth_key = var.ssh_auth_key - source_format = var.source_format - hierarchy_controller = var.hierarchy_controller - - operator_latest_manifest_url = "gs://config-management-release/released/latest/config-sync-operator.yaml" - operator_cr_template_path = "${path.module}/templates/config-sync-config.yml.tpl" - operator_credential_namespace = "config-management-system" - operator_credential_name = "git-creds" - - rootsync_cr_template_path = "${path.module}/templates/root-sync.yml.tpl" -} diff --git a/modules/config-sync/templates/config-sync-config.yml.tpl b/modules/config-sync/templates/config-sync-config.yml.tpl deleted file mode 100644 index c288213fa9..0000000000 --- a/modules/config-sync/templates/config-sync-config.yml.tpl +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: configmanagement.gke.io/v1 -kind: ConfigManagement -metadata: - name: config-management -spec: - # clusterName is required and must be unique among all managed clusters - clusterName: ${cluster_name} - git: - syncRepo: ${sync_repo} - secretType: ${secret_type} - ${sync_branch_node} - ${policy_dir_node} - ${source_format_node} - ${hierarchy_controller_map_node} diff --git a/modules/config-sync/templates/root-sync.yml.tpl b/modules/config-sync/templates/root-sync.yml.tpl deleted file mode 100644 index c2ba6ec520..0000000000 --- a/modules/config-sync/templates/root-sync.yml.tpl +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: configsync.gke.io/v1beta1 -kind: RootSync -metadata: - name: root-sync - namespace: config-management-system -spec: - ${source_format_node} - git: - repo: ${sync_repo} - auth: ${secret_type} - ${policy_dir_node} - ${sync_branch_node} - ${sync_revision_node} - ${secret_ref_node} diff --git a/modules/config-sync/variables.tf b/modules/config-sync/variables.tf deleted file mode 100644 index 512786dc71..0000000000 --- a/modules/config-sync/variables.tf +++ /dev/null @@ -1,99 +0,0 @@ -/** - * Copyright 2018 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -variable "cluster_name" { - description = "GCP cluster name used to reach cluster and which becomes the cluster name in the Config Sync kubernetes custom resource." - type = string -} - -variable "project_id" { - description = "GCP project_id used to reach cluster." - type = string -} - -variable "location" { - description = "GCP location used to reach cluster." - type = string -} - -variable "operator_path" { - description = "Path to the operator yaml config. If unset, will download from GCS releases." - type = string - default = null -} - -variable "enable_multi_repo" { - description = "Whether to use ACM Config Sync [multi-repo mode](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/multi-repo)." - type = bool - default = false -} - -variable "sync_repo" { - description = "ACM Git repo address" - type = string -} - -variable "sync_branch" { - description = "ACM repo Git branch. If un-set, uses Config Management default." - type = string - default = "" -} - -variable "sync_revision" { - description = "ACM repo Git revision. If un-set, uses Config Management default." - type = string - default = "" -} - -variable "policy_dir" { - description = "Subfolder containing configs in ACM Git repo. If un-set, uses Config Management default." - type = string - default = "" -} - -variable "cluster_endpoint" { - description = "Kubernetes cluster endpoint." - type = string -} - -variable "create_ssh_key" { - description = "Controls whether a key will be generated for Git authentication" - type = bool - default = true -} - -variable "secret_type" { - description = "credential secret type, passed through to ConfigManagement spec.git.secretType. Overriden to value 'ssh' if `create_ssh_key` is true" - type = string -} - -variable "ssh_auth_key" { - description = "Key for Git authentication. Overrides 'create_ssh_key' variable. Can be set using 'file(path/to/file)'-function." - type = string - default = null -} - -variable "source_format" { - description = "Configures a non-hierarchical repo if set to 'unstructured'. Uses [Config Sync defaults](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/installing#configuring-config-management-operator) when unset." - type = string - default = "" -} - -variable "hierarchy_controller" { - description = "Configurations for Hierarchy Controller. See [Hierarchy Controller docs](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/installing-hierarchy-controller) for more details." - type = map(any) - default = null -} diff --git a/modules/fleet-membership/README.md b/modules/fleet-membership/README.md new file mode 100644 index 0000000000..fc960f1040 --- /dev/null +++ b/modules/fleet-membership/README.md @@ -0,0 +1,43 @@ +# Terraform Kubernetes Engine Hub Submodule + +This module [registers a GKE cluster](https://cloud.google.com/anthos/multicluster-management/connect/registering-a-cluster) in an Anthos [Environ](https://cloud.google.com/anthos/multicluster-management/environs). + +Specifically, this module automates the following steps for [registering a cluster](https://cloud.google.com/anthos/multicluster-management/connect/registering-a-cluster#register_cluster) + +## Usage + +There is [GKE full example](../../examples/simple_zonal_with_asm) and a [Generic K8s example](../../examples/simple_zonal_with_hub_kubeconfig) provided. Simple usage is as follows: + +```tf +module "hub" { + source = "terraform-google-modules/kubernetes-engine/google//modules/fleet-membership" + + project_id = "my-project-id" + cluster_name = "my-cluster-name" +} +``` + +To deploy this config: +1. Run `terraform apply` + + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| cluster\_name | The GKE cluster name | `string` | n/a | yes | +| enable\_fleet\_registration | Enables GKE Hub Registration when set to true | `bool` | `true` | no | +| hub\_project\_id | The project in which the GKE Hub belongs. Defaults to GKE cluster project\_id. | `string` | `""` | no | +| location | The location (zone or region) this cluster has been created in. | `string` | n/a | yes | +| membership\_name | Membership name that uniquely represents the cluster being registered. Defaults to `$project_id-$location-$cluster_name`. | `string` | `""` | no | +| module\_depends\_on | List of modules or resources this module depends on. | `list(any)` | `[]` | no | +| project\_id | The project in which the GKE cluster belongs. | `string` | n/a | yes | + +## Outputs + +| Name | Description | +|------|-------------| +| cluster\_membership\_id | The ID of the hub membership | +| wait | An output to use when you want to depend on registration finishing | + + diff --git a/modules/fleet-membership/main.tf b/modules/fleet-membership/main.tf new file mode 100644 index 0000000000..4a67696719 --- /dev/null +++ b/modules/fleet-membership/main.tf @@ -0,0 +1,49 @@ +/** + * Copyright 2018 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + hub_project_id = var.hub_project_id == "" ? var.project_id : var.hub_project_id + gke_hub_membership_name = var.membership_name != "" ? var.membership_name : "${var.project_id}-${var.location}-${var.cluster_name}" +} + +# Retrieve GKE cluster info +data "google_container_cluster" "primary" { + name = var.cluster_name + location = var.location + project = var.project_id +} + +# Give the service agent permissions on hub project +resource "google_project_iam_member" "hub_service_agent_gke" { + count = var.hub_project_id == "" ? 0 : 1 + project = var.hub_project_id + role = "roles/gkehub.serviceAgent" + member = "serviceAccount:${google_project_service_identity.sa_gkehub[0].email}" +} + +resource "google_project_iam_member" "hub_service_agent_hub" { + count = var.hub_project_id == "" ? 0 : 1 + project = local.hub_project_id + role = "roles/gkehub.serviceAgent" + member = "serviceAccount:${google_project_service_identity.sa_gkehub[0].email}" +} + +resource "google_project_service_identity" "sa_gkehub" { + count = var.hub_project_id == "" ? 0 : 1 + provider = google-beta + project = local.hub_project_id + service = "gkehub.googleapis.com" +} diff --git a/modules/fleet-membership/membership.tf b/modules/fleet-membership/membership.tf new file mode 100644 index 0000000000..c5ef4cce39 --- /dev/null +++ b/modules/fleet-membership/membership.tf @@ -0,0 +1,33 @@ +/** + * Copyright 2018 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +# Create the membership +resource "google_gke_hub_membership" "primary" { + count = var.enable_fleet_registration ? 1 : 0 + provider = google-beta + + project = local.hub_project_id + membership_id = local.gke_hub_membership_name + + endpoint { + gke_cluster { + resource_link = "//container.googleapis.com/${data.google_container_cluster.primary.id}" + } + } + authority { + issuer = "https://container.googleapis.com/v1/${data.google_container_cluster.primary.id}" + } +} diff --git a/modules/k8s-operator-crd-support/outputs.tf b/modules/fleet-membership/outputs.tf similarity index 56% rename from modules/k8s-operator-crd-support/outputs.tf rename to modules/fleet-membership/outputs.tf index 6b4f7e321c..fd605c91d3 100644 --- a/modules/k8s-operator-crd-support/outputs.tf +++ b/modules/fleet-membership/outputs.tf @@ -1,5 +1,5 @@ /** - * Copyright 2018 Google LLC + * Copyright 2022 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,15 +14,12 @@ * limitations under the License. */ -output "git_creds_public" { - description = "Public key of SSH keypair to allow the Anthos Operator to authenticate to your Git repository." - value = var.create_ssh_key ? tls_private_key.k8sop_creds.*.public_key_openssh : null +output "cluster_membership_id" { + description = "The ID of the hub membership" + value = var.enable_fleet_registration ? google_gke_hub_membership.primary[0].membership_id : local.gke_hub_membership_name } output "wait" { - description = "An output to use when you want to depend on cmd finishing" - value = var.enable_policy_controller ? module.wait_for_gatekeeper.wait : module.k8sop_config.wait + description = "An output to use when you want to depend on registration finishing" + value = var.enable_fleet_registration ? google_gke_hub_membership.primary[0].membership_id : local.gke_hub_membership_name } - - - diff --git a/modules/fleet-membership/variables.tf b/modules/fleet-membership/variables.tf new file mode 100644 index 0000000000..eec5b4fd9b --- /dev/null +++ b/modules/fleet-membership/variables.tf @@ -0,0 +1,48 @@ +/** + * Copyright 2018 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "cluster_name" { + description = "The GKE cluster name" + type = string +} + +variable "project_id" { + description = "The project in which the GKE cluster belongs." + type = string +} + +variable "hub_project_id" { + description = "The project in which the GKE Hub belongs. Defaults to GKE cluster project_id." + type = string + default = "" +} + +variable "location" { + description = "The location (zone or region) this cluster has been created in." + type = string +} + +variable "enable_fleet_registration" { + description = "Enables GKE Hub Registration when set to true" + type = bool + default = true +} + +variable "membership_name" { + description = "Membership name that uniquely represents the cluster being registered. Defaults to `$project_id-$location-$cluster_name`." + type = string + default = "" +} diff --git a/modules/hub/versions.tf b/modules/fleet-membership/versions.tf similarity index 100% rename from modules/hub/versions.tf rename to modules/fleet-membership/versions.tf diff --git a/modules/hub/README.md b/modules/hub-legacy/README.md similarity index 93% rename from modules/hub/README.md rename to modules/hub-legacy/README.md index d978c49f71..8686b2c56e 100644 --- a/modules/hub/README.md +++ b/modules/hub-legacy/README.md @@ -1,8 +1,10 @@ -# Terraform Kubernetes Engine Hub Submodule +# Hub Submodule + +**Note**: This module is legacy. For registering GKE clusters, you should switch to the new, native [module](../fleet-membership). This module [registers a Kubernetes cluster](https://cloud.google.com/anthos/multicluster-management/connect/registering-a-cluster) in an Anthos [Environ](https://cloud.google.com/anthos/multicluster-management/environs). -Specifically, this module automates the following steps for [registering a cluster](https://cloud.google.com/anthos/multicluster-management/connect/registering-a-cluster#register_cluster) +Specifically, this module automates the following steps for [registering a cluster](https://cloud.google.com/anthos/multicluster-management/connect/registering-a-cluster#register_cluster). ## Usage diff --git a/modules/hub/main.tf b/modules/hub-legacy/main.tf similarity index 100% rename from modules/hub/main.tf rename to modules/hub-legacy/main.tf diff --git a/modules/hub/outputs.tf b/modules/hub-legacy/outputs.tf similarity index 99% rename from modules/hub/outputs.tf rename to modules/hub-legacy/outputs.tf index 9ad677933c..c41635b172 100644 --- a/modules/hub/outputs.tf +++ b/modules/hub-legacy/outputs.tf @@ -14,6 +14,7 @@ * limitations under the License. */ + output "wait" { description = "An output to use when you want to depend on registration finishing" value = module.gke_hub_registration.wait diff --git a/modules/hub/scripts/gke_hub_registration.sh b/modules/hub-legacy/scripts/gke_hub_registration.sh similarity index 100% rename from modules/hub/scripts/gke_hub_registration.sh rename to modules/hub-legacy/scripts/gke_hub_registration.sh diff --git a/modules/hub/scripts/gke_hub_unregister.sh b/modules/hub-legacy/scripts/gke_hub_unregister.sh similarity index 100% rename from modules/hub/scripts/gke_hub_unregister.sh rename to modules/hub-legacy/scripts/gke_hub_unregister.sh diff --git a/modules/hub/variables.tf b/modules/hub-legacy/variables.tf similarity index 100% rename from modules/hub/variables.tf rename to modules/hub-legacy/variables.tf diff --git a/modules/config-sync/outputs.tf b/modules/hub-legacy/versions.tf similarity index 70% rename from modules/config-sync/outputs.tf rename to modules/hub-legacy/versions.tf index 61be7ba700..eaa4b37431 100644 --- a/modules/config-sync/outputs.tf +++ b/modules/hub-legacy/versions.tf @@ -1,5 +1,6 @@ + /** - * Copyright 2018 Google LLC + * Copyright 2021 Google LLC * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,8 +15,10 @@ * limitations under the License. */ -output "git_creds_public" { - description = "Public key of SSH keypair to allow the Config Sync Operator to authenticate to your Git repository." - value = module.configsync_operator.git_creds_public -} +terraform { + required_version = ">= 0.13.0" + provider_meta "google" { + module_name = "blueprints/terraform/terraform-google-kubernetes-engine:hub/v16.1.0" + } +} diff --git a/modules/k8s-operator-crd-support/.gitignore b/modules/k8s-operator-crd-support/.gitignore deleted file mode 100644 index ca3790e410..0000000000 --- a/modules/k8s-operator-crd-support/.gitignore +++ /dev/null @@ -1,5 +0,0 @@ -# This fill will be always downloaded by terraform local-exec command from gc bucket -config-management-operator.yaml -/terraform.tfvars -/apply.out -/local.tfvars diff --git a/modules/k8s-operator-crd-support/main.tf b/modules/k8s-operator-crd-support/main.tf deleted file mode 100644 index e10f006b6e..0000000000 --- a/modules/k8s-operator-crd-support/main.tf +++ /dev/null @@ -1,196 +0,0 @@ -/** - * Copyright 2018 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -locals { - cluster_endpoint = "https://${var.cluster_endpoint}" - private_key = var.create_ssh_key && var.ssh_auth_key == null ? tls_private_key.k8sop_creds[0].private_key_pem : var.ssh_auth_key - k8sop_creds_secret_key = var.secret_type == "cookiefile" ? "cookie_file" : var.secret_type - should_download_manifest = var.operator_path == null ? true : false - manifest_path = local.should_download_manifest ? "${path.root}/.terraform/tmp/${var.project_id}-${var.cluster_name}/config-management-operator.yaml" : var.operator_path - sync_branch_property = var.enable_multi_repo ? "branch" : "syncBranch" - sync_branch_node = var.sync_branch != "" ? format("%s: %s", local.sync_branch_property, var.sync_branch) : "" - sync_revision_property = var.enable_multi_repo ? "revision" : "syncRev" - sync_revision_node = var.sync_revision != "" ? format("%s: %s", local.sync_revision_property, var.sync_revision) : "" - policy_dir_property = var.enable_multi_repo ? "dir" : "policyDir" - policy_dir_node = var.policy_dir != "" ? format("%s: %s", local.policy_dir_property, var.policy_dir) : "" - secret_ref_node = var.create_ssh_key == true || var.ssh_auth_key != null ? format("secretRef:\n name: %s", var.operator_credential_name) : "" - hierarchy_controller_map_node = var.hierarchy_controller == null ? "" : format("hierarchyController:\n %s", indent(4, replace(yamlencode(var.hierarchy_controller), "/((?:^|\n)[\\s-]*)\"([\\w-]+)\":/", "$1$2:"))) - source_format_node = var.source_format != "" ? format("sourceFormat: %s", var.source_format) : "" - append_arg_use_existing_context = var.use_existing_context ? "USE_EXISTING_CONTEXT_ARG" : "" -} - -module "k8sop_manifest" { - source = "terraform-google-modules/gcloud/google" - version = "~> 3.1" - - enabled = local.should_download_manifest - - create_cmd_entrypoint = "gsutil" - create_cmd_body = "cp ${var.operator_latest_manifest_url} ${local.manifest_path}" - destroy_cmd_entrypoint = "rm" - destroy_cmd_body = "-f ${local.manifest_path}" -} - - -module "k8s_operator" { - source = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper" - version = "~> 3.1" - - module_depends_on = [module.k8sop_manifest.wait, var.cluster_endpoint] - cluster_name = var.cluster_name - cluster_location = var.location - project_id = var.project_id - service_account_key_file = var.service_account_key_file - use_existing_context = var.use_existing_context - impersonate_service_account = var.impersonate_service_account - - kubectl_create_command = "kubectl apply -f ${local.manifest_path}" - kubectl_destroy_command = "kubectl delete -f ${local.manifest_path}" -} - - -resource "tls_private_key" "k8sop_creds" { - count = var.create_ssh_key ? 1 : 0 - algorithm = "RSA" - rsa_bits = 4096 -} - -module "k8sop_creds_secret" { - source = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper" - version = "~> 3.1" - - enabled = var.create_ssh_key == true || var.ssh_auth_key != null ? "true" : "false" - module_depends_on = [module.k8s_operator.wait] - cluster_name = var.cluster_name - cluster_location = var.location - project_id = var.project_id - service_account_key_file = var.service_account_key_file - use_existing_context = var.use_existing_context - impersonate_service_account = var.impersonate_service_account - - kubectl_create_command = local.private_key != null ? "kubectl create secret generic ${var.operator_credential_name} -n=${var.operator_credential_namespace} --from-literal=${local.k8sop_creds_secret_key}='${local.private_key}'" : "" - kubectl_destroy_command = "kubectl delete secret ${var.operator_credential_name} -n=${var.operator_credential_namespace}" -} - - -data "template_file" "k8sop_config" { - - template = file(var.operator_cr_template_path) - vars = { - cluster_name = var.cluster_name - enable_multi_repo = var.enable_multi_repo - sync_repo = var.sync_repo - sync_branch_node = local.sync_branch_node - sync_revision_node = local.sync_revision_node - policy_dir_node = local.policy_dir_node - secret_type = var.create_ssh_key ? "ssh" : var.secret_type - enable_policy_controller = var.enable_policy_controller ? "true" : "false" - install_template_library = var.install_template_library ? "true" : "false" - source_format_node = local.source_format_node - hierarchy_controller_map_node = local.hierarchy_controller_map_node - enable_log_denies = var.enable_log_denies - } -} - -module "k8sop_config" { - source = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper" - version = "~> 3.1" - - module_depends_on = [module.k8s_operator.wait, module.k8sop_creds_secret.wait] - cluster_name = var.cluster_name - cluster_location = var.location - project_id = var.project_id - create_cmd_triggers = { configmanagement = data.template_file.k8sop_config.rendered } - service_account_key_file = var.service_account_key_file - use_existing_context = var.use_existing_context - impersonate_service_account = var.impersonate_service_account - - kubectl_create_command = "kubectl apply -f - < /dev/null - export exit_code=$? - while [ ! " ${exit_code} " -eq 0 ] - do - sleep 5 - echo -e "Waiting for namespace config-mangement-system in cluster $1 to be created..." - kubectl --context "$1" get namespace config-management-system &> /dev/null - export exit_code=$? - done - echo -e "Namespace config-management-system in cluster $1 created." - - # Once namespace is created, check if config-managment pods are ready - kubectl --context "$1" -n config-management-system wait --timeout 60s --for=condition=Ready pod --all &> /dev/null - export exit_code=$? - - while [ ! " ${exit_code} " -eq 0 ] - do - sleep 5 - echo -e "Waiting for config-management pods in cluster $1 to become ready..." - kubectl --context "$1" -n config-management-system wait --timeout 60s --for=condition=Ready pod --all &> /dev/null - export exit_code=$? - done - - echo -e "Config-management pods in cluster $1 are ready." - return -} - -if [ "$#" -lt 3 ]; then - >&2 echo "Not all expected arguments set." - exit 1 -fi - -PROJECT_ID=$1 -CLUSTER_NAME=$2 -CLUSTER_LOCATION=$3 -USE_EXISTING_CONTEXT=$4 - -# Check if we need to use the current context -if [ -z ${USE_EXISTING_CONTEXT+x} ]; then - # GKE Cluster. Use the GKE cluster context - is_configsync_ready gke_"${PROJECT_ID}"_"${CLUSTER_LOCATION}"_"${CLUSTER_NAME}" -else - echo "USE_EXISTING_CONTEXT variable is set. Using current context to wait for deployment to be ready." - # Get the current context. This can be used for non GKE Clusters - CURRENT_CONTEXT=$(kubectl config current-context) - is_configsync_ready "${CURRENT_CONTEXT}" -fi diff --git a/modules/k8s-operator-crd-support/scripts/wait_for_gatekeeper.sh b/modules/k8s-operator-crd-support/scripts/wait_for_gatekeeper.sh deleted file mode 100755 index 108d2d4bca..0000000000 --- a/modules/k8s-operator-crd-support/scripts/wait_for_gatekeeper.sh +++ /dev/null @@ -1,92 +0,0 @@ -#!/usr/bin/env bash -# Copyright 2018 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -is_deployment_ready() { - kubectl --context "$1" -n "$2" get deploy "$3" &> /dev/null - export exit_code=$? - while [ ! " ${exit_code} " -eq 0 ] - do - sleep 5 - echo -e "Waiting for deployment $3 in cluster $1 to be created..." - kubectl --context "$1" -n "$2" get deploy "$3" &> /dev/null - export exit_code=$? - done - echo -e "Deployment $3 in cluster $1 created." - - # Once deployment is created, check for deployment status.availableReplicas is greater than 0 - availableReplicas=$(kubectl --context "$1" -n "$2" get deploy "$3" -o json | jq -r '.status.availableReplicas') - while [[ " ${availableReplicas} " == " null " ]] - do - sleep 5 - echo -e "Waiting for deployment $3 in cluster $1 to become ready..." - availableReplicas=$(kubectl --context "$1" -n "$2" get deploy "$3" -o json | jq -r '.status.availableReplicas') - done - - echo -e "$3 in cluster $1 is ready with replicas ${availableReplicas}." - return "${availableReplicas}" -} - -is_service_ready() { - kubectl --context "$1" -n "$2" get service "$3" &> /dev/null - export exit_code=$? - while [ ! " ${exit_code} " -eq 0 ] - do - sleep 5 - echo -e "Waiting for service $3 in cluster $1 to be created..." - kubectl --context "$1" -n "$2" get service "$3" &> /dev/null - export exit_code=$? - done - echo -e "Service $3 in cluster $1 created." - - # Once service is created, check endpoints is greater than 0 - kubectl --context "$1" -n "$2" get endpoints "$3" - export exit_code=$? - - while [ ! " ${exit_code} " -eq 0 ] - do - sleep 5 - echo -e "Waiting for endpoints for service $3 in cluster $1 to become ready..." - kubectl --context "$1" -n "$2" get endpoints "$3" - export exit_code=$? - done - - echo -e "Service $3 in cluster $1 is ready with endpoints." - return -} - -if [ "$#" -lt 3 ]; then - >&2 echo "Not all expected arguments set." - exit 1 -fi - -PROJECT_ID=$1 -CLUSTER_NAME=$2 -CLUSTER_LOCATION=$3 -USE_EXISTING_CONTEXT=$4 - -# Gatekeeper causes issues if not ready - -# Check if we need to use the current context -if [ -z ${USE_EXISTING_CONTEXT+x} ]; then - # GKE Cluster. Use the GKE cluster context - is_deployment_ready gke_"${PROJECT_ID}"_"${CLUSTER_LOCATION}"_"${CLUSTER_NAME}" gatekeeper-system gatekeeper-controller-manager - is_service_ready gke_"${PROJECT_ID}"_"${CLUSTER_LOCATION}"_"${CLUSTER_NAME}" gatekeeper-system gatekeeper-webhook-service -else - echo "USE_EXISTING_CONTEXT variable is set. Using current context to wait for deployment to be ready." - # Get the current context. This can be used for non GKE Clusters - CURRENT_CONTEXT=$(kubectl config current-context) - is_deployment_ready "${CURRENT_CONTEXT}" gatekeeper-system gatekeeper-controller-manager - is_service_ready "${CURRENT_CONTEXT}" gatekeeper-system gatekeeper-webhook-service -fi diff --git a/modules/k8s-operator-crd-support/variables.tf b/modules/k8s-operator-crd-support/variables.tf deleted file mode 100644 index 7affe693db..0000000000 --- a/modules/k8s-operator-crd-support/variables.tf +++ /dev/null @@ -1,170 +0,0 @@ -/** - * Copyright 2018 Google LLC - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -variable "cluster_name" { - description = "GCP cluster name used to reach cluster and which becomes the cluster name in the Config Sync kubernetes custom resource." - type = string -} - -variable "project_id" { - description = "GCP project_id used to reach cluster." - type = string -} - -variable "location" { - description = "GCP location used to reach cluster." - type = string -} - -variable "operator_path" { - description = "Path to the operator yaml config. If unset, will download from `var.operator_latest_manifest_url`." - type = string - default = null -} - -variable "operator_latest_manifest_url" { - description = "Url to the latest downloadable manifest for the operator. To be supplied by operator module providers, not end users." - type = string -} - -variable "enable_multi_repo" { - description = "Whether to use Config Sync [multi-repo mode](https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/how-to/multi-repo)." - type = bool - default = false -} - -variable "sync_repo" { - description = "ACM Git repo address" - type = string -} - -variable "secret_type" { - description = "git authentication secret type, is passed through to ConfigManagement spec.git.secretType. Overriden to value 'ssh' if `create_ssh_key` is true" - type = string -} - -variable "secret_ref_name" { - description = "Name of Secret to use for authentication (Config Sync multi-repo setup only). If un-set, uses Config Management default." - type = string - default = "" -} - -variable "sync_branch" { - description = "ACM repo Git branch. If un-set, uses Config Management default." - type = string - default = "" -} - -variable "sync_revision" { - description = "ACM repo Git revision. If un-set, uses Config Management default." - type = string - default = "" -} - -variable "policy_dir" { - description = "Subfolder containing configs in ACM Git repo. If un-set, uses Config Management default." - type = string - default = "" -} - -variable "cluster_endpoint" { - description = "Kubernetes cluster endpoint." - type = string -} - -variable "operator_credential_name" { - description = "Allows calling modules to specify the name of operator credentials to match what is expected." - type = string -} -variable "operator_credential_namespace" { - description = "Allows calling modules to specify the namespace for the operator credential to match what is expected." - type = string -} - -variable "create_ssh_key" { - description = "Controls whether a key will be generated for Git authentication" - type = bool - default = true -} - -variable "ssh_auth_key" { - description = "Key for Git authentication. Overrides 'create_ssh_key' variable. Can be set using 'file(path/to/file)'-function." - type = string - default = null -} - -variable "enable_policy_controller" { - description = "Whether to enable the ACM Policy Controller on the cluster" - type = bool - default = false -} - -variable "install_template_library" { - description = "Whether to install the default Policy Controller template library" - type = bool - default = false -} - -variable "operator_cr_template_path" { - description = "path to template file to use for the operator" - type = string -} - -variable "rootsync_cr_template_path" { - description = "path to template file to use for the root sync definition (Config Sync multi-repo setup only)" - type = string -} - -variable "source_format" { - description = <