Skip to content

Commit

Permalink
feat: Switch to native Terraform resources for hub registration and ACM
Browse files Browse the repository at this point in the history
  • Loading branch information
morgante committed May 3, 2022
1 parent 151c8c4 commit 9d8912a
Show file tree
Hide file tree
Showing 54 changed files with 900 additions and 1,161 deletions.
133 changes: 132 additions & 1 deletion docs/upgrading_to_v21.0.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Upgrading to v21.0

The v21.0 release of *kubernetes-engine* is a backwards incompatible
release.

Expand All @@ -14,3 +13,135 @@ The [Terraform Kubernetes Engine Module](https://github.com/terraform-google-mod
### Kubernetes Provider upgrade
The Terraform Kubernetes Engine module now requires version 2.10 or higher of
the Kubernetes Provider.

### Hub module rewrite
The old [Hub submodule](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/v20.0.0/modules/hub)
has been renamed to `hub-legacy` and deprecated. It is replaced with a new [fleet membership](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/fleet-membership)
module to handle registering GKE clusters to [fleets](https://cloud.google.com/anthos/multicluster-management/fleets) using the native API.

The new module relies exclusively on native Terraform resources and should therefore be more robust.

### Migrating
For GKE clusters, you should update your configuration as follows:

```diff
module "register" {
- source = "terraform-google-modules/kubernetes-engine/google//modules/hub"
- version = "~> 20.0"
+ source = "terraform-google-modules/kubernetes-engine/google//modules/fleet-membership"
+ version = "~> 21.0"

project_id = "my-project-id"
cluster_name = "my-cluster-name"
- gke_hub_membership_name = "gke-membership"
+ membership_name = "gke-hub-membership"
location = module.gke.location
- cluster_endpoint = module.gke.endpoint
- gke_hub_sa_name = "sa-for-kind-cluster-membership"
- use_kubeconfig = true
- labels = "testlabel=usekubecontext"
- module_depends_on = [module.gke]
}
```

You also need to follow these migration steps:

1. Remove the old module from your state:

```
terraform state rm module.register
```
2. Remove the cluster from the fleet:
```
gcloud container fleet memberships delete gke-hub-membership-name
```
3. Apply the new configuration to re-register the cluster:
```
terraform apply
```
#### Legacy module
**The native API only supports registering GKE clusters**. Therefore, the old hub module is preserved as `hub-legacy`.
You can continue using it by updating your configuration to point to the new location.
```diff
module "register" {
- source = "terraform-google-modules/kubernetes-engine/google//modules/hub"
- version = "~> 20.0"
+ source = "terraform-google-modules/kubernetes-engine/google//modules/hub-legacy"
+ version = "~> 21.0"
project_id = "my-project-id"
cluster_name = "my-cluster-name"
location = module.gke.location
cluster_endpoint = module.gke.endpoint
}
```

### Anthos Config Management (ACM) and Config Sync Module Rewrite
Together with the rewrite of the Hub module, the ACM module also has been rewritten to use native resources.

You will need to follow these migration steps:

1. Update your configuration to use the new module:

```diff
module "acm" {
source = "terraform-google-modules/kubernetes-engine/google//modules/acm"
- version = "~> 20.0"
+ version = "~> 21.0"

project_id = "my-project-id"
cluster_name = "simple-zonal-cluster"
location = "us-central1-a"
- cluster_endpoint = module.auth.host

sync_repo = "[email protected]:GoogleCloudPlatform/csp-config-management.git"
sync_branch = "1.0.0"
policy_dir = "foo-corp"

secret_type = "ssh"
}
```

1. Make sure you have the `kubernetes` provider configured:

```hcl
provider "kubernetes" {
cluster_ca_certificate = module.auth.cluster_ca_certificate
host = module.auth.host
token = module.auth.token
}
```

1. Remove the old module from your state:

```
terraform state rm module.acm
```

2. Import the old `git-creds` secret into Terraform:

```
terraform import 'module.acm.module.acm_operator.kubernetes_secret_v1.creds' 'config-management-system/git-creds'
```

3. Apply the new configuration to re-register ACM and confirm everything is working:

```
terraform apply
```

#### Feature Activation

Only the first cluster in a fleet should activate the ACM fleet feature.
Other clusters should disable feature activation by setting `enable_fleet_feature = false`.

#### Config Sync Module Removed
The dedicated Config Sync submodule has been removed.
To use Config Sync, just invoke the ACM module with `enable_policy_controller = false`.
37 changes: 26 additions & 11 deletions examples/simple_zonal_with_acm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,38 @@ This example illustrates how to create a simple cluster and install [Anthos Conf

It incorporates the standard cluster module and the [ACM install module](../../modules/acm).

## Verifying Success

After applying the Terraform configuration, you can run the following commands to verify that your cluster has synced correctly:

1. Check ACM install status:

```
gcloud config set project $(terraform output --raw project_id)
gcloud alpha container hub config-management status
```
2. Connect to the cluster:
```
gcloud container clusters get-credentials $(terraform output --raw cluster_name) --zone=$(terraform output --raw location)
```
3. Confirm the `shipping-dev` namespace was created:
```
kubectl describe ns shipping-dev
```
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| acm\_policy\_dir | Subfolder containing configs in ACM Git repo | `string` | `"foo-corp"` | no |
| acm\_sync\_branch | Anthos config management Git branch | `string` | `"1.0.0"` | no |
| acm\_sync\_repo | Anthos config management Git repo | `string` | `"[email protected]:GoogleCloudPlatform/csp-config-management.git"` | no |
| cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no |
| ip\_range\_pods | The secondary ip range to use for pods | `any` | n/a | yes |
| ip\_range\_services | The secondary ip range to use for services | `any` | n/a | yes |
| network | The VPC network to host the cluster in | `any` | n/a | yes |
| operator\_path | Path to the operator yaml config. If unset, will download from GCS releases. | `string` | `null` | no |
| project\_id | The project ID to host the cluster in | `any` | n/a | yes |
| region | The region to host the cluster in | `any` | n/a | yes |
| subnetwork | The subnetwork to host the cluster in | `any` | n/a | yes |
| zones | The zone to host the cluster in (required if is a zonal cluster) | `list(string)` | n/a | yes |
| region | The region to host the cluster in | `string` | `"us-central1"` | no |
| zone | The zone to host the cluster in | `string` | `"us-central1-a"` | no |
## Outputs
Expand All @@ -36,7 +51,7 @@ It incorporates the standard cluster module and the [ACM install module](../../m
| location | n/a |
| master\_kubernetes\_version | The master Kubernetes version |
| network | n/a |
| project\_id | n/a |
| project\_id | Standard test outputs |
| region | n/a |
| service\_account | The default service account used for running nodes. |
| subnetwork | n/a |
Expand Down
19 changes: 10 additions & 9 deletions examples/simple_zonal_with_acm/acm.tf
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@
*/

module "acm" {
source = "../../modules/acm"
project_id = var.project_id
location = module.gke.location
cluster_name = module.gke.name
sync_repo = var.acm_sync_repo
sync_branch = var.acm_sync_branch
policy_dir = var.acm_policy_dir
cluster_endpoint = module.gke.endpoint
operator_path = var.operator_path
source = "../../modules/acm"
project_id = var.project_id
location = module.gke.location
cluster_name = module.gke.name

sync_repo = "[email protected]:GoogleCloudPlatform/csp-config-management.git"
sync_branch = "1.0.0"
policy_dir = "foo-corp"

secret_type = "ssh"
}
29 changes: 18 additions & 11 deletions examples/simple_zonal_with_acm/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ locals {
cluster_type = "simple-zonal"
}

provider "google" {
region = var.region
}

data "google_client_config" "default" {}

provider "kubernetes" {
Expand All @@ -27,17 +31,20 @@ provider "kubernetes" {
}

module "gke" {
source = "../../"
project_id = var.project_id
name = "${local.cluster_type}-cluster${var.cluster_name_suffix}"
regional = false
region = var.region
zones = var.zones
network = var.network
subnetwork = var.subnetwork
ip_range_pods = var.ip_range_pods
ip_range_services = var.ip_range_services
service_account = "create"
source = "../../"
project_id = var.project_id
regional = false
region = var.region
zones = [var.zone]

name = "${local.cluster_type}-cluster${var.cluster_name_suffix}"

network = google_compute_network.main.name
subnetwork = google_compute_subnetwork.main.name
ip_range_pods = google_compute_subnetwork.main.secondary_ip_range[0].range_name
ip_range_services = google_compute_subnetwork.main.secondary_ip_range[1].range_name

service_account = "create"
node_pools = [
{
name = "acm-node-pool"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/**
* Copyright 2018 Google LLC
* Copyright 2021 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand All @@ -20,16 +20,14 @@ resource "random_string" "suffix" {
upper = false
}

provider "google" {
project = var.project_ids[1]
}

resource "google_compute_network" "main" {
project = var.project_id
name = "cft-gke-test-${random_string.suffix.result}"
auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "main" {
project = var.project_id
name = "cft-gke-test-${random_string.suffix.result}"
ip_cidr_range = "10.0.0.0/17"
region = var.region
Expand Down
48 changes: 47 additions & 1 deletion examples/simple_zonal_with_acm/outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ output "client_token" {
}

output "ca_certificate" {
value = module.gke.ca_certificate
value = module.gke.ca_certificate
sensitive = true
}

output "service_account" {
Expand All @@ -38,3 +39,48 @@ output "acm_git_creds_public" {
value = module.acm.git_creds_public
}

# Standard test outputs
output "project_id" {
value = var.project_id
}

output "region" {
value = module.gke.region
}

output "cluster_name" {
description = "Cluster name"
value = module.gke.name
}

output "network" {
value = google_compute_network.main.name
}

output "subnetwork" {
value = google_compute_subnetwork.main.name
}

output "location" {
value = module.gke.location
}

output "ip_range_pods" {
description = "The secondary IP range used for pods"
value = google_compute_subnetwork.main.secondary_ip_range[0].range_name
}

output "ip_range_services" {
description = "The secondary IP range used for services"
value = google_compute_subnetwork.main.secondary_ip_range[1].range_name
}

output "zones" {
description = "List of zones in which the cluster resides"
value = module.gke.zones
}

output "master_kubernetes_version" {
description = "The master Kubernetes version"
value = module.gke.master_version
}
1 change: 0 additions & 1 deletion examples/simple_zonal_with_acm/test_outputs.tf

This file was deleted.

Loading

0 comments on commit 9d8912a

Please sign in to comment.