feat: Switch to native Terraform resources for hub registration and ACM

Author: morgante

docs/upgrading_to_v21.0.md

@@ -1,5 +1,4 @@
 # Upgrading to v21.0
-
 The v21.0 release of *kubernetes-engine* is a backwards incompatible
 release.
 
@@ -14,3 +13,130 @@ The [Terraform Kubernetes Engine Module](https://github.com/terraform-google-mod
 ### Kubernetes Provider upgrade
 The Terraform Kubernetes Engine module now requires version 2.10 or higher of
 the Kubernetes Provider.
+
+### Hub module rewrite
+The old [Hub submodule](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/v20.0.0/modules/hub)
+has been renamed to `hub-legacy` and deprecated. It is replaced with a new [fleet membership](https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/master/modules/fleet-membership)
+module to handle registering GKE clusters to [fleets](https://cloud.google.com/anthos/multicluster-management/fleets) using the native API.
+
+The new module relies exclusively on native Terraform resources and should therefore be more robust.
+
+### Migrating
+For GKE clusters, you should update your configuration as follows:
+
+```diff
+  module "register" {
+-   source           = "terraform-google-modules/kubernetes-engine/google//modules/hub"
+-   version          = "~> 20.0"
++   source           = "terraform-google-modules/kubernetes-engine/google//modules/fleet-membership"
++   version          = "~> 21.0"
+
+    project_id       = "my-project-id"
+    cluster_name     = "my-cluster-name"
+-   gke_hub_membership_name = "gke-membership"
++   membership_name  = "gke-hub-membership"
+    location         = module.gke.location
+-   cluster_endpoint = module.gke.endpoint
+-   gke_hub_sa_name         = "sa-for-kind-cluster-membership"
+-   use_kubeconfig          = true
+-   labels                  = "testlabel=usekubecontext"
+}
+```
+
+You also need to follow these migration steps:
+
+1. Remove the old module from your state:
+
+    ```
+    terraform state rm module.register
+    ```
+
+2. Remove the cluster from the fleet:
+
+    ```
+    gcloud container fleet memberships delete gke-hub-membership-name
+    ```
+
+3. Apply the new configuration to re-register the cluster:
+
+    ```
+    terraform apply
+    ```
+
+#### Legacy module
+**The native API only supports registering GKE clusters**. Therefore, the old hub module is preserved as `hub-legacy`.
+
+You can continue using it by updating your configuration to point to the new location.
+
+```diff
+  module "register" {
+-   source           = "terraform-google-modules/kubernetes-engine/google//modules/hub"
+-   version          = "~> 20.0"
++   source           = "terraform-google-modules/kubernetes-engine/google//modules/hub-legacy"
++   version          = "~> 21.0"
+
+    project_id       = "my-project-id"
+    cluster_name     = "my-cluster-name"
+    location         = module.gke.location
+    cluster_endpoint = module.gke.endpoint
+  }
+```
+
+### Anthos Config Management (ACM) and Config Sync Module Rewrite
+Together with the rewrite of the Hub module, the ACM module also has been rewritten to use native resources.
+
+You will need to follow these migration steps:
+
+1. Update your configuration to use the new module:
+
+    ```diff
+      module "acm" {
+        source           = "terraform-google-modules/kubernetes-engine/google//modules/acm"
+    -   version          = "~> 20.0"
+    +   version          = "~> 21.0"
+
+        project_id      = "my-project-id"
+        cluster_name    = "simple-zonal-cluster"
+        location        = "us-central1-a"
+    -   cluster_endpoint = module.auth.host
+
+        sync_repo   = "git@github.com:GoogleCloudPlatform/csp-config-management.git"
+        sync_branch = "1.0.0"
+        policy_dir  = "foo-corp"
+
+        secret_type = "ssh"
+      }
+    ```
+
+1. Make sure you have the `kubernetes` provider configured:
+
+    ```hcl
+    provider "kubernetes" {
+      cluster_ca_certificate = module.auth.cluster_ca_certificate
+      host                   = module.auth.host
+      token                  = module.auth.token
+    }
+    ```
+
+1. Remove the old module from your state:
+
+    ```
+    terraform state rm module.acm
+    ```
+
+2. Import the old `git-creds` secret into Terraform:
+
+    ```
+    terraform import 'module.acm.module.acm_operator.kubernetes_secret_v1.creds' 'config-management-system/git-creds'
+    ```
+
+3. Apply the new configuration to re-register ACM and confirm everything is working:
+
+    ```
+    terraform apply
+    ```
+
+#### Feature Activation
+
+Only the first cluster in a fleet should activate the ACM fleet feature.
+Other clusters should disable feature activation by setting `enable_fleet_feature = false`.

examples/simple_zonal_with_acm/README.md

@@ -4,23 +4,38 @@ This example illustrates how to create a simple cluster and install [Anthos Conf
 
 It incorporates the standard cluster module and the [ACM install module](../../modules/acm).
 
+## Verifying Success
+
+After applying the Terraform configuration, you can run the following commands to verify that your cluster has synced correctly:
+
+1. Check ACM install status:
+
+    ```
+    gcloud config set project $(terraform output --raw project_id)
+    gcloud alpha container hub config-management status
+    ```
+
+2. Connect to the cluster:
+
+    ```
+    gcloud container clusters get-credentials $(terraform output --raw cluster_name) --zone=$(terraform output --raw location)
+    ```
+
+3. Confirm the `shipping-dev` namespace was created:
+
+    ```
+    kubectl describe ns shipping-dev
+    ```
+
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| acm\_policy\_dir | Subfolder containing configs in ACM Git repo | `string` | `"foo-corp"` | no |
-| acm\_sync\_branch | Anthos config management Git branch | `string` | `"1.0.0"` | no |
-| acm\_sync\_repo | Anthos config management Git repo | `string` | `"git@github.com:GoogleCloudPlatform/csp-config-management.git"` | no |
 | cluster\_name\_suffix | A suffix to append to the default cluster name | `string` | `""` | no |
-| ip\_range\_pods | The secondary ip range to use for pods | `any` | n/a | yes |
-| ip\_range\_services | The secondary ip range to use for services | `any` | n/a | yes |
-| network | The VPC network to host the cluster in | `any` | n/a | yes |
-| operator\_path | Path to the operator yaml config. If unset, will download from GCS releases. | `string` | `null` | no |
 | project\_id | The project ID to host the cluster in | `any` | n/a | yes |
-| region | The region to host the cluster in | `any` | n/a | yes |
-| subnetwork | The subnetwork to host the cluster in | `any` | n/a | yes |
-| zones | The zone to host the cluster in (required if is a zonal cluster) | `list(string)` | n/a | yes |
+| region | The region to host the cluster in | `string` | `"us-central1"` | no |
+| zone | The zone to host the cluster in | `string` | `"us-central1-a"` | no |
 
 ## Outputs
 
@@ -36,7 +51,7 @@ It incorporates the standard cluster module and the [ACM install module](../../m
 | location | n/a |
 | master\_kubernetes\_version | The master Kubernetes version |
 | network | n/a |
-| project\_id | n/a |
+| project\_id | Standard test outputs |
 | region | n/a |
 | service\_account | The default service account used for running nodes. |
 | subnetwork | n/a |

examples/simple_zonal_with_acm/acm.tf

@@ -15,13 +15,14 @@
  */
 
 module "acm" {
-  source           = "../../modules/acm"
-  project_id       = var.project_id
-  location         = module.gke.location
-  cluster_name     = module.gke.name
-  sync_repo        = var.acm_sync_repo
-  sync_branch      = var.acm_sync_branch
-  policy_dir       = var.acm_policy_dir
-  cluster_endpoint = module.gke.endpoint
-  operator_path    = var.operator_path
+  source       = "../../modules/acm"
+  project_id   = var.project_id
+  location     = module.gke.location
+  cluster_name = module.gke.name
+
+  sync_repo   = "git@github.com:GoogleCloudPlatform/csp-config-management.git"
+  sync_branch = "1.0.0"
+  policy_dir  = "foo-corp"
+
+  secret_type = "ssh"
 }

examples/simple_zonal_with_acm/main.tf

@@ -18,6 +18,10 @@ locals {
   cluster_type = "simple-zonal"
 }
 
+provider "google" {
+  region = var.region
+}
+
 data "google_client_config" "default" {}
 
 provider "kubernetes" {
@@ -27,17 +31,20 @@ provider "kubernetes" {
 }
 
 module "gke" {
-  source            = "../../"
-  project_id        = var.project_id
-  name              = "${local.cluster_type}-cluster${var.cluster_name_suffix}"
-  regional          = false
-  region            = var.region
-  zones             = var.zones
-  network           = var.network
-  subnetwork        = var.subnetwork
-  ip_range_pods     = var.ip_range_pods
-  ip_range_services = var.ip_range_services
-  service_account   = "create"
+  source     = "../../"
+  project_id = var.project_id
+  regional   = false
+  region     = var.region
+  zones      = [var.zone]
+
+  name = "${local.cluster_type}-cluster${var.cluster_name_suffix}"
+
+  network           = google_compute_network.main.name
+  subnetwork        = google_compute_subnetwork.main.name
+  ip_range_pods     = google_compute_subnetwork.main.secondary_ip_range[0].range_name
+  ip_range_services = google_compute_subnetwork.main.secondary_ip_range[1].range_name
+
+  service_account = "create"
   node_pools = [
     {
       name         = "acm-node-pool"

test/fixtures/simple_zonal/network.tf renamed to examples/simple_zonal_with_acm/network.tf

@@ -1,5 +1,5 @@
 /**
- * Copyright 2018 Google LLC
+ * Copyright 2021 Google LLC
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -20,16 +20,14 @@ resource "random_string" "suffix" {
   upper   = false
 }
 
-provider "google" {
-  project = var.project_ids[1]
-}
-
 resource "google_compute_network" "main" {
+  project                 = var.project_id
   name                    = "cft-gke-test-${random_string.suffix.result}"
   auto_create_subnetworks = false
 }
 
 resource "google_compute_subnetwork" "main" {
+  project       = var.project_id
   name          = "cft-gke-test-${random_string.suffix.result}"
   ip_cidr_range = "10.0.0.0/17"
   region        = var.region

examples/simple_zonal_with_acm/outputs.tf

@@ -25,7 +25,8 @@ output "client_token" {
 }
 
 output "ca_certificate" {
-  value = module.gke.ca_certificate
+  value     = module.gke.ca_certificate
+  sensitive = true
 }
 
 output "service_account" {
@@ -38,3 +39,48 @@ output "acm_git_creds_public" {
   value       = module.acm.git_creds_public
 }
 
+# Standard test outputs
+output "project_id" {
+  value = var.project_id
+}
+
+output "region" {
+  value = module.gke.region
+}
+
+output "cluster_name" {
+  description = "Cluster name"
+  value       = module.gke.name
+}
+
+output "network" {
+  value = google_compute_network.main.name
+}
+
+output "subnetwork" {
+  value = google_compute_subnetwork.main.name
+}
+
+output "location" {
+  value = module.gke.location
+}
+
+output "ip_range_pods" {
+  description = "The secondary IP range used for pods"
+  value       = google_compute_subnetwork.main.secondary_ip_range[0].range_name
+}
+
+output "ip_range_services" {
+  description = "The secondary IP range used for services"
+  value       = google_compute_subnetwork.main.secondary_ip_range[1].range_name
+}
+
+output "zones" {
+  description = "List of zones in which the cluster resides"
+  value       = module.gke.zones
+}
+
+output "master_kubernetes_version" {
+  description = "The master Kubernetes version"
+  value       = module.gke.master_version
+}