Skip to content

Commit 1c08b99

Browse files
romanini-ciandtDrFaust92apeabody
authored
feat: Add support for CMEK in Autopilot GKE module (#2230)
Signed-off-by: drfaust92 <ilia.lazebnik@gmail.com> Co-authored-by: Ilia Lazebnik <Ilia.lazebnik@gmail.com> Co-authored-by: Andrew Peabody <andrewpeabody@google.com>
1 parent 2566fec commit 1c08b99

File tree

16 files changed

+356
-8
lines changed

16 files changed

+356
-8
lines changed

Diff for: ‎autogen/main/cluster.tf.tmpl

+3-2
Original file line numberDiff line numberDiff line change
@@ -203,10 +203,11 @@ resource "google_container_cluster" "primary" {
203203
{% if autopilot_cluster == true %}
204204
cluster_autoscaling {
205205
dynamic "auto_provisioning_defaults" {
206-
for_each = (var.create_service_account || var.service_account != "") ? [1] : []
206+
for_each = (var.create_service_account || var.service_account != "" || var.boot_disk_kms_key != null) ? [1] : []
207207

208208
content {
209-
service_account = local.service_account
209+
service_account = local.service_account
210+
boot_disk_kms_key = var.boot_disk_kms_key
210211
}
211212
}
212213
}

Diff for: ‎autogen/main/variables.tf.tmpl

-2
Original file line numberDiff line numberDiff line change
@@ -447,14 +447,12 @@ variable "service_account_name" {
447447
default = ""
448448
}
449449

450-
{% if autopilot_cluster != true %}
451450
variable "boot_disk_kms_key" {
452451
type = string
453452
description = "The Customer Managed Encryption Key used to encrypt the boot disk attached to each node in the node pool, if not overridden in `node_pools`. This should be of the form projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME]. For more information about protecting resources with Cloud KMS Keys please see: https://cloud.google.com/compute/docs/disks/customer-managed-encryption"
454453
default = null
455454
}
456455

457-
{% endif %}
458456
variable "issue_client_certificate" {
459457
type = bool
460458
description = "Issues a client certificate to authenticate to the cluster endpoint. To maximize the security of your cluster, leave this option disabled. Client certificates don't automatically rotate and aren't easily revocable. WARNING: changing this after cluster creation is destructive!"

Diff for: ‎build/int.cloudbuild.yaml

+15
Original file line numberDiff line numberDiff line change
@@ -466,6 +466,21 @@ steps:
466466
- verify simple-autopilot-private-non-default-sa
467467
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
468468
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateNonDefaultSA --stage teardown --verbose']
469+
- id: apply simple-autopilot-private-cmek
470+
waitFor:
471+
- init-all
472+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
473+
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateCMEK --stage apply --verbose']
474+
- id: verify simple-autopilot-private-cmek
475+
waitFor:
476+
- apply simple-autopilot-private-cmek
477+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
478+
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateCMEK --stage verify --verbose']
479+
- id: teardown simple-autopilot-private-cmek
480+
waitFor:
481+
- verify simple-autopilot-private-cmek
482+
name: 'gcr.io/cloud-foundation-cicd/$_DOCKER_IMAGE_DEVELOPER_TOOLS:$_DOCKER_TAG_VERSION_DEVELOPER_TOOLS'
483+
args: ['/bin/bash', '-c', 'cft test run TestSimpleAutopilotPrivateCMEK --stage teardown --verbose']
469484
- id: apply simple-fleet-app-operator-permissions
470485
waitFor:
471486
- init-all

Diff for: ‎examples/simple_autopilot_private_cmek/README.md

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Simple Regional Autopilot Cluster
2+
3+
This example illustrates how to create a simple autopilot cluster with beta features and
4+
using a Customer Managed Encryption Keys (CMEK).
5+
6+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
7+
## Inputs
8+
9+
| Name | Description | Type | Default | Required |
10+
|------|-------------|------|---------|:--------:|
11+
| project\_id | The project ID to host the cluster in | `any` | n/a | yes |
12+
13+
## Outputs
14+
15+
| Name | Description |
16+
|------|-------------|
17+
| boot\_disk\_kms\_key | CMEK used for disk encryption |
18+
| cluster\_name | Cluster name |
19+
| kubernetes\_endpoint | The cluster endpoint |
20+
| location | n/a |
21+
| master\_kubernetes\_version | Kubernetes version of the master |
22+
| network\_name | The name of the VPC being created |
23+
| region | The region in which the cluster resides |
24+
| service\_account | The service account to default running nodes as if not overridden in `node_pools`. |
25+
| subnet\_names | The names of the subnet being created |
26+
| zones | List of zones in which the cluster resides |
27+
28+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
29+
30+
To provision this example, run the following from within this directory:
31+
- `terraform init` to get the plugins
32+
- `terraform plan` to see the infrastructure plan
33+
- `terraform apply` to apply the infrastructure build
34+
- `terraform destroy` to destroy the built infrastructure

Diff for: ‎examples/simple_autopilot_private_cmek/main.tf

+77
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
locals {
18+
cluster_type = "simple-autopilot-private-cmek"
19+
network_name = "simple-autopilot-private-cmek-network"
20+
subnet_name = "simple-autopilot-private-cmek-subnet"
21+
master_auth_subnetwork = "simple-autopilot-private-master-subnet"
22+
pods_range_name = "ip-range-pods-simple-autopilot-private"
23+
svc_range_name = "ip-range-svc-simple-autopilot-private"
24+
subnet_names = [for subnet_self_link in module.gcp-network.subnets_self_links : split("/", subnet_self_link)[length(split("/", subnet_self_link)) - 1]]
25+
}
26+
27+
data "google_client_config" "default" {}
28+
29+
data "google_project" "main" {
30+
project_id = var.project_id
31+
}
32+
33+
module "kms" {
34+
source = "terraform-google-modules/kms/google"
35+
version = "~> 3.2"
36+
37+
project_id = var.project_id
38+
key_protection_level = "HSM"
39+
location = "us-central1"
40+
keyring = "keyring"
41+
keys = ["key"]
42+
prevent_destroy = false
43+
}
44+
45+
resource "google_kms_crypto_key_iam_member" "main" {
46+
crypto_key_id = values(module.kms.keys)[0]
47+
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
48+
member = "serviceAccount:service-${data.google_project.main.number}@compute-system.iam.gserviceaccount.com"
49+
}
50+
51+
provider "kubernetes" {
52+
host = "https://${module.gke.endpoint}"
53+
token = data.google_client_config.default.access_token
54+
cluster_ca_certificate = base64decode(module.gke.ca_certificate)
55+
}
56+
57+
module "gke" {
58+
source = "terraform-google-modules/kubernetes-engine/google//modules/beta-autopilot-private-cluster"
59+
version = "~> 35.0"
60+
61+
project_id = var.project_id
62+
name = "${local.cluster_type}-cluster"
63+
regional = true
64+
region = "us-central1"
65+
network = module.gcp-network.network_name
66+
subnetwork = local.subnet_names[index(module.gcp-network.subnets_names, local.subnet_name)]
67+
ip_range_pods = local.pods_range_name
68+
ip_range_services = local.svc_range_name
69+
release_channel = "REGULAR"
70+
enable_vertical_pod_autoscaling = true
71+
enable_private_endpoint = true
72+
enable_private_nodes = true
73+
network_tags = [local.cluster_type]
74+
deletion_protection = false
75+
boot_disk_kms_key = values(module.kms.keys)[0]
76+
depends_on = [google_kms_crypto_key_iam_member.main]
77+
}

Diff for: ‎examples/simple_autopilot_private_cmek/network.tf

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
module "gcp-network" {
18+
source = "terraform-google-modules/network/google"
19+
version = ">= 7.5"
20+
21+
project_id = var.project_id
22+
network_name = local.network_name
23+
24+
subnets = [
25+
{
26+
subnet_name = local.subnet_name
27+
subnet_ip = "10.0.0.0/17"
28+
subnet_region = "us-central1"
29+
subnet_private_access = true
30+
},
31+
{
32+
subnet_name = local.master_auth_subnetwork
33+
subnet_ip = "10.60.0.0/17"
34+
subnet_region = "us-central1"
35+
},
36+
]
37+
38+
secondary_ranges = {
39+
(local.subnet_name) = [
40+
{
41+
range_name = local.pods_range_name
42+
ip_cidr_range = "192.168.0.0/18"
43+
},
44+
{
45+
range_name = local.svc_range_name
46+
ip_cidr_range = "192.168.64.0/18"
47+
},
48+
]
49+
}
50+
}

Diff for: ‎examples/simple_autopilot_private_cmek/outputs.tf

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
output "kubernetes_endpoint" {
18+
description = "The cluster endpoint"
19+
sensitive = true
20+
value = module.gke.endpoint
21+
}
22+
23+
output "cluster_name" {
24+
description = "Cluster name"
25+
value = module.gke.name
26+
}
27+
28+
output "location" {
29+
value = module.gke.location
30+
}
31+
32+
output "master_kubernetes_version" {
33+
description = "Kubernetes version of the master"
34+
value = module.gke.master_version
35+
}
36+
37+
output "service_account" {
38+
description = "The service account to default running nodes as if not overridden in `node_pools`."
39+
value = module.gke.service_account
40+
}
41+
42+
output "network_name" {
43+
description = "The name of the VPC being created"
44+
value = module.gcp-network.network_name
45+
}
46+
47+
output "subnet_names" {
48+
description = "The names of the subnet being created"
49+
value = module.gcp-network.subnets_names
50+
}
51+
52+
output "region" {
53+
description = "The region in which the cluster resides"
54+
value = module.gke.region
55+
}
56+
57+
output "zones" {
58+
description = "List of zones in which the cluster resides"
59+
value = module.gke.zones
60+
}
61+
62+
output "boot_disk_kms_key" {
63+
description = "CMEK used for disk encryption"
64+
value = values(module.kms.keys)[0]
65+
}

Diff for: ‎examples/simple_autopilot_private_cmek/variables.tf

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
variable "project_id" {
18+
description = "The project ID to host the cluster in"
19+
}

Diff for: ‎examples/simple_autopilot_private_cmek/versions.tf

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
terraform {
18+
required_providers {
19+
google = {
20+
source = "hashicorp/google"
21+
}
22+
kubernetes = {
23+
source = "hashicorp/kubernetes"
24+
}
25+
}
26+
required_version = ">= 1.3"
27+
}

Diff for: ‎modules/beta-autopilot-private-cluster/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ Then perform the following commands on the root folder:
7777
| additional\_ip\_range\_pods | List of _names_ of the additional secondary subnet ip ranges to use for pods | `list(string)` | `[]` | no |
7878
| allow\_net\_admin | (Optional) Enable NET\_ADMIN for the cluster. | `bool` | `null` | no |
7979
| authenticator\_security\_group | The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format gke-security-groups@yourdomain.com | `string` | `null` | no |
80+
| boot\_disk\_kms\_key | The Customer Managed Encryption Key used to encrypt the boot disk attached to each node in the node pool, if not overridden in `node_pools`. This should be of the form projects/[KEY\_PROJECT\_ID]/locations/[LOCATION]/keyRings/[RING\_NAME]/cryptoKeys/[KEY\_NAME]. For more information about protecting resources with Cloud KMS Keys please see: https://cloud.google.com/compute/docs/disks/customer-managed-encryption | `string` | `null` | no |
8081
| cluster\_ipv4\_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | `string` | `null` | no |
8182
| cluster\_resource\_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | `map(string)` | `{}` | no |
8283
| configure\_ip\_masq | Enables the installation of ip masquerading, which is usually no longer required when using aliasied IP addresses. IP masquerading uses a kubectl call, so when you have a private cluster, you will need access to the API server. | `bool` | `false` | no |

Diff for: ‎modules/beta-autopilot-private-cluster/cluster.tf

+3-2
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,11 @@ resource "google_container_cluster" "primary" {
8989

9090
cluster_autoscaling {
9191
dynamic "auto_provisioning_defaults" {
92-
for_each = (var.create_service_account || var.service_account != "") ? [1] : []
92+
for_each = (var.create_service_account || var.service_account != "" || var.boot_disk_kms_key != null) ? [1] : []
9393

9494
content {
95-
service_account = local.service_account
95+
service_account = local.service_account
96+
boot_disk_kms_key = var.boot_disk_kms_key
9697
}
9798
}
9899
}

Diff for: ‎modules/beta-autopilot-private-cluster/variables.tf

+6
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,12 @@ variable "service_account_name" {
240240
default = ""
241241
}
242242

243+
variable "boot_disk_kms_key" {
244+
type = string
245+
description = "The Customer Managed Encryption Key used to encrypt the boot disk attached to each node in the node pool, if not overridden in `node_pools`. This should be of the form projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME]. For more information about protecting resources with Cloud KMS Keys please see: https://cloud.google.com/compute/docs/disks/customer-managed-encryption"
246+
default = null
247+
}
248+
243249
variable "issue_client_certificate" {
244250
type = bool
245251
description = "Issues a client certificate to authenticate to the cluster endpoint. To maximize the security of your cluster, leave this option disabled. Client certificates don't automatically rotate and aren't easily revocable. WARNING: changing this after cluster creation is destructive!"

Diff for: ‎modules/beta-autopilot-public-cluster/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ Then perform the following commands on the root folder:
7272
| additional\_ip\_range\_pods | List of _names_ of the additional secondary subnet ip ranges to use for pods | `list(string)` | `[]` | no |
7373
| allow\_net\_admin | (Optional) Enable NET\_ADMIN for the cluster. | `bool` | `null` | no |
7474
| authenticator\_security\_group | The name of the RBAC security group for use with Google security groups in Kubernetes RBAC. Group name must be in format gke-security-groups@yourdomain.com | `string` | `null` | no |
75+
| boot\_disk\_kms\_key | The Customer Managed Encryption Key used to encrypt the boot disk attached to each node in the node pool, if not overridden in `node_pools`. This should be of the form projects/[KEY\_PROJECT\_ID]/locations/[LOCATION]/keyRings/[RING\_NAME]/cryptoKeys/[KEY\_NAME]. For more information about protecting resources with Cloud KMS Keys please see: https://cloud.google.com/compute/docs/disks/customer-managed-encryption | `string` | `null` | no |
7576
| cluster\_ipv4\_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | `string` | `null` | no |
7677
| cluster\_resource\_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | `map(string)` | `{}` | no |
7778
| configure\_ip\_masq | Enables the installation of ip masquerading, which is usually no longer required when using aliasied IP addresses. IP masquerading uses a kubectl call, so when you have a private cluster, you will need access to the API server. | `bool` | `false` | no |

Diff for: ‎modules/beta-autopilot-public-cluster/cluster.tf

+3-2
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,11 @@ resource "google_container_cluster" "primary" {
8989

9090
cluster_autoscaling {
9191
dynamic "auto_provisioning_defaults" {
92-
for_each = (var.create_service_account || var.service_account != "") ? [1] : []
92+
for_each = (var.create_service_account || var.service_account != "" || var.boot_disk_kms_key != null) ? [1] : []
9393

9494
content {
95-
service_account = local.service_account
95+
service_account = local.service_account
96+
boot_disk_kms_key = var.boot_disk_kms_key
9697
}
9798
}
9899
}

Diff for: ‎modules/beta-autopilot-public-cluster/variables.tf

+6
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,12 @@ variable "service_account_name" {
240240
default = ""
241241
}
242242

243+
variable "boot_disk_kms_key" {
244+
type = string
245+
description = "The Customer Managed Encryption Key used to encrypt the boot disk attached to each node in the node pool, if not overridden in `node_pools`. This should be of the form projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME]. For more information about protecting resources with Cloud KMS Keys please see: https://cloud.google.com/compute/docs/disks/customer-managed-encryption"
246+
default = null
247+
}
248+
243249
variable "issue_client_certificate" {
244250
type = bool
245251
description = "Issues a client certificate to authenticate to the cluster endpoint. To maximize the security of your cluster, leave this option disabled. Client certificates don't automatically rotate and aren't easily revocable. WARNING: changing this after cluster creation is destructive!"

0 commit comments

Comments
 (0)