Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add terraform code to deploy patch release notification service #6853

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions infra/aws/terraform/sig-release/OWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# See the OWNERS docs at https://go.k8s.io/owners

approvers:
- release-engineering-approvers
- sig-k8s-infra-leads
reviewers:
- release-engineering-reviewers
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# patch-release-notification service

This terraform code deploys the code to notify the K8s community about the cherry pick deadline for the patch releases.

The patch-release-notification code can be found in `cmd/patch-release-notification`

Right now, the terraform is applied manually by the release managers that have access to the AWS account for the SIG-release.

# Deploy

To deploy will require to have both repositories cloned:

- https://github.com/kubernetes/release/
- https://github.com/kubernetes/k8s.io

from https://github.com/kubernetes/k8s.io
Comment on lines +11 to +16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of doing this because I think it's going to be very hard to automate the Terraform execution in the future. @upodroid Maybe you can help here, I'm thinking out loud, but can we setup SA with IRSA in EKS build cluster that would allow pushing to this registry, and then we run a ProwJob that's going to use that SA to build the image and push it to the registry?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the main idea here is to use terraform to build the image when is needed and deploy the lambda, then everytime we run that and if there are changes in the code it will generate a new image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but it is not hard to automate, in the automation we clone both repos, we do this in several places already for other things


```
# loging to the AWS ECR
$ aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 433650573627.dkr.ecr.us-west-2.amazonaws.com

$ cd k8s.io/infra/aws/terraform/sig-release/patch-release-notification

$ terraform init

$ terraform plan -out=plan.out

$ terraform apply "plan.out"
```

_note_: you will need to configure your AWS credentials before.
_note2_: this is setup to run in the AWS SIG-Release account.
125 changes: 125 additions & 0 deletions infra/aws/terraform/sig-release/patch-release-notification/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
/*
Copyright 2024 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

provider "aws" {
region = var.region
}

resource "aws_sesv2_email_identity" "sig_release_email_identity" {
email_identity = var.email_identity
}

resource "aws_iam_role" "lambda_ses_role" {
name = "lambda_ses_role"
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Action = "sts:AssumeRole",
Effect = "Allow",
Principal = {
Service = "lambda.amazonaws.com"
}
}
]
})
}

resource "aws_iam_policy" "lambda_ses_policy" {
name = "lambda_ses_policy"
description = "IAM policy for Lambda to access SES"
policy = jsonencode({
Version = "2012-10-17",
Statement = [
{
Action = [
"ses:SendEmail",
"ses:SendRawEmail"
],
Effect = "Allow",
Resource = "*"
}
]
})
}

resource "aws_ecr_repository" "repo" {
name = var.repository
image_tag_mutability = "MUTABLE"

image_scanning_configuration {
scan_on_push = false
}
}

resource "aws_ecr_repository" "cherry_pick_notification_repo" {
name = "${var.repository}/patch-release-notification"
image_tag_mutability = "MUTABLE"

image_scanning_configuration {
scan_on_push = false
}
}

resource "ko_build" "cherry_pick_notification_image" {
repo = aws_ecr_repository.cherry_pick_notification_repo.repository_url
base_image = "public.ecr.aws/lambda/provided:al2023"
working_dir = "${path.module}/../../../../../../release/cmd/patch-release-notification"
importpath = "k8s.io/release/cmd/patch-release-notification"
}

resource "aws_iam_role_policy_attachment" "lambda_ses_policy_attachment" {
role = aws_iam_role.lambda_ses_role.name
policy_arn = aws_iam_policy.lambda_ses_policy.arn
}

resource "aws_lambda_function" "cherry_pick_notification" {
function_name = "patch-release-notification"
role = aws_iam_role.lambda_ses_role.arn
image_uri = ko_build.cherry_pick_notification_image.image_ref
package_type = "Image"

environment {
variables = {
FROM_EMAIL = var.email_identity
TO_EMAIL = var.to_email
SCHEDULE_PATH = var.schedule_path
DAYS_TO_ALERT = var.days_to_alert
NO_MOCK = var.no_mock
AWS_REGION = var.region
}
}
}

resource "aws_cloudwatch_event_rule" "trigger_lambda_cron" {
name = "trigger-patch-release-notification-cron"
description = "Trigger Lambda function on a schedule"
schedule_expression = "cron(0 16 * * ? *)" # Example cron expression to run at 16:00 PM UTC every day
}

resource "aws_cloudwatch_event_target" "trigger_lambda_target" {
rule = aws_cloudwatch_event_rule.trigger_lambda_cron.name
target_id = "send_email_lambda"
arn = aws_lambda_function.cherry_pick_notification.arn
}

resource "aws_lambda_permission" "allow_cloudwatch_to_invoke_lambda" {
statement_id = "AllowExecutionFromCloudWatch"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.cherry_pick_notification.function_name
principal = "events.amazonaws.com"
source_arn = aws_cloudwatch_event_rule.trigger_lambda_cron.arn
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
/*
Copyright 2024 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

output "email_identity_arn" {
value = aws_sesv2_email_identity.sig_release_email_identity.arn
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also commit terraform.tfvars (if no sensitive information) or add default values to this file.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will commit, but as we discussed here, I will need to refactor not to build the image here and use a prebuilt image that will be done in another job.

but, I need to know how to push the image to the AWS account and more specifically to the aws account we are interested in, how that can be done? cc @upodroid

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cpanato I would try the following approach if it's okay with @upodroid:

  • Create a new IAM role that can push to the said registry
  • Create a new ServiceAccount inside the EKS Prow build cluster
  • Bind SA and IAM role via IRSA
  • Create a ProwJob that utilizes the said SA and make it build and push the image

There might some additional steps between 1 and 2 in terms of configuring the AWS account, but we have done that before.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If y'all are okay with that, I can take care of setting up everything because I think I have access to all the resources/accounts that we need for this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we will build it using cloudbuild? or can i use ko (https://github.com/ko-build/ko) to do the image/push build?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to use ko, let me take a look.

Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/*
Copyright 2024 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

variable "region" {
description = "The AWS region to deploy the resources"
type = string
}

variable "email_identity" {
description = "The email address or domain to verify"
type = string
}

variable "to_email" {
description = "The email address to send the notification"
type = string
}

variable "no_mock" {
description = "if will send the message to [email protected] or just internal"
type = bool
}

variable "days_to_alert" {
description = "when to send the notification"
type = number
}

variable "schedule_path" {
description = "path where we can find the schedule.yaml"
type = string
}

variable "repository" {
description = "The ECR repository to use for the image"
type = string
default = ""
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
Copyright 2024 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

terraform {
backend "s3" {
bucket = "tf-state-sig-release"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this bucket manually created?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we need to bootstrap some how :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I have been doing before for both SIG Release and SIG K8s Infra is using CloudFormation: https://github.com/kubernetes/k8s.io/blob/main/infra/aws/cloudformation/iam/cdn.packages.k8s.io/cloudformation.yaml

Maybe we can take the same approach here if that's okay for y'all. It still requires manual bootstrapping (i.e. running that CloudFormation script), but all the resources are created automatically.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yikes! cloudformation, i thought that did not exist anymore LOL

but sounds good to me, i can add that, and delete the bucket that I've create before.
I will add that as well in this PR
thanks!

key = "cherry-pick-notification"
region = "us-west-2"
}

required_version = "1.8.0"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we relax the version if we don't have any dependency on 1.8.0?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure


required_providers {
ko = {
source = "ko-build/ko"
}

aws = {
source = "hashicorp/aws"
version = "5.51.1"
}
}
}