Skip to content

Commit

Permalink
Merge pull request #20 from caetano-colin/main
Browse files Browse the repository at this point in the history
chore: lint fixes and adding missing headers
  • Loading branch information
sleighton2022 authored May 2, 2024
2 parents 02a16a1 + a192376 commit 647ca4a
Show file tree
Hide file tree
Showing 72 changed files with 1,114 additions and 1,131 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,6 @@ helpers/foundation-deployer/.steps.json

# File to populate env vars used by Docker test runs
.envrc

#ignore directory pynb checkpoints
.ipynb_checkpoints/
8 changes: 4 additions & 4 deletions 1-org/envs/shared/ml_ops_org_policy.tf
Original file line number Diff line number Diff line change
Expand Up @@ -27,25 +27,25 @@ locals {
#Disable root access on new Vertex AI Workbench user-managed notebooks and instances
#Control ID: VAI-CO-4.3
#NIST 800-53: AC-3 AC-17 AC-20
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
"ainotebooks.disableRootAccess",

#Disable terminal on new Vertexx AI Workbench instances
#Control ID: VAI-CO-4.4
#NIST 800-53: AC-3 AC-17 AC-20
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
"ainotebooks.disableTerminal",

#Restrict public IP access on new Vertex AI Workbench notebooks and instances
#Control ID: VAI-CO-4.7
#NIST 800-53: AC-3 AC-17 AC-20
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
"ainotebooks.restrictPublicIp",

#Require automatic scheduled upgrades on new Vertex AI Workbench user-managed notebooks and instances
#Control ID: VAI-CO-4.6
#NIST 800-53: AC-3 AC-17 AC-20
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
#CRI Profile: PR.AC-3.1 PR.AC-3.2 PR.AC-4.1 PR.AC-4.2 PR.AC-4.3 PR.AC-6.1 PR.PT-3.1 PR.PT-4.1
"ainotebooks.requireAutoUpgradeSchedule",

#Require VPC Connector
Expand Down
2 changes: 1 addition & 1 deletion 2-environments/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get

### `N.B.` Read this before continuing further!!

A logging project will be created in every environment (`development`, `non-production`, `production`) when running this code. This project contains a storage bucket for the purposes of project logging within its respective environment. This requires the `[email protected]` group permissions for the storage bucket. Since foundations has more restricted security measures, a domain restriction constraint is enforced. This restraint will prevent Google service accounts to be added to any permissions. In order for this terraform code to execute without error, manual intervention must be made to ensure everything applies without issue.
A logging project will be created in every environment (`development`, `non-production`, `production`) when running this code. This project contains a storage bucket for the purposes of project logging within its respective environment. This requires the `[email protected]` group permissions for the storage bucket. Since foundations has more restricted security measures, a domain restriction constraint is enforced. This restraint will prevent Google service accounts to be added to any permissions. In order for this terraform code to execute without error, manual intervention must be made to ensure everything applies without issue.
You must disable the contraint in every folder that is about to be configured by terraform, push your code and then apply the contraint again:

#### Do this before you push development, non-production & production
Expand Down
2 changes: 1 addition & 1 deletion 4-projects/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,4 @@ credentials.json

# Ignore any tfplan files
*.tfplan
**/*.tfplan
**/*.tfplan
14 changes: 14 additions & 0 deletions 4-projects/business_unit_3/development/locals.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
locals {
repo_name = "bu3-composer"
business_code = "bu3"
Expand Down
14 changes: 14 additions & 0 deletions 4-projects/business_unit_3/development/versions.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
terraform {
required_version = ">= 0.13"

Expand Down
14 changes: 14 additions & 0 deletions 4-projects/business_unit_3/non-production/locals.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
locals {
repo_name = "bu3-composer"
business_code = "bu3"
Expand Down
14 changes: 14 additions & 0 deletions 4-projects/business_unit_3/production/locals.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
locals {
repo_name = "bu3-composer"
business_code = "bu3"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ resource "google_project_service_identity" "secretmanager_agent" {
service = "secretmanager.googleapis.com"
}

// Add Secret Manager Service Agent to key with encrypt/decrypt permissions
// Add Secret Manager Service Agent to key with encrypt/decrypt permissions
resource "google_kms_crypto_key_iam_member" "secretmanager_agent" {
for_each = module.app_service_catalog_project[0].kms_keys
crypto_key_id = each.value.id
Expand Down
14 changes: 14 additions & 0 deletions 4-projects/business_unit_3/shared/versions.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
terraform {
required_version = ">= 0.13"

Expand Down
2 changes: 1 addition & 1 deletion 4-projects/modules/composer_env/crypto.tf
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ resource "google_kms_crypto_key_iam_member" "app_key" {
member = "serviceAccount:${local.app_infra_pipeline_service_accounts[var.repo_name]}"
}

// Add Secret Manager Service Agent to key with encrypt/decrypt permissions
// Add Secret Manager Service Agent to key with encrypt/decrypt permissions
resource "google_kms_crypto_key_iam_binding" "secretmanager_agent" {
for_each = module.app_cloudbuild_project.crypto_key
crypto_key_id = each.value.id
Expand Down
2 changes: 1 addition & 1 deletion 4-projects/modules/composer_env/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ module "app_cloudbuild_project" {
# member = "serviceAccount:${local.app_infra_pipeline_service_accounts[var.repo_name]}"
# }

# // Add Secret Manager Service Agent to key with encrypt/decrypt permissions
# // Add Secret Manager Service Agent to key with encrypt/decrypt permissions
# resource "google_kms_crypto_key_iam_member" "secretmanager_agent" {
# for_each = module.app_cloudbuild_project.crypto_key
# crypto_key_id = each.value.id
Expand Down
31 changes: 24 additions & 7 deletions 4-projects/modules/composer_env/playground.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,31 @@

# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
sa_accounts = [
"sa1",
"sa2",
"sa3",
]
]

service_catalog_crypto_key = {
"projects/prj-d-kms-cgvl/locations/us-central1/keyRings/sample-keyring": {
"destroy_scheduled_duration": "86400s",
"id": "projects/prj-d-kms-cgvl/locations/us-central1/keyRings/sample-keyring/cryptoKeys/prj-d-bu3cmpsr-pipeln",
"id": ("projects/prj-d-kms-cgvl/locations/us-central1/keyRings"
"/sample-keyring/cryptoKeys/prj-d-bu3cmpsr-pipeln"),
"import_only": False,
"key_ring": "projects/prj-d-kms-cgvl/locations/us-central1/keyRings/sample-keyring",
"key_ring": ("projects/prj-d-kms-cgvl/locations/us-central1"
"/keyRings/sample-keyring"),
"labels": {},
"name": "prj-d-bu3cmpsr-pipeln",
"purpose": "ENCRYPT_DECRYPT",
Expand All @@ -26,9 +41,11 @@
},
"projects/prj-d-kms-cgvl/locations/us-east4/keyRings/sample-keyring": {
"destroy_scheduled_duration": "86400s",
"id": "projects/prj-d-kms-cgvl/locations/us-east4/keyRings/sample-keyring/cryptoKeys/prj-d-bu3cmpsr-pipeln",
"id": ("projects/prj-d-kms-cgvl/locations/us-east4/keyRings"
"/sample-keyring/cryptoKeys/prj-d-bu3cmpsr-pipeln"),
"import_only": False,
"key_ring": "projects/prj-d-kms-cgvl/locations/us-east4/keyRings/sample-keyring",
"key_ring": ("projects/prj-d-kms-cgvl/locations/us-east4"
"/keyRings/sample-keyring"),
"labels": {},
"name": "prj-d-bu3cmpsr-pipeln",
"purpose": "ENCRYPT_DECRYPT",
Expand All @@ -51,4 +68,4 @@
result_list.append({"id": value["id"], "sa_account": sa})

# Print the result list
print(result_list)
print(result_list)
14 changes: 14 additions & 0 deletions 4-projects/modules/composer_env/versions.tf
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
terraform {
required_version = ">= 0.13"

Expand Down
2 changes: 1 addition & 1 deletion 5-app-infra/0-gcp-policies/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,4 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get

```bash
cd ..
```
```
2 changes: 1 addition & 1 deletion 5-app-infra/1-artifact-publish/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS.
echo "remote_state_bucket = ${remote_state_bucket}"
sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars
```

1. Update `backend.tf` with your bucket from the infra pipeline output.

```bash
Expand Down
4 changes: 2 additions & 2 deletions 5-app-infra/2-artifact-publish-repo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,9 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS.
```shell
export ARTIFACT_PROJECT_ID=$(terraform -chdir="gcp-projects/business_unit_3/shared" output -raw common_artifacts_project_id)
echo ${ARTIFACT_PROJECT_ID}
```
```

1. Clone the freshly minted Cloud Source Repository that was created for this project.
1. Clone the freshly minted Cloud Source Repository that was created for this project.
```shell
gcloud source repos clone publish-artifacts --project=${ARTIFACT_PROJECT_ID}
```
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-13:latest
RUN echo "Hello World" > helloworld.txt
RUN echo "Hello World" > helloworld.txt

Original file line number Diff line number Diff line change
@@ -1,2 +1,16 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM us-docker.pkg.dev/cloud-aiplatform/prediction/tf2-cpu.2-8:cmle_op_images_20240312_0210_RC00

Original file line number Diff line number Diff line change
@@ -1,2 +1,16 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM us-docker.pkg.dev/vertex-ai/prediction/tf2-gpu.2-13:latest
RUN echo "Hello World" > helloworld.txt
RUN echo "Hello World" > helloworld.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,16 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM tensorflow/tensorflow:2.8.0
RUN pip install tensorflow-io==0.25.0 protobuf==3.20.0 google-cloud-bigquery==3.13.0 pandas==2.0.3 db-dtypes==1.2.0 google-cloud-aiplatform==1.36.0 google-cloud-storage==2.14.0 kfp google-cloud-pipeline-components
RUN pip install tensorflow-io==0.25.0 protobuf==3.20.0 google-cloud-bigquery==3.13.0 pandas==2.0.3 db-dtypes==1.2.0 google-cloud-aiplatform==1.36.0 google-cloud-storage==2.14.0 kfp google-cloud-pipeline-components
6 changes: 3 additions & 3 deletions 5-app-infra/3-service-catalog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,11 @@ file.
This project has two main purposes:

1. To deploy a pipeline and a bucket which is linked to a Google Cloud Repository that houses terraform modules for the use in Service Catalog.
Although Service Catalog itself must be manually deployed, the modules which will be used can still be automated.
Although Service Catalog itself must be manually deployed, the modules which will be used can still be automated.

2. To deploy infrastructure for operational environments (ie. `non-production` & `production`.)

The resoning behind utilizing one repository with two deployment methodologies is due to how close interactive (`development`) and operational environments are.
The resoning behind utilizing one repository with two deployment methodologies is due to how close interactive (`development`) and operational environments are.

The repository has the structure (truncated for brevity):
```
Expand Down Expand Up @@ -100,7 +100,7 @@ The repository has the structure (truncated for brevity):
│   ├── outputs.tf
│   └── variables.tf
```
Each folder under `modules` represents a terraform module.
Each folder under `modules` represents a terraform module.
When there is a change in any of the terraform module folders, the pipeline will find whichever module has been changed since the last push, `tar.gz` that file and place it in a bucket for Service Catalog to access.

This pipeline is listening to the `main` branch of this repository for changes in order for the modules to be uploaded to service catalog.
Expand Down
4 changes: 2 additions & 2 deletions 5-app-infra/4-service-catalog-repo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,9 @@ This repo provides a number of the [Google Service Catalog](https://cloud.google
```shell
export SERVICE_CATALOG_PROJECT_ID=$(terraform -chdir="gcp-projects/business_unit_3/shared" output -raw service_catalog_project_id)
echo ${SERVICE_CATALOG_PROJECT_ID}
```
```

1. Clone the freshly minted Cloud Source Repository that was created for this project.
1. Clone the freshly minted Cloud Source Repository that was created for this project.
```shell
gcloud source repos clone service-catalog --project=${SERVICE_CATALOG_PROJECT_ID}
```
Expand Down
3 changes: 2 additions & 1 deletion 5-app-infra/4-service-catalog-repo/modules/bucket/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,4 +145,5 @@ The following table outlines which of the suggested controls for Vertex Generati
|------|-------------|
| storage\_bucket | Storage Bucket. |

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,4 @@ web_server_allowed_ip_ranges = [
github_name_prefix = "github-composer-cloudbuild"
github_app_installation_id = "APP_INSTALATION_ID_HERE"
github_api_token = "GITHUB_API_TOKEN_HERE"
github_remote_uri = "LINK_TO_GITHUB_REPO_CONTAINING_DAGS"
github_remote_uri = "LINK_TO_GITHUB_REPO_CONTAINING_DAGS"
Loading

0 comments on commit 647ca4a

Please sign in to comment.