From 3a2e40a84f571fcf3cfc7b142c91df19b95efc63 Mon Sep 17 00:00:00 2001 From: Duda Martins <109083465+mariammartins@users.noreply.github.com> Date: Thu, 13 Jun 2024 14:01:44 -0300 Subject: [PATCH] Refactor 5-app-infra-step (#48) * Initial commit * moving gcp-policies step to root README * refactoring the publish-artifacts and service-catalog steps to the root README * refactoring the artifact publish and service catalog repos steps * changing step 5-vps-sc to root README * refactoring assets * updata service catalog module name * add VPC-SC on root README * add VPC-SC bash commands * update VPC-SC bash commands * changing step 7-ml-post-deployment to 6-ml README * refactoring modules and source repos * fix 6-machine-learning step after app-infra refactoring * update 6-machine-learning step README * update README * update README * Update 5-app-infra/README.md Co-authored-by: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> * update README * remove VPC-SC step * insertion of the VPC-SC step as a prerequisite for the example * merge of 6-machine-learning-steps and 6-mal-pipeline steps * Update 5-app-infra/README.md Co-authored-by: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> * Update 5-app-infra/README.md Co-authored-by: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> * Update 5-app-infra/README.md Co-authored-by: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> * update 6-machine-learning README * update README * update README * update 6-machine-learning step * update title * update README with terraform locally steps * update README * fix merge * fix merge * fix merge * update README * update README * fix lint and example * rename reference * fix end of file * fix title Machine Learning Pipeline * fix whitespace --------- Co-authored-by: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> --- .github/workflows/lint.yaml | 4 +- 5-app-infra/0-gcp-policies/README.md | 90 -- 5-app-infra/1-artifact-publish/.gitignore | 58 - 5-app-infra/1-artifact-publish/README.md | 280 ----- .../business_unit_3/shared/README.md | 15 - .../business_unit_3/shared/backend.tf | 22 - .../business_unit_3/shared/locals.tf | 20 - .../business_unit_3/shared/outputs.tf | 34 - .../shared/publish_artifacts.tf | 37 - .../business_unit_3/shared/remote.tf | 30 - .../business_unit_3/shared/variables.tf | 25 - .../common.auto.example.tfvars | 19 - 5-app-infra/2-artifact-publish-repo/README.md | 86 -- .../images/tf2-cpu.2-13:0.1/Dockerfile | 17 - .../images/tf2-gpu.2-13:0.1/Dockerfile | 16 - 5-app-infra/3-service-catalog/.gitignore | 58 - 5-app-infra/3-service-catalog/README.md | 298 ----- .../business_unit_3/shared/README.md | 16 - .../business_unit_3/shared/backend.tf | 22 - .../business_unit_3/shared/locals.tf | 21 - .../business_unit_3/shared/outputs.tf | 25 - .../business_unit_3/shared/remote.tf | 39 - .../business_unit_3/shared/service_catalog.tf | 25 - .../business_unit_3/shared/variables.tf | 25 - .../common.auto.example.tfvars | 26 - 5-app-infra/4-service-catalog-repo/README.md | 164 --- .../4-service-catalog-repo/img/workflow.png | Bin 87776 -> 0 bytes .../modules/artifact_registry/README.md | 74 -- .../modules/artifact_registry/data.tf | 34 - .../modules/artifact_registry/locals.tf | 23 - .../modules/artifact_registry/main.tf | 64 -- .../terraform.tfvars.example | 43 - .../modules/artifact_registry/variables.tf | 86 -- .../modules/bigquery/README.md | 76 -- .../modules/bigquery/data.tf | 34 - .../modules/bigquery/locals.tf | 21 - .../modules/bigquery/main.tf | 24 - .../modules/bigquery/variables.tf | 66 -- .../modules/bucket/README.md | 149 --- .../modules/bucket/data.tf | 38 - .../modules/bucket/locals.tf | 20 - .../modules/bucket/main.tf | 148 --- .../modules/bucket/outputs.tf | 20 - .../modules/bucket/variables.tf | 186 --- .../modules/composer/README.md | 129 --- .../modules/composer/data.tf | 66 -- .../modules/composer/locals.tf | 74 -- .../modules/composer/main.tf | 93 -- .../modules/composer/outputs.tf | 40 - .../modules/composer/pipeline.tf | 121 -- .../modules/composer/terraform.tfvars.example | 29 - .../modules/composer/variables.tf | 138 --- .../modules/composer/vpc.tf | 288 ----- .../modules/metadata/README.md | 70 -- .../modules/metadata/data.tf | 34 - .../modules/metadata/locals.tf | 19 - .../modules/metadata/main.tf | 26 - .../modules/metadata/outputs.tf | 20 - .../modules/metadata/variables.tf | 37 - .../modules/notebook/README.md | 137 --- .../modules/notebook/data.tf | 65 -- .../modules/notebook/locals.tf | 20 - .../modules/notebook/main.tf | 95 -- .../modules/notebook/outputs.tf | 50 - .../modules/notebook/variables.tf | 150 --- .../modules/pubsub/README.md | 82 -- .../modules/pubsub/data.tf | 35 - .../modules/pubsub/locals.tf | 20 - .../modules/pubsub/main.tf | 39 - .../modules/pubsub/outputs.tf | 20 - .../modules/pubsub/variables.tf | 52 - .../modules/secrets/README.md | 79 -- .../modules/secrets/data.tf | 39 - .../modules/secrets/locals.tf | 20 - .../modules/secrets/main.tf | 56 - .../modules/secrets/outputs.tf | 20 - .../modules/secrets/variables.tf | 36 - .../modules/tensorboard/README.md | 69 -- .../modules/tensorboard/data.tf | 36 - .../modules/tensorboard/locals.tf | 19 - .../modules/tensorboard/main.tf | 26 - .../modules/tensorboard/outputs.tf | 20 - .../modules/tensorboard/variables.tf | 37 - 5-app-infra/5-vpc-sc/README.md | 175 --- 5-app-infra/6-machine-learning/README.md | 407 ------- .../non-production/common.auto.tfvars | 1 - .../non-production/versions.tf | 43 - .../production/common.auto.tfvars | 1 - .../business_unit_3/production/versions.tf | 43 - .../README.md | 252 ----- 5-app-infra/README.md | 721 +++++++++++- .../modules/publish_artifacts/data.tf | 0 .../modules/publish_artifacts/locals.tf | 0 .../modules/publish_artifacts/main.tf | 0 .../modules/publish_artifacts/outputs.tf | 0 .../modules/publish_artifacts/variables.tf | 0 .../modules/publish_artifacts/versions.tf | 0 .../service_catalog}/data.tf | 0 .../service_catalog}/locals.tf | 0 .../service_catalog}/main.tf | 0 .../service_catalog}/outputs.tf | 0 .../service_catalog}/variables.tf | 0 .../projects/artifact-publish/README.md | 344 ------ .../shared/publish_artifacts.tf | 2 +- .../modules/publish_artifacts/data.tf | 35 - .../modules/publish_artifacts/locals.tf | 33 - .../modules/publish_artifacts/main.tf | 162 --- .../modules/publish_artifacts/outputs.tf | 20 - .../modules/publish_artifacts/variables.tf | 71 -- .../modules/publish_artifacts/versions.tf | 46 - .../projects/machine-learning/.gitignore | 16 - .../projects/machine-learning/README.md | 444 -------- .../business_unit_3/development/README.md | 15 - .../business_unit_3/development/backend.tf | 22 - .../development/common.auto.tfvars | 1 - .../business_unit_3/development/locals.tf | 23 - .../business_unit_3/development/main.tf | 26 - .../business_unit_3/development/outputs.tf | 16 - .../business_unit_3/development/remote.tf | 40 - .../business_unit_3/development/variables.tf | 35 - .../business_unit_3/development/versions.tf | 43 - .../business_unit_3/non-production/README.md | 20 - .../business_unit_3/non-production/backend.tf | 22 - .../non-production/common.auto.tfvars | 1 - .../business_unit_3/non-production/locals.tf | 23 - .../business_unit_3/non-production/main.tf | 56 - .../business_unit_3/non-production/outputs.tf | 49 - .../business_unit_3/non-production/remote.tf | 40 - .../non-production/variables.tf | 35 - .../non-production/versions.tf | 43 - .../business_unit_3/production/README.md | 20 - .../business_unit_3/production/backend.tf | 22 - .../production/common.auto.tfvars | 1 - .../business_unit_3/production/locals.tf | 23 - .../business_unit_3/production/main.tf | 57 - .../business_unit_3/production/outputs.tf | 49 - .../business_unit_3/production/remote.tf | 40 - .../business_unit_3/production/variables.tf | 35 - .../business_unit_3/production/versions.tf | 43 - .../common.auto.example.tfvars | 23 - .../machine-learning/modules/base_env/data.tf | 37 - .../machine-learning/modules/base_env/iam.tf | 169 --- .../machine-learning/modules/base_env/main.tf | 121 -- .../modules/base_env/outputs.tf | 50 - .../modules/base_env/roles.tf | 293 ----- .../modules/base_env/variables.tf | 380 ------- .../projects/service-catalog/README.md | 361 ------ .../business_unit_3/shared/common.auto.tfvars | 1 - .../business_unit_3/shared/service_catalog.tf | 2 +- .../service-catalog/modules/svc_ctlg/data.tf | 39 - .../modules/svc_ctlg/locals.tf | 41 - .../service-catalog/modules/svc_ctlg/main.tf | 143 --- .../modules/svc_ctlg/outputs.tf | 25 - .../modules/svc_ctlg/variables.tf | 44 - .../images/tf2-cpu.2-8:01/Dockerfile | 0 .../images/vertexpipeline:v2/Dockerfile | 0 6-ml-pipeline/dev/Readme.md | 164 --- Makefile | 2 +- README.md | 2 +- docs/assets/terraform/1-org/README.md | 11 + .../assets/terraform/2-environments/README.md | 11 + .../machine-learning-pipeline}/.gitignore | 0 examples/machine-learning-pipeline/README.md | 1002 +++++++++++++++++ .../business_unit_3/development/README.md | 0 .../business_unit_3/development/backend.tf | 0 .../development}/common.auto.tfvars | 0 .../business_unit_3/development/locals.tf | 0 .../business_unit_3/development/main.tf | 0 .../business_unit_3/development/outputs.tf | 0 .../business_unit_3/development/remote.tf | 0 .../business_unit_3/development/variables.tf | 0 .../business_unit_3/development}/versions.tf | 0 .../business_unit_3/non-production/README.md | 0 .../business_unit_3/non-production/backend.tf | 0 .../non-production}/common.auto.tfvars | 0 .../business_unit_3/non-production/locals.tf | 0 .../business_unit_3/non-production/main.tf | 0 .../business_unit_3/non-production/outputs.tf | 0 .../business_unit_3/non-production/remote.tf | 0 .../non-production/variables.tf | 0 .../non-production}/versions.tf | 0 .../business_unit_3/production/README.md | 0 .../business_unit_3/production/backend.tf | 0 .../production}/common.auto.tfvars | 0 .../business_unit_3/production/locals.tf | 0 .../business_unit_3/production/main.tf | 0 .../business_unit_3/production/outputs.tf | 0 .../business_unit_3/production/remote.tf | 0 .../business_unit_3/production/variables.tf | 0 .../business_unit_3/production}/versions.tf | 0 .../common.auto.example.tfvars | 0 .../modules/base_env/data.tf | 0 .../modules/base_env/iam.tf | 0 .../modules/base_env/main.tf | 0 .../modules/base_env/outputs.tf | 0 .../modules/base_env/roles.tf | 0 .../modules/base_env/variables.tf | 0 197 files changed, 1691 insertions(+), 9830 deletions(-) delete mode 100644 5-app-infra/0-gcp-policies/README.md delete mode 100644 5-app-infra/1-artifact-publish/.gitignore delete mode 100644 5-app-infra/1-artifact-publish/README.md delete mode 100644 5-app-infra/1-artifact-publish/business_unit_3/shared/README.md delete mode 100644 5-app-infra/1-artifact-publish/business_unit_3/shared/backend.tf delete mode 100644 5-app-infra/1-artifact-publish/business_unit_3/shared/locals.tf delete mode 100644 5-app-infra/1-artifact-publish/business_unit_3/shared/outputs.tf delete mode 100644 5-app-infra/1-artifact-publish/business_unit_3/shared/publish_artifacts.tf delete mode 100644 5-app-infra/1-artifact-publish/business_unit_3/shared/remote.tf delete mode 100644 5-app-infra/1-artifact-publish/business_unit_3/shared/variables.tf delete mode 100644 5-app-infra/1-artifact-publish/common.auto.example.tfvars delete mode 100644 5-app-infra/2-artifact-publish-repo/README.md delete mode 100644 5-app-infra/2-artifact-publish-repo/images/tf2-cpu.2-13:0.1/Dockerfile delete mode 100644 5-app-infra/2-artifact-publish-repo/images/tf2-gpu.2-13:0.1/Dockerfile delete mode 100644 5-app-infra/3-service-catalog/.gitignore delete mode 100644 5-app-infra/3-service-catalog/README.md delete mode 100644 5-app-infra/3-service-catalog/business_unit_3/shared/README.md delete mode 100644 5-app-infra/3-service-catalog/business_unit_3/shared/backend.tf delete mode 100644 5-app-infra/3-service-catalog/business_unit_3/shared/locals.tf delete mode 100644 5-app-infra/3-service-catalog/business_unit_3/shared/outputs.tf delete mode 100644 5-app-infra/3-service-catalog/business_unit_3/shared/remote.tf delete mode 100644 5-app-infra/3-service-catalog/business_unit_3/shared/service_catalog.tf delete mode 100644 5-app-infra/3-service-catalog/business_unit_3/shared/variables.tf delete mode 100644 5-app-infra/3-service-catalog/common.auto.example.tfvars delete mode 100644 5-app-infra/4-service-catalog-repo/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/img/workflow.png delete mode 100644 5-app-infra/4-service-catalog-repo/modules/artifact_registry/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/artifact_registry/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/artifact_registry/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/artifact_registry/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/artifact_registry/terraform.tfvars.example delete mode 100644 5-app-infra/4-service-catalog-repo/modules/artifact_registry/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bigquery/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bigquery/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bigquery/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bigquery/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bigquery/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bucket/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bucket/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bucket/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bucket/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bucket/outputs.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/bucket/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/outputs.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/pipeline.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/terraform.tfvars.example delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/composer/vpc.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/metadata/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/metadata/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/metadata/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/metadata/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/metadata/outputs.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/metadata/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/notebook/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/notebook/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/notebook/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/notebook/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/notebook/outputs.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/notebook/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/pubsub/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/pubsub/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/pubsub/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/pubsub/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/pubsub/outputs.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/pubsub/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/secrets/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/secrets/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/secrets/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/secrets/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/secrets/outputs.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/secrets/variables.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/tensorboard/README.md delete mode 100644 5-app-infra/4-service-catalog-repo/modules/tensorboard/data.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/tensorboard/locals.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/tensorboard/main.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/tensorboard/outputs.tf delete mode 100644 5-app-infra/4-service-catalog-repo/modules/tensorboard/variables.tf delete mode 100644 5-app-infra/5-vpc-sc/README.md delete mode 100644 5-app-infra/6-machine-learning/README.md delete mode 120000 5-app-infra/6-machine-learning/business_unit_3/non-production/common.auto.tfvars delete mode 100644 5-app-infra/6-machine-learning/business_unit_3/non-production/versions.tf delete mode 120000 5-app-infra/6-machine-learning/business_unit_3/production/common.auto.tfvars delete mode 100644 5-app-infra/6-machine-learning/business_unit_3/production/versions.tf delete mode 100644 5-app-infra/7-machine-learning-post-deployment/README.md rename 5-app-infra/{1-artifact-publish => }/modules/publish_artifacts/data.tf (100%) rename 5-app-infra/{1-artifact-publish => }/modules/publish_artifacts/locals.tf (100%) rename 5-app-infra/{1-artifact-publish => }/modules/publish_artifacts/main.tf (100%) rename 5-app-infra/{1-artifact-publish => }/modules/publish_artifacts/outputs.tf (100%) rename 5-app-infra/{1-artifact-publish => }/modules/publish_artifacts/variables.tf (100%) rename 5-app-infra/{1-artifact-publish => }/modules/publish_artifacts/versions.tf (100%) rename 5-app-infra/{3-service-catalog/modules/svc_ctlg => modules/service_catalog}/data.tf (100%) rename 5-app-infra/{3-service-catalog/modules/svc_ctlg => modules/service_catalog}/locals.tf (100%) rename 5-app-infra/{3-service-catalog/modules/svc_ctlg => modules/service_catalog}/main.tf (100%) rename 5-app-infra/{3-service-catalog/modules/svc_ctlg => modules/service_catalog}/outputs.tf (100%) rename 5-app-infra/{3-service-catalog/modules/svc_ctlg => modules/service_catalog}/variables.tf (100%) delete mode 100644 5-app-infra/projects/artifact-publish/README.md delete mode 100644 5-app-infra/projects/artifact-publish/modules/publish_artifacts/data.tf delete mode 100644 5-app-infra/projects/artifact-publish/modules/publish_artifacts/locals.tf delete mode 100644 5-app-infra/projects/artifact-publish/modules/publish_artifacts/main.tf delete mode 100644 5-app-infra/projects/artifact-publish/modules/publish_artifacts/outputs.tf delete mode 100644 5-app-infra/projects/artifact-publish/modules/publish_artifacts/variables.tf delete mode 100644 5-app-infra/projects/artifact-publish/modules/publish_artifacts/versions.tf delete mode 100644 5-app-infra/projects/machine-learning/.gitignore delete mode 100644 5-app-infra/projects/machine-learning/README.md delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/README.md delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/backend.tf delete mode 120000 5-app-infra/projects/machine-learning/business_unit_3/development/common.auto.tfvars delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/locals.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/main.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/outputs.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/remote.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/variables.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/development/versions.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/README.md delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/backend.tf delete mode 120000 5-app-infra/projects/machine-learning/business_unit_3/non-production/common.auto.tfvars delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/locals.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/main.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/outputs.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/remote.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/variables.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/non-production/versions.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/README.md delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/backend.tf delete mode 120000 5-app-infra/projects/machine-learning/business_unit_3/production/common.auto.tfvars delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/locals.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/main.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/outputs.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/remote.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/variables.tf delete mode 100644 5-app-infra/projects/machine-learning/business_unit_3/production/versions.tf delete mode 100644 5-app-infra/projects/machine-learning/common.auto.example.tfvars delete mode 100644 5-app-infra/projects/machine-learning/modules/base_env/data.tf delete mode 100644 5-app-infra/projects/machine-learning/modules/base_env/iam.tf delete mode 100644 5-app-infra/projects/machine-learning/modules/base_env/main.tf delete mode 100644 5-app-infra/projects/machine-learning/modules/base_env/outputs.tf delete mode 100644 5-app-infra/projects/machine-learning/modules/base_env/roles.tf delete mode 100644 5-app-infra/projects/machine-learning/modules/base_env/variables.tf delete mode 120000 5-app-infra/projects/service-catalog/business_unit_3/shared/common.auto.tfvars delete mode 100644 5-app-infra/projects/service-catalog/modules/svc_ctlg/data.tf delete mode 100644 5-app-infra/projects/service-catalog/modules/svc_ctlg/locals.tf delete mode 100644 5-app-infra/projects/service-catalog/modules/svc_ctlg/main.tf delete mode 100644 5-app-infra/projects/service-catalog/modules/svc_ctlg/outputs.tf delete mode 100644 5-app-infra/projects/service-catalog/modules/svc_ctlg/variables.tf rename 5-app-infra/{2-artifact-publish-repo => source_repos/artifact-publish}/images/tf2-cpu.2-8:01/Dockerfile (100%) rename 5-app-infra/{2-artifact-publish-repo => source_repos/artifact-publish}/images/vertexpipeline:v2/Dockerfile (100%) delete mode 100644 6-ml-pipeline/dev/Readme.md rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/.gitignore (100%) create mode 100644 examples/machine-learning-pipeline/README.md rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/development/README.md (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/development/backend.tf (100%) rename {5-app-infra/1-artifact-publish/business_unit_3/shared => examples/machine-learning-pipeline/business_unit_3/development}/common.auto.tfvars (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/development/locals.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/development/main.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/development/outputs.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/development/remote.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/development/variables.tf (100%) rename {5-app-infra/1-artifact-publish/business_unit_3/shared => examples/machine-learning-pipeline/business_unit_3/development}/versions.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/non-production/README.md (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/non-production/backend.tf (100%) rename {5-app-infra/3-service-catalog/business_unit_3/shared => examples/machine-learning-pipeline/business_unit_3/non-production}/common.auto.tfvars (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/non-production/locals.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/non-production/main.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/non-production/outputs.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/non-production/remote.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/non-production/variables.tf (100%) rename {5-app-infra/3-service-catalog/business_unit_3/shared => examples/machine-learning-pipeline/business_unit_3/non-production}/versions.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/production/README.md (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/production/backend.tf (100%) rename {5-app-infra/6-machine-learning/business_unit_3/development => examples/machine-learning-pipeline/business_unit_3/production}/common.auto.tfvars (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/production/locals.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/production/main.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/production/outputs.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/production/remote.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/business_unit_3/production/variables.tf (100%) rename {5-app-infra/6-machine-learning/business_unit_3/development => examples/machine-learning-pipeline/business_unit_3/production}/versions.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/common.auto.example.tfvars (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/modules/base_env/data.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/modules/base_env/iam.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/modules/base_env/main.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/modules/base_env/outputs.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/modules/base_env/roles.tf (100%) rename {5-app-infra/6-machine-learning => examples/machine-learning-pipeline}/modules/base_env/variables.tf (100%) diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 0b89cae9..2d576017 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -48,10 +48,10 @@ jobs: env: DISABLE_TFLINT: 1 ENABLE_PARALLEL: 0 - EXCLUDE_LINT_DIRS: \./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning|\./docs/assets/terraform + EXCLUDE_LINT_DIRS: \./examples/machine-learning-pipeline|\./docs/assets/terraform - run: docker run --rm -e DISABLE_TFLINT -e ENABLE_PARALLEL -e EXCLUDE_LINT_DIRS -v ${{ github.workspace }}:/workspace ${{ steps.variables.outputs.dev-tools }} /usr/local/bin/test_lint.sh env: DISABLE_TFLINT: 1 ENABLE_PARALLEL: 0 - EXCLUDE_LINT_DIRS: \./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning|\./docs/assets/terraform + EXCLUDE_LINT_DIRS: \./examples/machine-learning-pipeline|\./docs/assets/terraform diff --git a/5-app-infra/0-gcp-policies/README.md b/5-app-infra/0-gcp-policies/README.md deleted file mode 100644 index 8e75a01e..00000000 --- a/5-app-infra/0-gcp-policies/README.md +++ /dev/null @@ -1,90 +0,0 @@ -# 5-app-infra - -This repo is part of a multi-part guide that shows how to configure and deploy -the example.com reference architecture described in -[Google Cloud security foundations guide](https://cloud.google.com/architecture/security-foundations). The following table lists the parts of the guide. - -
0-bootstrap | -Bootstraps a Google Cloud organization, creating all the required resources -and permissions to start using the Cloud Foundation Toolkit (CFT). This -step also configures a CI/CD Pipeline for foundations code in subsequent -stages. | -
1-org | -Sets up top-level shared folders, monitoring and networking projects, -organization-level logging, and baseline security settings through -organizational policies. | -
2-environments | -Sets up development, non-production, and production environments within the -Google Cloud organization that you've created. | -
3-networks-dual-svpc | -Sets up base and restricted shared VPCs with default DNS, NAT (optional), -Private Service networking, VPC service controls, on-premises Dedicated -Interconnect, and baseline firewall rules for each environment. It also sets -up the global DNS hub. | -
3-networks-hub-and-spoke | -Sets up base and restricted shared VPCs with all the default configuration -found on step 3-networks-dual-svpc, but here the architecture will be based on the -Hub and Spoke network model. It also sets up the global DNS hub | -
4-projects | -Sets up a folder structure, projects, and an application infrastructure pipeline for applications, - which are connected as service projects to the shared VPC created in the previous stage. | -
5-app-infra - 01-gcp-polcies(this file) | -Configures GCP Policies repository for Cloud Build | -
0-bootstrap | -Bootstraps a Google Cloud organization, creating all the required resources -and permissions to start using the Cloud Foundation Toolkit (CFT). This -step also configures a CI/CD Pipeline for foundations code in subsequent -stages. | -
1-org | -Sets up top-level shared folders, monitoring and networking projects, -organization-level logging, and baseline security settings through -organizational policies. | -
2-environments | -Sets up development, non-production, and production environments within the -Google Cloud organization that you've created. | -
3-networks-dual-svpc | -Sets up base and restricted shared VPCs with default DNS, NAT (optional), -Private Service networking, VPC service controls, on-premises Dedicated -Interconnect, and baseline firewall rules for each environment. It also sets -up the global DNS hub. | -
3-networks-hub-and-spoke | -Sets up base and restricted shared VPCs with all the default configuration -found on step 3-networks-dual-svpc, but here the architecture will be based on the -Hub and Spoke network model. It also sets up the global DNS hub | -
4-projects | -Sets up a folder structure, projects, and an application infrastructure pipeline for applications, - which are connected as service projects to the shared VPC created in the previous stage. | -
5-app-infra 01-artifact-publish(this file) | -Deploys Artifact Repository | -
0-bootstrap | -Bootstraps a Google Cloud organization, creating all the required resources -and permissions to start using the Cloud Foundation Toolkit (CFT). This -step also configures a CI/CD Pipeline for foundations code in subsequent -stages. | -
1-org | -Sets up top-level shared folders, monitoring and networking projects, -organization-level logging, and baseline security settings through -organizational policies. | -
2-environments | -Sets up development, non-production, and production environments within the -Google Cloud organization that you've created. | -
3-networks-dual-svpc | -Sets up base and restricted shared VPCs with default DNS, NAT (optional), -Private Service networking, VPC service controls, on-premises Dedicated -Interconnect, and baseline firewall rules for each environment. It also sets -up the global DNS hub. | -
3-networks-hub-and-spoke | -Sets up base and restricted shared VPCs with all the default configuration -found on step 3-networks-dual-svpc, but here the architecture will be based on the -Hub and Spoke network model. It also sets up the global DNS hub | -
4-projects | -Sets up a folder structure, projects, and an application infrastructure pipeline for applications, - which are connected as service projects to the shared VPC created in the previous stage. | -
5-app-infra 02-artifact-publish-repo(this file) | -Configures a cloud build repository for Docker builds | -
0-bootstrap | -Bootstraps a Google Cloud organization, creating all the required resources -and permissions to start using the Cloud Foundation Toolkit (CFT). This -step also configures a CI/CD Pipeline for foundations code in subsequent -stages. | -
1-org | -Sets up top-level shared folders, monitoring and networking projects, -organization-level logging, and baseline security settings through -organizational policies. | -
2-environments | -Sets up development, non-production, and production environments within the -Google Cloud organization that you've created. | -
3-networks-dual-svpc | -Sets up base and restricted shared VPCs with default DNS, NAT (optional), -Private Service networking, VPC service controls, on-premises Dedicated -Interconnect, and baseline firewall rules for each environment. It also sets -up the global DNS hub. | -
3-networks-hub-and-spoke | -Sets up base and restricted shared VPCs with all the default configuration -found on step 3-networks-dual-svpc, but here the architecture will be based on the -Hub and Spoke network model. It also sets up the global DNS hub | -
4-projects | -Sets up a folder structure, projects, and an application infrastructure pipeline for applications, - which are connected as service projects to the shared VPC created in the previous stage. | -
5-app-infra - 3-service-catalog (this file) | -Deploys Composer and a pipeline | -
artifact_registry | -Artifact Registry is the next generation of Container Registry. Store, manage, and secure your build artifacts. | -
bigquery | -BigQuery is a serverless and cost-effective enterprise data warehouse that works across clouds and scales with your data. Use built-in ML/AI and BI for insights at scale. | -
bucket | -Cloud Storage is a managed service for storing unstructured data. | -
composer | -Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. | -
metadata | -Vertex ML Metadata is a service that provides capabilities for managing the lifecycle of metadata consumed and produced by machine-learning (ML) workflows. | -
notebook | -Vertex AI Workbench is a Jupyter notebook-based development environment for the entire data science workflow. | -
pubsub | -Pub/Sub is an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages. | -
secrets | -Secret Manager lets you store, manage, and access secrets as binary blobs or text strings. With the appropriate permissions, you can view the contents of the secret. | -
tensorboard | -Vertex AI TensorBoard is an enterprise-ready managed service for machine learning experiment visualization. | -