From b628a35b88a4a5adaa759b1612abd12d10a91e91 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Thu, 24 Oct 2019 00:01:11 -0500 Subject: [PATCH 01/16] Testing Templates - Init Intro, Goals and Prereqs after discussion --- docs/4_TEMPLATE_TESTING.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 3336017e..1e5043bb 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -1,13 +1,29 @@ # 4. Template Testing ## 4.1 Overview -< Describe the general context around this walkthrough/lab > + +Prior to this point in the walkthrough, you deployed infrastructure by running a CIT you built yourself with guidance from the [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough. You also deployed infrastructure using the [*Azure Hello World CIT*](../infra/templates/az-hello-world/README.md "AZ Hello World - Cobalt Infrastructure Template") from the [Quickstart Guide](./2_QUICK_START_GUIDE.md). In other words, that's two separate experiences with the Cobalt Developer Workflow (i.e. create/choose a template ---> init ---> select workspace ---> plan ---> apply ---> destroy) and so this workflow is no longer a foreign concept. + +With your new level of comfort, let's build on what you know about the Cobalt Developer Workflow and introduce the testing phases. Simply put, the Cobalt Developer Workflow described above is not considered complete without testing phases. + +> *Have yet to create a Cobalt Infrastructure Template or CIT (/kɪt/) of your own? Design and create your first infrastructure template with Cobalt by completing our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough.* ## 4.2 Goals and Objectives -< Outline the goals and takeaways of this walkthrough/lab > + +🔲 Understand how testing fits into the SDLC for writing *CIT*s + +🔲 Avoid regressions with robust unit and integration tests + +🔲 Feel confident in moving forward to our next recommended section: *[Operationalizing CITs - From A to Z.. err CICD](./5_OPERATIONALIZE_TEMPLATE.md).* ## 4.3 Prerequisites < Enumerate any prerequisite walkthroughs that should be completed prior to this, and any technical prerequisites such as developer environment / tools > +| Prereqs | Description | +|----------|--------------| +| [Quickstart Guide](./2_QUICK_START_GUIDE.md) | The quickstart guide provides all of the prerequisites you'll need to create your own *CIT* and run it.| +| [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | The Cobalt Templating From Scratch walkthrough provides all of the prerequisites you'll need to create your own *CIT* and run it.| +| [Terraform Modules](https://www.terraform.io/docs/configuration/modules.html) | An introductory understanding of Terraform modules.| + ## 4.4 Walkthrough < Step-by-step instructions for completing this walkthrough > \ No newline at end of file From 7f8888a354d11ccbfe004e0088ce38256b841325 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Thu, 24 Oct 2019 13:45:11 -0500 Subject: [PATCH 02/16] testing templates - reword intro, expand prereqs --- docs/4_TEMPLATE_TESTING.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 1e5043bb..93c16cfb 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -2,9 +2,7 @@ ## 4.1 Overview -Prior to this point in the walkthrough, you deployed infrastructure by running a CIT you built yourself with guidance from the [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough. You also deployed infrastructure using the [*Azure Hello World CIT*](../infra/templates/az-hello-world/README.md "AZ Hello World - Cobalt Infrastructure Template") from the [Quickstart Guide](./2_QUICK_START_GUIDE.md). In other words, that's two separate experiences with the Cobalt Developer Workflow (i.e. create/choose a template ---> init ---> select workspace ---> plan ---> apply ---> destroy) and so this workflow is no longer a foreign concept. - -With your new level of comfort, let's build on what you know about the Cobalt Developer Workflow and introduce the testing phases. Simply put, the Cobalt Developer Workflow described above is not considered complete without testing phases. +Because we haven't introduced the concept of what it means to test a Cobalt Infrastructure Template, on two separate occasions we set you up to get away with deploying infrastructure without any mention of creating or running tests. You deployed infrastructure by running a CIT you built yourself with guidance from the [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough. You also deployed infrastructure using the [*Azure Hello World CIT*](../infra/templates/az-hello-world/README.md "AZ Hello World - Cobalt Infrastructure Template") from the [Quickstart Guide](./2_QUICK_START_GUIDE.md). Simply put, you've had two experiences with the Cobalt Developer Workflow (i.e. create/choose a template ---> init ---> select workspace ---> plan ---> apply ---> destroy) that did not include testing. Let's further build on what you know so far by surfacing the approach Cobalt has taken to infrastructure testing and how that will impact the Cobalt Developer Workflow. > *Have yet to create a Cobalt Infrastructure Template or CIT (/kɪt/) of your own? Design and create your first infrastructure template with Cobalt by completing our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough.* @@ -14,16 +12,19 @@ With your new level of comfort, let's build on what you know about the Cobalt De 🔲 Avoid regressions with robust unit and integration tests -🔲 Feel confident in moving forward to our next recommended section: *[Operationalizing CITs - From A to Z.. err CICD](./5_OPERATIONALIZE_TEMPLATE.md).* +🔲 Feel confident in moving forward to our next recommended section: *[Operationalizing CITs - CICD Templates](./5_OPERATIONALIZE_TEMPLATE.md).* ## 4.3 Prerequisites -< Enumerate any prerequisite walkthroughs that should be completed prior to this, and any technical prerequisites such as developer environment / tools > | Prereqs | Description | |----------|--------------| -| [Quickstart Guide](./2_QUICK_START_GUIDE.md) | The quickstart guide provides all of the prerequisites you'll need to create your own *CIT* and run it.| -| [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | The Cobalt Templating From Scratch walkthrough provides all of the prerequisites you'll need to create your own *CIT* and run it.| +| [Quickstart Guide](./2_QUICK_START_GUIDE.md) | This should have served as your first Cobalt Infrastructure deployment. | +| [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | Completing this prequisite leaves you with a CIT and prior knowledge needed for this walkthrough. | | [Terraform Modules](https://www.terraform.io/docs/configuration/modules.html) | An introductory understanding of Terraform modules.| +| [Go](Golang) | An introductory understanding of Go. | +| [Go Test](Golang) | An introductory understanding of Go Tests | +| [TerraTest](Gruntworks) | An introductory understanding of TerraTest | + +## 4.4 Walkthrough - Test a Cobalt Infrastructure Template -## 4.4 Walkthrough < Step-by-step instructions for completing this walkthrough > \ No newline at end of file From 561f4ca0d6fc6050f613b8f6ea47414b17d5f576 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Thu, 24 Oct 2019 16:28:45 -0500 Subject: [PATCH 03/16] testing a template - reword intro --- docs/4_TEMPLATE_TESTING.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 93c16cfb..b468f3e6 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -2,7 +2,9 @@ ## 4.1 Overview -Because we haven't introduced the concept of what it means to test a Cobalt Infrastructure Template, on two separate occasions we set you up to get away with deploying infrastructure without any mention of creating or running tests. You deployed infrastructure by running a CIT you built yourself with guidance from the [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough. You also deployed infrastructure using the [*Azure Hello World CIT*](../infra/templates/az-hello-world/README.md "AZ Hello World - Cobalt Infrastructure Template") from the [Quickstart Guide](./2_QUICK_START_GUIDE.md). Simply put, you've had two experiences with the Cobalt Developer Workflow (i.e. create/choose a template ---> init ---> select workspace ---> plan ---> apply ---> destroy) that did not include testing. Let's further build on what you know so far by surfacing the approach Cobalt has taken to infrastructure testing and how that will impact the Cobalt Developer Workflow. +Previously, to keep things simple, our walkthroughs set you up to deploy infrastructure without creating or running tests. Using the guidance from our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough, you deployed infrastructure by running a CIT you built yourself. Using the [Quickstart Guide](./2_QUICK_START_GUIDE.md) you deployed infrastructure by running our [*Azure Hello World CIT*](../infra/templates/az-hello-world/README.md "AZ Hello World - Cobalt Infrastructure Template"). + +Simply put, you've had two experiences with the Cobalt Developer Workflow (i.e. create/choose a template ---> init ---> select workspace ---> plan ---> apply ---> destroy) that did not include testing. However, we strongly encourage testing as a part of your dev worklflow. The reason is ... > *Have yet to create a Cobalt Infrastructure Template or CIT (/kɪt/) of your own? Design and create your first infrastructure template with Cobalt by completing our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough.* @@ -12,7 +14,7 @@ Because we haven't introduced the concept of what it means to test a Cobalt Infr 🔲 Avoid regressions with robust unit and integration tests -🔲 Feel confident in moving forward to our next recommended section: *[Operationalizing CITs - CICD Templates](./5_OPERATIONALIZE_TEMPLATE.md).* +🔲 Feel confident in moving forward to our next recommended section: *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* ## 4.3 Prerequisites @@ -24,7 +26,8 @@ Because we haven't introduced the concept of what it means to test a Cobalt Infr | [Go](Golang) | An introductory understanding of Go. | | [Go Test](Golang) | An introductory understanding of Go Tests | | [TerraTest](Gruntworks) | An introductory understanding of TerraTest | +| [Docker](Docker.io) | An introductory understanding of Docker | -## 4.4 Walkthrough - Test a Cobalt Infrastructure Template +## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template -< Step-by-step instructions for completing this walkthrough > \ No newline at end of file +### **Step 1:** Use Test Harness Feature From b4dacaa797b07cf708ecac691a0690665754b948 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Fri, 25 Oct 2019 11:28:41 -0500 Subject: [PATCH 04/16] testing a template - reword intro --- docs/4_TEMPLATE_TESTING.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index b468f3e6..7a85f495 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -2,9 +2,7 @@ ## 4.1 Overview -Previously, to keep things simple, our walkthroughs set you up to deploy infrastructure without creating or running tests. Using the guidance from our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough, you deployed infrastructure by running a CIT you built yourself. Using the [Quickstart Guide](./2_QUICK_START_GUIDE.md) you deployed infrastructure by running our [*Azure Hello World CIT*](../infra/templates/az-hello-world/README.md "AZ Hello World - Cobalt Infrastructure Template"). - -Simply put, you've had two experiences with the Cobalt Developer Workflow (i.e. create/choose a template ---> init ---> select workspace ---> plan ---> apply ---> destroy) that did not include testing. However, we strongly encourage testing as a part of your dev worklflow. The reason is ... +As software developers, we seek out opportunities to grow and improve the codebase. Growth and improvement, whether major or gradual, requires change and all changes bring with them a chance for unexpected behavior to occur. Automated testing is chance for us to gain more predictability and control over the outcomes of those changes. Without automated tests, our codebase will simply be too unpredictable to be considered scalable. With that being said, the task of building automated test can vary from project to project. For this project, let's further build on what you've learned so far by introducing the recommended approach Cobalt has taken to infrastructure testing and how that will impact the Cobalt Developer Workflow: > *Have yet to create a Cobalt Infrastructure Template or CIT (/kɪt/) of your own? Design and create your first infrastructure template with Cobalt by completing our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough.* @@ -30,4 +28,6 @@ Simply put, you've had two experiences with the Cobalt Developer Workflow (i.e. ## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template -### **Step 1:** Use Test Harness Feature +### **Step 1:** Code to the Terraform Harness Hooks - Unit Testing + +### **Step 2:** Code to the Terraform Harness Hooks - Integration Testing From c901855579aae3f5601b5d32fc632985546b2613 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Tue, 29 Oct 2019 13:30:48 -0500 Subject: [PATCH 05/16] testing a template - reword walkthrough steps --- docs/4_TEMPLATE_TESTING.md | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 7a85f495..a0702cdd 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -2,7 +2,9 @@ ## 4.1 Overview -As software developers, we seek out opportunities to grow and improve the codebase. Growth and improvement, whether major or gradual, requires change and all changes bring with them a chance for unexpected behavior to occur. Automated testing is chance for us to gain more predictability and control over the outcomes of those changes. Without automated tests, our codebase will simply be too unpredictable to be considered scalable. With that being said, the task of building automated test can vary from project to project. For this project, let's further build on what you've learned so far by introducing the recommended approach Cobalt has taken to infrastructure testing and how that will impact the Cobalt Developer Workflow: +As software developers, we seek out opportunities to improve and grow projects via code contributions. Code contributions, whether major or gradual, require changes that unfortunately bring with them a chance for unexpected behavior to occur. Fortunately, automated testing grants us a chance to gain more predictability and control over the changes that occur from our contributions. Without automated testing, contributions to our codebase would simply be too unpredictable to be considered scalable. + +With that being said, the task of building automated tests can vary from software project to project. For this project, let's further build on what you've learned so far by introducing the approach Cobalt has taken to infrastructure testing and how that will impact what you know about the Cobalt Developer Workflow: > *Have yet to create a Cobalt Infrastructure Template or CIT (/kɪt/) of your own? Design and create your first infrastructure template with Cobalt by completing our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough.* @@ -22,12 +24,24 @@ As software developers, we seek out opportunities to grow and improve the codeba | [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | Completing this prequisite leaves you with a CIT and prior knowledge needed for this walkthrough. | | [Terraform Modules](https://www.terraform.io/docs/configuration/modules.html) | An introductory understanding of Terraform modules.| | [Go](Golang) | An introductory understanding of Go. | -| [Go Test](Golang) | An introductory understanding of Go Tests | -| [TerraTest](Gruntworks) | An introductory understanding of TerraTest | -| [Docker](Docker.io) | An introductory understanding of Docker | +| [Go Test](Golang) | An introductory understanding of Go Tests. | +| [TerraTest](Gruntworks) | An introductory understanding of TerraTest. | +| [Docker](Docker.io) | An introductory understanding of Docker. | + +## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template (CIT) + +CIT's are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. Therefore, they are made up of the same properties that make up modules. The following steps provide guidance on using your knowledge of these Terraform properties to properly write tests in Terraform. + +> **NOTE:** This walkthrough only focuses on running automated tests locally. Our next section *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various stages. + +### **Step 1:** Choose which properties of the CIT you care about testing + +Testing a Cobalt Infrastructure Template begins with the exercise of walking through the commonly shared properties that all CITs are made of (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) and carefully thinking about their impact on that CIT's Terraform state along with it's deployed infrastructure during the plan, apply and destroy phases of the Cobalt Developer Workflow. This exercise will assist you in developing assertions that you care about testing in relation to your CIT. + +1. Choose Input Variables For Testing - *More on Input Variables:* https://www.terraform.io/docs/configuration/variables.html -## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template + Decide on the test values to use when running tests. The values you provide should be non-production values so as to not affect end-user environments. Here are a few examples of test values we used as input for our az-hello-world CIT: -### **Step 1:** Code to the Terraform Harness Hooks - Unit Testing +2. Choosing Output Variables For Testing - *More on Input Variables:* https://www.terraform.io/docs/configuration/outputs.html -### **Step 2:** Code to the Terraform Harness Hooks - Integration Testing + Outputs are return values for modules, therefore, your CIT also has return values. Outputs are great candidates for testing because they explicitly are visibile within your Terraform Plan. Here are a few examples of the assertions we decided to test for our az-hello-world CIT: From 283c3b01659d9237f66930c7cd4dfdf3bfc38f69 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Wed, 30 Oct 2019 12:16:55 -0500 Subject: [PATCH 06/16] testing a template - reword most steps --- docs/4_TEMPLATE_TESTING.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index a0702cdd..cbb52cff 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -4,7 +4,7 @@ As software developers, we seek out opportunities to improve and grow projects via code contributions. Code contributions, whether major or gradual, require changes that unfortunately bring with them a chance for unexpected behavior to occur. Fortunately, automated testing grants us a chance to gain more predictability and control over the changes that occur from our contributions. Without automated testing, contributions to our codebase would simply be too unpredictable to be considered scalable. -With that being said, the task of building automated tests can vary from software project to project. For this project, let's further build on what you've learned so far by introducing the approach Cobalt has taken to infrastructure testing and how that will impact what you know about the Cobalt Developer Workflow: +With that being said, the task of building automated tests for infrastructure can feel a bit different then the kinds of tests you are used to. For this project, let's further build on what you've learned so far by introducing the approach Cobalt has taken to infrastructure testing and how that will impact what you know about the Cobalt Developer Workflow: > *Have yet to create a Cobalt Infrastructure Template or CIT (/kɪt/) of your own? Design and create your first infrastructure template with Cobalt by completing our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough.* @@ -30,18 +30,22 @@ With that being said, the task of building automated tests can vary from softwar ## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template (CIT) -CIT's are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. Therefore, they are made up of the same properties that make up modules. The following steps provide guidance on using your knowledge of these Terraform properties to properly write tests in Terraform. +If you are used to writing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. In order to write tests in Terraform, you'll be automating this workflow. In order to make automation useful for testing, you'll have to know where in the various phases of your developer workflow would allow you to make sure your tests are isolated. You'll also have to know where it is that you can start making assertions about your CIT in an automated way. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write tests for your CIT in Terraform. -> **NOTE:** This walkthrough only focuses on running automated tests locally. Our next section *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various stages. +> **NOTE:** This walkthrough only focuses on running automated tests locally. Our next walkthrough *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various stages. -### **Step 1:** Choose which properties of the CIT you care about testing +### **Step 1:** Prepare for Test Isolation -Testing a Cobalt Infrastructure Template begins with the exercise of walking through the commonly shared properties that all CITs are made of (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) and carefully thinking about their impact on that CIT's Terraform state along with it's deployed infrastructure during the plan, apply and destroy phases of the Cobalt Developer Workflow. This exercise will assist you in developing assertions that you care about testing in relation to your CIT. +You'll want to make sure that your tests are using actual test values. The values you provide should be non-production values so as to not affect end-user environments. In the Cobalt Develop Workflow, your first opportunities to enforce isolation begins with setting up your local environment variables, running `terraform init` and then running `terraform workspace new ` from your CIT's directory. You'll be automating this, so make sure you have those values ready. -1. Choose Input Variables For Testing - *More on Input Variables:* https://www.terraform.io/docs/configuration/variables.html +1. Ensure that your .env file is using non-production values and take note of them. - Decide on the test values to use when running tests. The values you provide should be non-production values so as to not affect end-user environments. Here are a few examples of test values we used as input for our az-hello-world CIT: +2. Ensure that the workspaceName that gets created on each dev workflow cycle is unique enough to allow for tests to ru in parallel. It's very possible that you are on a team with dev who have to share cloud provider resources and account. The workspace name may be your solution to working in isolation from other devs. -2. Choosing Output Variables For Testing - *More on Input Variables:* https://www.terraform.io/docs/configuration/outputs.html +3. Local the input variables in your CIT that further help you achieve test isolation. Here are a few examples of required input values we used to help enforce isolation for our az-hello-world CIT. - *More on Input Variables:* https://www.terraform.io/docs/configuration/variables.html - Outputs are return values for modules, therefore, your CIT also has return values. Outputs are great candidates for testing because they explicitly are visibile within your Terraform Plan. Here are a few examples of the assertions we decided to test for our az-hello-world CIT: + | Input Variable Name | Value | Description | + |--------|----------|-----------| + | `workspace` | "az-hello-world-" + guid | It's important that a random guid is attached to the workspace of your tests as that enables unique workspaces ensuring that tests can be run in parallel. | + | `prefix` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests as those kinds of tests create actual infrastructure.| + | `storage_account_name` | os.Getenv("TF_VAR_remote_state_account") | It's important that these tests run in isolation and do not impact production environments, therefore, this value should be the storage account dedicated to the dev environment. | From 37827ef83b0ea1d5e36329f38f2df3f8ac204a8b Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Wed, 30 Oct 2019 14:15:05 -0500 Subject: [PATCH 07/16] testing a template - expand in isolation step --- docs/4_TEMPLATE_TESTING.md | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index cbb52cff..ee5a4692 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -30,22 +30,34 @@ With that being said, the task of building automated tests for infrastructure ca ## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template (CIT) -If you are used to writing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. In order to write tests in Terraform, you'll be automating this workflow. In order to make automation useful for testing, you'll have to know where in the various phases of your developer workflow would allow you to make sure your tests are isolated. You'll also have to know where it is that you can start making assertions about your CIT in an automated way. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write tests for your CIT in Terraform. +If you are used to writing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. In order to write tests in Terraform, you'll be automating this workflow. + +To make automation useful for testing, you'll have to know where in the various phases of your developer workflow allow you to make sure your tests are isolated. You'll also have to know where it is that you can start making assertions about your CIT in an automated way. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write automated tests for your CIT in Terraform. > **NOTE:** This walkthrough only focuses on running automated tests locally. Our next walkthrough *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various stages. ### **Step 1:** Prepare for Test Isolation -You'll want to make sure that your tests are using actual test values. The values you provide should be non-production values so as to not affect end-user environments. In the Cobalt Develop Workflow, your first opportunities to enforce isolation begins with setting up your local environment variables, running `terraform init` and then running `terraform workspace new ` from your CIT's directory. You'll be automating this, so make sure you have those values ready. +You'll want to make sure that your tests are using actual test values before running `terraform plan`. The values you provide should be non-production values so as to not affect end-user environments. In the Cobalt Develop Workflow, the opportunities to enforce isolation first begin with setting up your local environment variables, and then running `terraform init` and `terraform workspace new ` from your CIT's directory. You'll be automating this workflow, so make sure you have the following values ready. + +> **NOTE:** Due to the beginning of the Cobalt Developer Workflow having a hard dependency on a back-end state file, both unit testing and integration testing will need these environment variables. Therefore, we recommend making these decisions early-on. + +1. Prepare your environment variables. Ensure that your .env file is using non-production values as you will be referencing these when writing your automated tests. These values are usually specific to the provider your CIT is targeting. Here are a few examples: -1. Ensure that your .env file is using non-production values and take note of them. + | Env Variables | Test Value | Description | + |--------|----------|-----------| + | `storage_account_name` | Value from "TF_VAR_remote_state_account" | This value should be the storage account dedicated to the dev environment. The value lives within your .env file | + | `container_name` | Value from "TF_VAR_remote_state_container" | This value will be the dedicated container that will holder multiple backend remote workspace terraform state files. The value lives within your .env file . | -2. Ensure that the workspaceName that gets created on each dev workflow cycle is unique enough to allow for tests to ru in parallel. It's very possible that you are on a team with dev who have to share cloud provider resources and account. The workspace name may be your solution to working in isolation from other devs. +2. Locate the input variables from your CIT that further help you achieve test isolation. Here are all the input variables and values we used to help further prepare the az-hello-world CIT for test isolation. - *More on Input Variables:* [Terraform Input Variables](https://www.terraform.io/docs/configuration/variables.html) -3. Local the input variables in your CIT that further help you achieve test isolation. Here are a few examples of required input values we used to help enforce isolation for our az-hello-world CIT. - *More on Input Variables:* https://www.terraform.io/docs/configuration/variables.html + > **NOTE:** The az-hello-world CIT relies on a commons.tf file that enforces uniqueness of resource names for varying reasons, therefore, the names you choose below will be further transformed based on the implementation of the commons.tf file. Some of the reasons for the name transformations include work isolation and other reasons are due to resources that require unique names like fqdns. - | Input Variable Name | Value | Description | + | Input Var Name | Test Value | Description | |--------|----------|-----------| - | `workspace` | "az-hello-world-" + guid | It's important that a random guid is attached to the workspace of your tests as that enables unique workspaces ensuring that tests can be run in parallel. | - | `prefix` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests as those kinds of tests create actual infrastructure.| - | `storage_account_name` | os.Getenv("TF_VAR_remote_state_account") | It's important that these tests run in isolation and do not impact production environments, therefore, this value should be the storage account dedicated to the dev environment. | + | `workspace` | "az-hello-world-" + guid | It's important that a random guid is attached to the workspace of your tests. Teams share cloud provider resources and accounts. The workspace name may be your only solution to working in isolation from other devs on your team. | + | `name` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests as those kinds of tests create actual infrastructure.| + | `resource_group_location` | "eastus" | The geo-location of the Azure datacenter in which you want the resource group that contains your infrastructure to live. A location different then production. | + +### **Step 2:** Decide on which properties of your Terraform plan are testable (Unit Tests) + From 828791b8497198ead72fc8d06fa52b61fc76a097 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Thu, 31 Oct 2019 12:47:11 -0500 Subject: [PATCH 08/16] testing a template - expanding instructions on unit testing --- docs/4_TEMPLATE_TESTING.md | 54 +++++++++++++++++++++++++++++++++----- 1 file changed, 47 insertions(+), 7 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index ee5a4692..760035fe 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -30,17 +30,17 @@ With that being said, the task of building automated tests for infrastructure ca ## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template (CIT) -If you are used to writing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. In order to write tests in Terraform, you'll be automating this workflow. +If you are used to developing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. Executing this workflow so far has required you to interact with Terraform via it's cli. However, you might not have known that the Terraform cli runs commands against [Terraform Core](https://www.terraform.io/docs/extend/how-terraform-works.html#terraform-core), a statically compiled Golang binary. Therefore, in order to programmatically interact with Terraform to achieve automation, you'll have to write tests in Golang. -To make automation useful for testing, you'll have to know where in the various phases of your developer workflow allow you to make sure your tests are isolated. You'll also have to know where it is that you can start making assertions about your CIT in an automated way. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write automated tests for your CIT in Terraform. +Furthermore, in order to make automation useful for testing, you'll have to know where in the various phases of your developer workflow enable test isolation. You'll also have to know where it is that you can start making assertions about your CIT's deployment plans and the infrastructure that was stood up in an automated way. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write automated test in Golang for your CIT. > **NOTE:** This walkthrough only focuses on running automated tests locally. Our next walkthrough *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various stages. ### **Step 1:** Prepare for Test Isolation -You'll want to make sure that your tests are using actual test values before running `terraform plan`. The values you provide should be non-production values so as to not affect end-user environments. In the Cobalt Develop Workflow, the opportunities to enforce isolation first begin with setting up your local environment variables, and then running `terraform init` and `terraform workspace new ` from your CIT's directory. You'll be automating this workflow, so make sure you have the following values ready. +You'll want to make sure that your tests are using actual test values before programmatically running `terraform plan`. The values you provide should be non-production values so as to not affect end-user environments. In the Cobalt Develop Workflow, the opportunities to enforce test isolation first begin with setting up your local environment variables, and then running `terraform init` and `terraform workspace new ` from your CIT's directory. -> **NOTE:** Due to the beginning of the Cobalt Developer Workflow having a hard dependency on a back-end state file, both unit testing and integration testing will need these environment variables. Therefore, we recommend making these decisions early-on. +> **NOTE:** Due to the beginning of the Cobalt Developer Workflow having a hard dependency on a back-end state file, both unit testing and integration testing will need these environment variables. Therefore, we recommend making these decisions early-on as a first step. 1. Prepare your environment variables. Ensure that your .env file is using non-production values as you will be referencing these when writing your automated tests. These values are usually specific to the provider your CIT is targeting. Here are a few examples: @@ -49,7 +49,7 @@ You'll want to make sure that your tests are using actual test values before run | `storage_account_name` | Value from "TF_VAR_remote_state_account" | This value should be the storage account dedicated to the dev environment. The value lives within your .env file | | `container_name` | Value from "TF_VAR_remote_state_container" | This value will be the dedicated container that will holder multiple backend remote workspace terraform state files. The value lives within your .env file . | -2. Locate the input variables from your CIT that further help you achieve test isolation. Here are all the input variables and values we used to help further prepare the az-hello-world CIT for test isolation. - *More on Input Variables:* [Terraform Input Variables](https://www.terraform.io/docs/configuration/variables.html) +2. Locate the input variables from your CIT that further help you achieve test isolation and choose values for them. Here are all the input variables and values we used to help further prepare the az-hello-world CIT for test isolation: > **NOTE:** The az-hello-world CIT relies on a commons.tf file that enforces uniqueness of resource names for varying reasons, therefore, the names you choose below will be further transformed based on the implementation of the commons.tf file. Some of the reasons for the name transformations include work isolation and other reasons are due to resources that require unique names like fqdns. @@ -57,7 +57,47 @@ You'll want to make sure that your tests are using actual test values before run |--------|----------|-----------| | `workspace` | "az-hello-world-" + guid | It's important that a random guid is attached to the workspace of your tests. Teams share cloud provider resources and accounts. The workspace name may be your only solution to working in isolation from other devs on your team. | | `name` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests as those kinds of tests create actual infrastructure.| - | `resource_group_location` | "eastus" | The geo-location of the Azure datacenter in which you want the resource group that contains your infrastructure to live. A location different then production. | + | `resource_group_location` | "eastus" | The geo-location of the Azure datacenter in which you want the resource group that contains your infrastructure to live. A geo-location maybe different then production. | -### **Step 2:** Decide on which properties of your Terraform plan are testable (Unit Tests) +### **Step 2:** Develop Your Terraform Plan Based Assertions (Unit Testing) +The previous step properly prepared you for test isolation. Now you can concern yourself with the business of developing assertions. The best way to make assertions about your CIT is to rely on what's visible in the Terraform Plan. Terraform Plans are what's presented to you in standard out when running `terraform plan`. This is the earliest phase in the dev workflow that you can rely on for your test assertions. + +1. Learn to read the Terraform Plan + + You'll want to learn how to read a Terraform Plan. This will be the most efficient piece of information to use in discovering most of the infrastructure that your CIT plans to deploy before actual deployment. Terraform uses all module properties (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) present in your CIT in order to build a [Terraform Dependency Graph](https://www.terraform.io/docs/internals/graph.html) at plan time. This dependency graph is then represented as a list of resource address nodes that make up the Terraform plan visible in standard out. The state of that plan is what gets executed when running `terraform apply`. + +2. Plan assertions to make about your CIT + + The resource addresses in your Terraform Plan have varying levels of nested information. Some of the values located at each resource address can be seen at plan time and some are so dynamic that they are not discoverable until the `terraform apply` command has finished executing. A good rule of thumb is values that are visible at plan time are unit testable, values not visible require integration testing. + + > **TIP:** Test as much as you can here because integration tests require spinning up and tearing down actual infrastructure beyond a state file. + + * Run `terraform plan` and look over the Terraform Plan that is presented so that you can decide on which assertions you'd like to make. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. + + Here are a few examples of assertions we decided to make for tests within our az-hello-world CIT: + + | Terraform Plan Resource Address | Module Property Type | Planned Assertion | + |--------|-----------|-----------|-----------| + | `module.app_service.azurerm_app_service.appsvc[0]` | module | Assert the presence of docker image in the configuration of the app service module. | + | `module.service_plan.azurerm_app_service_plan.svcplan` | module | Assert that the service plan module for the az-hello-world CIT is configured for the least expensive S1 tier. | + | `azurerm_resource_group.main` | resource | Assert the resource group contains the datacenter used for test environments. | + +### **Step 3:** Decide on which properties of your Terraform State are testable (Integration Tests) + + +### **Step 4:** Decide on a Go Testing Framework + +### **Step 5:** Decide on a Terraform Testing Framework + +Terratest vs kitchen-terraform. + +### **Step 6:** Write Unit Tests + +> **NOTE**: This example relies on the testing harness. + +### **Step 7:** Run Unit Tests + +### **Step 8:** Write Integration Tests + +### **Step 9:** Run Integration Tests From ce793aa0bf2b52eae31a125b640b03fad85230a7 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Thu, 31 Oct 2019 16:10:12 -0500 Subject: [PATCH 09/16] testing a template - create instructions on integration testing --- docs/4_TEMPLATE_TESTING.md | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 760035fe..6db32255 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -61,36 +61,47 @@ You'll want to make sure that your tests are using actual test values before pro ### **Step 2:** Develop Your Terraform Plan Based Assertions (Unit Testing) -The previous step properly prepared you for test isolation. Now you can concern yourself with the business of developing assertions. The best way to make assertions about your CIT is to rely on what's visible in the Terraform Plan. Terraform Plans are what's presented to you in standard out when running `terraform plan`. This is the earliest phase in the dev workflow that you can rely on for your test assertions. +The previous step properly prepared you for test isolation. Now you can concern yourself with the business of developing assertions. The best way to make assertions about your CIT is to rely on what's visible in the Terraform Plan. Terraform Plans are presented to you in standard out when running `terraform plan`. This is the earliest phase in the dev workflow that you'll run tests against programmatically. 1. Learn to read the Terraform Plan You'll want to learn how to read a Terraform Plan. This will be the most efficient piece of information to use in discovering most of the infrastructure that your CIT plans to deploy before actual deployment. Terraform uses all module properties (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) present in your CIT in order to build a [Terraform Dependency Graph](https://www.terraform.io/docs/internals/graph.html) at plan time. This dependency graph is then represented as a list of resource address nodes that make up the Terraform plan visible in standard out. The state of that plan is what gets executed when running `terraform apply`. -2. Plan assertions to make about your CIT +2. Inspect Terraform Plan for assertions to make about your CIT - The resource addresses in your Terraform Plan have varying levels of nested information. Some of the values located at each resource address can be seen at plan time and some are so dynamic that they are not discoverable until the `terraform apply` command has finished executing. A good rule of thumb is values that are visible at plan time are unit testable, values not visible require integration testing. + The resource addresses in your Terraform Plan have varying levels of nested information. Some of the values located at each resource address can be seen at plan time and some are so dynamic that they are not resolvable until the `terraform apply` command has finished executing. A good rule of thumb is, "values visible at plan time are unit testable, values not visible require integration testing.". - > **TIP:** Test as much as you can here because integration tests require spinning up and tearing down actual infrastructure beyond a state file. + > **TIP:** Test as much as you can here because integration tests require spinning up and tearing down real infrastructure. - * Run `terraform plan` and look over the Terraform Plan that is presented so that you can decide on which assertions you'd like to make. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. + * Run `terraform plan` and inspect the Terraform Plan that is presented so that you can decide on which assertions you'd like to make. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. - Here are a few examples of assertions we decided to make for tests within our az-hello-world CIT: + Here are a few examples of assertions we wanted our unit tests to make about our az-hello-world CIT: | Terraform Plan Resource Address | Module Property Type | Planned Assertion | - |--------|-----------|-----------|-----------| + |--------|-----------|-----------| | `module.app_service.azurerm_app_service.appsvc[0]` | module | Assert the presence of docker image in the configuration of the app service module. | - | `module.service_plan.azurerm_app_service_plan.svcplan` | module | Assert that the service plan module for the az-hello-world CIT is configured for the least expensive S1 tier. | + | `module.service_plan.azurerm_app_service_plan.svcplan` | module | Assert that the service plan module for the az-hello-world CIT is configured for the least expensive S1 tier. | | `azurerm_resource_group.main` | resource | Assert the resource group contains the datacenter used for test environments. | -### **Step 3:** Decide on which properties of your Terraform State are testable (Integration Tests) +### **Step 3:** Develop Your Terraform Apply Based Assertions (Integration Tests) +In the previous step we stated, "Values visible at plan time are unit testable, values not visible require integration testing.". After completing the exercise of developing Terraform Plan based assertions, it follows then that we must develop assertions about values that are not resolvable until the `terraform apply` command has finished executing. These values are generated from the actual infrastructure that deployed. Complete the following step for guidance on developing assertions about infrastructure for integration testing. + +* Map unresolvable Terraform Plan values to CIT outputs for inspection - *More on Output Values:* https://www.terraform.io/docs/configuration/outputs.html + + Outputs are return values for modules, therefore, your CIT also has return values visible in standard out when the `terraform apply` command has finished executing. If your CIT is not already configured for outputs that represent unresolvable Terraform Plan properties, it's recommended that you go back and make that happen. + + Here's an example of an assertion we wanted our integration test to make about the az-hello-world CIT deployment.: + + | Unresolvable Terraform Plan Value | Output Var Name | Assertion | + |--------|-----------|-----------| + | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url thats mapped to the CIT's output returns a status of 200. | ### **Step 4:** Decide on a Go Testing Framework ### **Step 5:** Decide on a Terraform Testing Framework -Terratest vs kitchen-terraform. +Terratest vs kitchen-terraform vs Terraform's built in testing framework (make test ) ### **Step 6:** Write Unit Tests From 4401078a38ba442f1de835a5ec16e70976c927af Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Thu, 31 Oct 2019 17:16:06 -0500 Subject: [PATCH 10/16] testing a template - create instructions for choosing a Terraform Testing Framework plus more --- docs/4_TEMPLATE_TESTING.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 6db32255..bea0184a 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -23,16 +23,16 @@ With that being said, the task of building automated tests for infrastructure ca | [Quickstart Guide](./2_QUICK_START_GUIDE.md) | This should have served as your first Cobalt Infrastructure deployment. | | [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | Completing this prequisite leaves you with a CIT and prior knowledge needed for this walkthrough. | | [Terraform Modules](https://www.terraform.io/docs/configuration/modules.html) | An introductory understanding of Terraform modules.| -| [Go](Golang) | An introductory understanding of Go. | -| [Go Test](Golang) | An introductory understanding of Go Tests. | -| [TerraTest](Gruntworks) | An introductory understanding of TerraTest. | -| [Docker](Docker.io) | An introductory understanding of Docker. | +| [Golang](https://golang.org/dl/) | An introductory understanding of Golang. | +| [Go Test](https://golang.org/pkg/testing/) | An introductory understanding of Golang's native testing package. | +| [TerraTest](https://github.com/gruntwork-io/terratest) | An introductory understanding of TerraTest. | +| [Docker](https://www.docker.com/) | An introductory understanding of Docker. | ## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template (CIT) If you are used to developing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. Executing this workflow so far has required you to interact with Terraform via it's cli. However, you might not have known that the Terraform cli runs commands against [Terraform Core](https://www.terraform.io/docs/extend/how-terraform-works.html#terraform-core), a statically compiled Golang binary. Therefore, in order to programmatically interact with Terraform to achieve automation, you'll have to write tests in Golang. -Furthermore, in order to make automation useful for testing, you'll have to know where in the various phases of your developer workflow enable test isolation. You'll also have to know where it is that you can start making assertions about your CIT's deployment plans and the infrastructure that was stood up in an automated way. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write automated test in Golang for your CIT. +Furthermore, in order to make automation useful for testing, you'll have to know where in the various phases of your developer workflow enable test isolation. You'll also have to know where in an automated way it is that you can start making assertions about your CIT's deployment plans and the infrastructure that it will stand up. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write automated test in Golang for your CIT. > **NOTE:** This walkthrough only focuses on running automated tests locally. Our next walkthrough *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various stages. @@ -93,22 +93,22 @@ In the previous step we stated, "Values visible at plan time are unit testable, Here's an example of an assertion we wanted our integration test to make about the az-hello-world CIT deployment.: - | Unresolvable Terraform Plan Value | Output Var Name | Assertion | + | Unresolvable Terraform Plan Value | Output Var Name | Planned Assertion | |--------|-----------|-----------| - | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url thats mapped to the CIT's output returns a status of 200. | + | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url that is mapped to the CIT's output returns a status of 200. | -### **Step 4:** Decide on a Go Testing Framework +### **Step 4:** Choose a Terraform Testing Framework -### **Step 5:** Decide on a Terraform Testing Framework +The job of the testing framework you choose is to make it easy to execute on the assertions you developed in the previous steps. We chose [TerraTest](https://github.com/gruntwork-io/terratest) to bake into our codebase a testing Framework that folds in alot of the test setup needed to run all future test assertions needed for a CIT in Cobalt. Visit [How-terratest-compares-to-other-testing-tools](https://github.com/gruntwork-io/terratest#how-terratest-compares-to-other-testing-tools) for a further explanation. TerraTest has a hard dependency on Golang's native testing package, therefore, the decision of which Golang testing framework to use has also been made. -Terratest vs kitchen-terraform vs Terraform's built in testing framework (make test ) +### **Step 5:** Write Unit Tests -### **Step 6:** Write Unit Tests +When running unit tests in Cobalt, we suggest coding against the provided test harness. Our test harness minimizes the boiler plate code required to effectively test CITs. This will in turn reduce the effort required to write robust unit tests for CITs in Cobalt. -> **NOTE**: This example relies on the testing harness. +### **Step 6:** Run Unit Tests -### **Step 7:** Run Unit Tests +### **Step 7:** Write Integration Tests -### **Step 8:** Write Integration Tests +When running integration tests in Cobalt, we suggest coding against the provided test harness. Our test harness minimizes the boiler plate code required to effectively test CITs. This will in turn reduce the effort required to write robust integration tests for CITs in Cobalt. -### **Step 9:** Run Integration Tests +### **Step 8:** Run Integration Tests From 9966f0dd0366f60311266b14aa7a78c44b4e6990 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Fri, 1 Nov 2019 00:17:19 -0500 Subject: [PATCH 11/16] testing a template - further refine steps 1-3 --- docs/4_TEMPLATE_TESTING.md | 54 +++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index bea0184a..55ba8476 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -2,7 +2,7 @@ ## 4.1 Overview -As software developers, we seek out opportunities to improve and grow projects via code contributions. Code contributions, whether major or gradual, require changes that unfortunately bring with them a chance for unexpected behavior to occur. Fortunately, automated testing grants us a chance to gain more predictability and control over the changes that occur from our contributions. Without automated testing, contributions to our codebase would simply be too unpredictable to be considered scalable. +As software developers, we seek out opportunities to improve and grow projects via code contributions. Code contributions, whether major or gradual, require changes that unfortunately bring with them a chance for unexpected behavior to occur. Fortunately, automated testing grants us a chance to gain more predictability and control over the changes that occur from our contributions. By running pre-existing automated tests every time new code changes are introduced, we can catch problems before they are introduced into production. With that being said, the task of building automated tests for infrastructure can feel a bit different then the kinds of tests you are used to. For this project, let's further build on what you've learned so far by introducing the approach Cobalt has taken to infrastructure testing and how that will impact what you know about the Cobalt Developer Workflow: @@ -34,68 +34,68 @@ If you are used to developing in Terraform, you'll realize that the Cobalt Devel Furthermore, in order to make automation useful for testing, you'll have to know where in the various phases of your developer workflow enable test isolation. You'll also have to know where in an automated way it is that you can start making assertions about your CIT's deployment plans and the infrastructure that it will stand up. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write automated test in Golang for your CIT. -> **NOTE:** This walkthrough only focuses on running automated tests locally. Our next walkthrough *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various stages. +> **NOTE:** This walkthrough only focuses on running automated tests locally. Our next walkthrough *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various deployment stages. ### **Step 1:** Prepare for Test Isolation -You'll want to make sure that your tests are using actual test values before programmatically running `terraform plan`. The values you provide should be non-production values so as to not affect end-user environments. In the Cobalt Develop Workflow, the opportunities to enforce test isolation first begin with setting up your local environment variables, and then running `terraform init` and `terraform workspace new ` from your CIT's directory. +You'll want to make sure that your tests are using actual test values before programmatically running `terraform plan` in Golang. The values you provide should be non-production values so as to not affect end-user environments. In the Cobalt Develop Workflow, the opportunities to enforce test isolation first begin with setting up your local environment variables, and then running `terraform init` and `terraform workspace new ` from your CIT's directory. -> **NOTE:** Due to the beginning of the Cobalt Developer Workflow having a hard dependency on a back-end state file, both unit testing and integration testing will need these environment variables. Therefore, we recommend making these decisions early-on as a first step. +> **NOTE:** Due to the beginning of the Cobalt Developer Workflow having a hard dependency on a remote back-end state file, both unit testing and integration testing will need these environment variables. Therefore, we highly recommend completing this step. -1. Prepare your environment variables. Ensure that your .env file is using non-production values as you will be referencing these when writing your automated tests. These values are usually specific to the provider your CIT is targeting. Here are a few examples: +1. Prepare your environment variables. + + Ensure that your .env file is using non-production values as you will be referencing these when writing your automated tests in Golang. These values are usually specific to the provider your CIT is targeting. Here are a few examples: | Env Variables | Test Value | Description | |--------|----------|-----------| | `storage_account_name` | Value from "TF_VAR_remote_state_account" | This value should be the storage account dedicated to the dev environment. The value lives within your .env file | - | `container_name` | Value from "TF_VAR_remote_state_container" | This value will be the dedicated container that will holder multiple backend remote workspace terraform state files. The value lives within your .env file . | + | `container_name` | Value from "TF_VAR_remote_state_container" | This value will be the dedicated remote container that will hold multiple Terraform backend workspace state files. The value lives within your .env file. | + +2. Locate the type of input variables from your CIT that further help you achieve test isolation and prepare values for them. -2. Locate the input variables from your CIT that further help you achieve test isolation and choose values for them. Here are all the input variables and values we used to help further prepare the az-hello-world CIT for test isolation: + > **NOTE:** Some CITs rely on a commons.tf file that resolves input names into unique names for cloud infrastructure resources like fqdns. This file is an answer to the inherit naming constraints that come with some cloud resources. Therefore, some of the names you provide as inputs for your CIT will be further sanitized by the commons.tf file. Another feature of this file is that it improves work isolation. - > **NOTE:** The az-hello-world CIT relies on a commons.tf file that enforces uniqueness of resource names for varying reasons, therefore, the names you choose below will be further transformed based on the implementation of the commons.tf file. Some of the reasons for the name transformations include work isolation and other reasons are due to resources that require unique names like fqdns. + Here are all the input variables and values we used to help further prepare the az-hello-world CIT for test isolation in Golang: | Input Var Name | Test Value | Description | |--------|----------|-----------| - | `workspace` | "az-hello-world-" + guid | It's important that a random guid is attached to the workspace of your tests. Teams share cloud provider resources and accounts. The workspace name may be your only solution to working in isolation from other devs on your team. | - | `name` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests as those kinds of tests create actual infrastructure.| - | `resource_group_location` | "eastus" | The geo-location of the Azure datacenter in which you want the resource group that contains your infrastructure to live. A geo-location maybe different then production. | + | `workspace` | "az-hello-world-" + guid | Teams share cloud provider resources and accounts. A workspace name with a random guid may be your only solution to working in isolation from other devs on your team. | + | `name` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests that create actual infrastructure.| + | `resource_group_location` | "eastus" | The geo-location of the Azure datacenter in which you want the resource group that contains your infrastructure to live for the dev environment. | ### **Step 2:** Develop Your Terraform Plan Based Assertions (Unit Testing) -The previous step properly prepared you for test isolation. Now you can concern yourself with the business of developing assertions. The best way to make assertions about your CIT is to rely on what's visible in the Terraform Plan. Terraform Plans are presented to you in standard out when running `terraform plan`. This is the earliest phase in the dev workflow that you'll run tests against programmatically. +The previous step properly prepared you for test isolation. Now you can concern yourself with the business of test assertions. The best way to formulate test assertions about your CIT is to rely on what's visible in the Terraform Plan. Terraform Plans are presented to you in standard out when running `terraform plan`. This is the earliest phase in the dev workflow where you'll use Golang to run tests programmatically. 1. Learn to read the Terraform Plan - You'll want to learn how to read a Terraform Plan. This will be the most efficient piece of information to use in discovering most of the infrastructure that your CIT plans to deploy before actual deployment. Terraform uses all module properties (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) present in your CIT in order to build a [Terraform Dependency Graph](https://www.terraform.io/docs/internals/graph.html) at plan time. This dependency graph is then represented as a list of resource address nodes that make up the Terraform plan visible in standard out. The state of that plan is what gets executed when running `terraform apply`. + You'll want to learn how to read a [Terraform Plan](https://www.terraform.io/docs/commands/plan.html). This will be the most efficient piece of information to use in discovering most of the infrastructure that your CIT plans to deploy before actual deployment. Terraform uses all module properties (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) present in your CIT in order to build a [Terraform Dependency Graph](https://www.terraform.io/docs/internals/graph.html) at plan time. This dependency graph is then represented as a list of resource address nodes that make up the Terraform plan visible in standard out. The state of that plan is what gets executed when running `terraform apply`. -2. Inspect Terraform Plan for assertions to make about your CIT +2. Inspect Terraform Plan to formulate test assertions about your CIT - The resource addresses in your Terraform Plan have varying levels of nested information. Some of the values located at each resource address can be seen at plan time and some are so dynamic that they are not resolvable until the `terraform apply` command has finished executing. A good rule of thumb is, "values visible at plan time are unit testable, values not visible require integration testing.". + The resource addresses visible in your Terraform Plan have varying levels of nested information. Some of the values located at each resource address can be seen at plan time and some are not resolvable until the `terraform apply` command has finished executing. A good rule of thumb is, "values visible at plan time are unit testable, values not visible require integration testing.". - > **TIP:** Test as much as you can here because integration tests require spinning up and tearing down real infrastructure. + > **TIP:** Unit test as much as you can here because integration tests require spinning up and tearing down real infrastructure and that takes time. Integration tests will drammatically slow down your Cobalt Developer Workflow. - * Run `terraform plan` and inspect the Terraform Plan that is presented so that you can decide on which assertions you'd like to make. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. + * Run `terraform plan` and inspect the Terraform Plan to formulate test assertions. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. x`Here are a few examples of test assertions we formulated for our az-hello-world CIT's unit tests: - Here are a few examples of assertions we wanted our unit tests to make about our az-hello-world CIT: - - | Terraform Plan Resource Address | Module Property Type | Planned Assertion | + | Terraform Plan Resource Address | Module Property Type | Test Assertion | |--------|-----------|-----------| - | `module.app_service.azurerm_app_service.appsvc[0]` | module | Assert the presence of docker image in the configuration of the app service module. | + | `module.app_service.azurerm_app_service.appsvc[0]` | module | Assert the presence of a docker image in the app service module's configuration. | | `module.service_plan.azurerm_app_service_plan.svcplan` | module | Assert that the service plan module for the az-hello-world CIT is configured for the least expensive S1 tier. | | `azurerm_resource_group.main` | resource | Assert the resource group contains the datacenter used for test environments. | ### **Step 3:** Develop Your Terraform Apply Based Assertions (Integration Tests) -In the previous step we stated, "Values visible at plan time are unit testable, values not visible require integration testing.". After completing the exercise of developing Terraform Plan based assertions, it follows then that we must develop assertions about values that are not resolvable until the `terraform apply` command has finished executing. These values are generated from the actual infrastructure that deployed. Complete the following step for guidance on developing assertions about infrastructure for integration testing. - -* Map unresolvable Terraform Plan values to CIT outputs for inspection - *More on Output Values:* https://www.terraform.io/docs/configuration/outputs.html +In the previous step we stated, "Values visible at plan time are unit testable, values not visible require integration testing.". After completing the exercise of developing Terraform Plan based assertions, it follows that we must also develop Terraform Apply based assertions. These are test assertions about values that are not resolvable until the `terraform apply` command finishes deploying real infrastructure. - Outputs are return values for modules, therefore, your CIT also has return values visible in standard out when the `terraform apply` command has finished executing. If your CIT is not already configured for outputs that represent unresolvable Terraform Plan properties, it's recommended that you go back and make that happen. +* Map unresolvable Terraform Plan values to CIT outputs for inspection - Visit [Terraform Outputs](https://www.terraform.io/docs/configuration/outputs.html) to learn more. - Here's an example of an assertion we wanted our integration test to make about the az-hello-world CIT deployment.: + Outputs are return values for modules, therefore, your CIT also has return values visible in standard out when the `terraform apply` command has finished executing. Reconfigure your CIT so that ouputs map to the unresolvable properties in your Terraform Plan that you care about. Here's an example of a test assertion we formulated for our az-hello-world CIT's integration tests: | Unresolvable Terraform Plan Value | Output Var Name | Planned Assertion | |--------|-----------|-----------| - | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url that is mapped to the CIT's output returns a status of 200. | + | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url (A value that is mapped to the CIT's output.) returns a status of 200. | ### **Step 4:** Choose a Terraform Testing Framework From 50ce7c853bffe90aed452b2fbc86f2b7e3699ccf Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Fri, 1 Nov 2019 08:28:34 -0500 Subject: [PATCH 12/16] test new template - improve explanation of decision to use terratest --- docs/4_TEMPLATE_TESTING.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 55ba8476..c9e61dba 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -77,7 +77,7 @@ The previous step properly prepared you for test isolation. Now you can concern > **TIP:** Unit test as much as you can here because integration tests require spinning up and tearing down real infrastructure and that takes time. Integration tests will drammatically slow down your Cobalt Developer Workflow. - * Run `terraform plan` and inspect the Terraform Plan to formulate test assertions. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. x`Here are a few examples of test assertions we formulated for our az-hello-world CIT's unit tests: + * Run `terraform plan` and inspect the Terraform Plan to formulate test assertions. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. Here are a few examples of test assertions we formulated for our az-hello-world CIT's unit tests: | Terraform Plan Resource Address | Module Property Type | Test Assertion | |--------|-----------|-----------| @@ -91,24 +91,28 @@ In the previous step we stated, "Values visible at plan time are unit testable, * Map unresolvable Terraform Plan values to CIT outputs for inspection - Visit [Terraform Outputs](https://www.terraform.io/docs/configuration/outputs.html) to learn more. - Outputs are return values for modules, therefore, your CIT also has return values visible in standard out when the `terraform apply` command has finished executing. Reconfigure your CIT so that ouputs map to the unresolvable properties in your Terraform Plan that you care about. Here's an example of a test assertion we formulated for our az-hello-world CIT's integration tests: + Outputs are simply return values for modules, therefore, your CIT also has return values. These outputs are visible in standard out when the `terraform apply` command has finished executing. Reconfigure your CIT by taking unresolvable properties that you care about from your Terraform Plan and mapping them to outputs configured in your CIT. Here's an example of a test assertion we formulated for our az-hello-world CIT's integration tests: | Unresolvable Terraform Plan Value | Output Var Name | Planned Assertion | |--------|-----------|-----------| | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url (A value that is mapped to the CIT's output.) returns a status of 200. | -### **Step 4:** Choose a Terraform Testing Framework +### **Step 4:** Choose a Testing Framework -The job of the testing framework you choose is to make it easy to execute on the assertions you developed in the previous steps. We chose [TerraTest](https://github.com/gruntwork-io/terratest) to bake into our codebase a testing Framework that folds in alot of the test setup needed to run all future test assertions needed for a CIT in Cobalt. Visit [How-terratest-compares-to-other-testing-tools](https://github.com/gruntwork-io/terratest#how-terratest-compares-to-other-testing-tools) for a further explanation. TerraTest has a hard dependency on Golang's native testing package, therefore, the decision of which Golang testing framework to use has also been made. +It is our opinion that the job of the testing framework you ultimately choose should be based on how well it lends itself to executing on the assertions you developed in the previous steps. In our case, we chose [TerraTest](https://github.com/gruntwork-io/terratest). TerraTest has a hard dependency on Golang's native testing package, therefore, the decision of which Golang testing framework to use has also been made. Visit [How-terratest-compares-to-other-testing-tools](https://github.com/gruntwork-io/terratest#how-terratest-compares-to-other-testing-tools) for a further explanation. ### **Step 5:** Write Unit Tests -When running unit tests in Cobalt, we suggest coding against the provided test harness. Our test harness minimizes the boiler plate code required to effectively test CITs. This will in turn reduce the effort required to write robust unit tests for CITs in Cobalt. +When running unit tests in Cobalt, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to effectively use [TerraTest](https://github.com/gruntwork-io/terratest) in order to test CITs. This will in turn reduce the effort required to write robust unit tests for CITs in Cobalt. + +1. Pending ### **Step 6:** Run Unit Tests +1. Pending + ### **Step 7:** Write Integration Tests -When running integration tests in Cobalt, we suggest coding against the provided test harness. Our test harness minimizes the boiler plate code required to effectively test CITs. This will in turn reduce the effort required to write robust integration tests for CITs in Cobalt. +When running integration tests in Cobalt, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to effectively use [TerraTest](https://github.com/gruntwork-io/terratest) in order to test CITs. This will in turn reduce the effort required to write robust integrations tests for CITs in Cobalt. ### **Step 8:** Run Integration Tests From 32759b658d067263f061f6a0e9e2844e9ede38f6 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Mon, 18 Nov 2019 17:32:04 -0600 Subject: [PATCH 13/16] expand unit and integration test instructions --- docs/4_TEMPLATE_TESTING.md | 168 +++++++++++++++++++++++++++++++------ 1 file changed, 144 insertions(+), 24 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index c9e61dba..cd839a56 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -2,7 +2,7 @@ ## 4.1 Overview -As software developers, we seek out opportunities to improve and grow projects via code contributions. Code contributions, whether major or gradual, require changes that unfortunately bring with them a chance for unexpected behavior to occur. Fortunately, automated testing grants us a chance to gain more predictability and control over the changes that occur from our contributions. By running pre-existing automated tests every time new code changes are introduced, we can catch problems before they are introduced into production. +As software developers, we seek out opportunities to improve and grow projects via code contributions. Code contributions, whether major or gradual, require changes that unfortunately bring with them a chance of unexpected behavior to occur. Fortunately, automated testing grants us a chance to gain more predictability and control over the changes that occur from our contributions. By running pre-existing automated tests every time new code changes are introduced, we can catch problems before they are deliverd into a production environment. With that being said, the task of building automated tests for infrastructure can feel a bit different then the kinds of tests you are used to. For this project, let's further build on what you've learned so far by introducing the approach Cobalt has taken to infrastructure testing and how that will impact what you know about the Cobalt Developer Workflow: @@ -23,16 +23,17 @@ With that being said, the task of building automated tests for infrastructure ca | [Quickstart Guide](./2_QUICK_START_GUIDE.md) | This should have served as your first Cobalt Infrastructure deployment. | | [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | Completing this prequisite leaves you with a CIT and prior knowledge needed for this walkthrough. | | [Terraform Modules](https://www.terraform.io/docs/configuration/modules.html) | An introductory understanding of Terraform modules.| -| [Golang](https://golang.org/dl/) | An introductory understanding of Golang. | +| [Golang](https://golang.org/dl/) (1.12.5 +) | Our testing strategy depends on Golang. Install it and gain an introductory understanding. | +| [Go Modules](https://blog.golang.org/using-go-modules) | An introductory understanding of Golang's latest dependency management system. | | [Go Test](https://golang.org/pkg/testing/) | An introductory understanding of Golang's native testing package. | | [TerraTest](https://github.com/gruntwork-io/terratest) | An introductory understanding of TerraTest. | -| [Docker](https://www.docker.com/) | An introductory understanding of Docker. | +| [Docker](https://docs.docker.com/install/) (18.09 +) | A secondary option for running our tests. An introductory understanding of Docker. | ## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template (CIT) -If you are used to developing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. Executing this workflow so far has required you to interact with Terraform via it's cli. However, you might not have known that the Terraform cli runs commands against [Terraform Core](https://www.terraform.io/docs/extend/how-terraform-works.html#terraform-core), a statically compiled Golang binary. Therefore, in order to programmatically interact with Terraform to achieve automation, you'll have to write tests in Golang. +If you are used to developing in Terraform, you'll realize that the Cobalt Developer Workflow is the way it is because CITs are primarily written in Terraform's HCL configuration language as are the modules that they are composed of. Executing this workflow so far has required you to use the Terraform cli. However, you might not have known that the Terraform cli is actually running commands against [Terraform Core](https://www.terraform.io/docs/extend/how-terraform-works.html#terraform-core), a statically compiled Golang binary. Therefore, in order to programmatically interact with Terraform to implement test automation, the most practical path is to use Golang. -Furthermore, in order to make automation useful for testing, you'll have to know where in the various phases of your developer workflow enable test isolation. You'll also have to know where in an automated way it is that you can start making assertions about your CIT's deployment plans and the infrastructure that it will stand up. The following steps provide guidance on using your knowledge of the Cobalt Developer Workflow to properly write automated test in Golang for your CIT. +Furthermore, you'll have to know where in an automated way it is that you can start making assertions about your CIT's deployment plans and the infrastructure that it will stand up. Keep your knowledge of the Cobatl Developer Workflow in mind as you complete the following steps provided for writing automated tests in Golang for your CIT. > **NOTE:** This walkthrough only focuses on running automated tests locally. Our next walkthrough *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* will cover ways that these automated tests can be run on code commits and various deployment stages. @@ -48,12 +49,12 @@ You'll want to make sure that your tests are using actual test values before pro | Env Variables | Test Value | Description | |--------|----------|-----------| - | `storage_account_name` | Value from "TF_VAR_remote_state_account" | This value should be the storage account dedicated to the dev environment. The value lives within your .env file | + | `storage_account_name` | Value from "TF_VAR_remote_state_account" | This value should be the storage account dedicated to the dev environment. The value lives within your .env file. | | `container_name` | Value from "TF_VAR_remote_state_container" | This value will be the dedicated remote container that will hold multiple Terraform backend workspace state files. The value lives within your .env file. | 2. Locate the type of input variables from your CIT that further help you achieve test isolation and prepare values for them. - > **NOTE:** Some CITs rely on a commons.tf file that resolves input names into unique names for cloud infrastructure resources like fqdns. This file is an answer to the inherit naming constraints that come with some cloud resources. Therefore, some of the names you provide as inputs for your CIT will be further sanitized by the commons.tf file. Another feature of this file is that it improves work isolation. + > **NOTE:** Some CITs rely on a commons.tf file that resolves input names into unique names for cloud infrastructure resources like fqdns. The commons.tf file is an answer to the inherit naming constraints that come with some cloud resources. Therefore, some of the names you provide as inputs for your CIT will be further sanitized by the commons.tf file. Another feature of this file is that it improves work isolation. Here are all the input variables and values we used to help further prepare the az-hello-world CIT for test isolation in Golang: @@ -61,7 +62,7 @@ You'll want to make sure that your tests are using actual test values before pro |--------|----------|-----------| | `workspace` | "az-hello-world-" + guid | Teams share cloud provider resources and accounts. A workspace name with a random guid may be your only solution to working in isolation from other devs on your team. | | `name` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests that create actual infrastructure.| - | `resource_group_location` | "eastus" | The geo-location of the Azure datacenter in which you want the resource group that contains your infrastructure to live for the dev environment. | + | `resource_group_location` | "eastus" | The geo-location of the Azure datacenters inside of which you want the resource group that will contain your dev infrastructure to live. | ### **Step 2:** Develop Your Terraform Plan Based Assertions (Unit Testing) @@ -69,13 +70,13 @@ The previous step properly prepared you for test isolation. Now you can concern 1. Learn to read the Terraform Plan - You'll want to learn how to read a [Terraform Plan](https://www.terraform.io/docs/commands/plan.html). This will be the most efficient piece of information to use in discovering most of the infrastructure that your CIT plans to deploy before actual deployment. Terraform uses all module properties (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) present in your CIT in order to build a [Terraform Dependency Graph](https://www.terraform.io/docs/internals/graph.html) at plan time. This dependency graph is then represented as a list of resource address nodes that make up the Terraform plan visible in standard out. The state of that plan is what gets executed when running `terraform apply`. + You'll want to learn how to read a [Terraform Plan](https://www.terraform.io/docs/commands/plan.html). This will be the most efficient piece of information to use in discovering most of the infrastructure that your CIT plans to deploy before actual deployment. In short, Terraform uses all module properties (i.e. input variables, output variables, provider blocks, resource blocks, other modules and more) present in your CIT in order to build a [Dependency Graph](https://www.terraform.io/docs/internals/graph.html) at plan time. This dependency graph is then represented as a list of resource address nodes that make up the Terraform plan visible in standard out. The state of that plan is what gets executed when running `terraform apply`. 2. Inspect Terraform Plan to formulate test assertions about your CIT - The resource addresses visible in your Terraform Plan have varying levels of nested information. Some of the values located at each resource address can be seen at plan time and some are not resolvable until the `terraform apply` command has finished executing. A good rule of thumb is, "values visible at plan time are unit testable, values not visible require integration testing.". + The resource addresses visible in your Terraform Plan have varying levels of nested information. Some of the values located at each resource address can be seen at plan time and some are not resolvable until the `terraform apply` command has finished executing. - > **TIP:** Unit test as much as you can here because integration tests require spinning up and tearing down real infrastructure and that takes time. Integration tests will drammatically slow down your Cobalt Developer Workflow. + > **TIP:** A good rule of thumb is, "values visible at plan time are unit testable, values not visible require integration testing.". Make full use of your unit tests because integration tests require spinning up and tearing down real infrastructure and that takes time. * Run `terraform plan` and inspect the Terraform Plan to formulate test assertions. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. Here are a few examples of test assertions we formulated for our az-hello-world CIT's unit tests: @@ -83,36 +84,155 @@ The previous step properly prepared you for test isolation. Now you can concern |--------|-----------|-----------| | `module.app_service.azurerm_app_service.appsvc[0]` | module | Assert the presence of a docker image in the app service module's configuration. | | `module.service_plan.azurerm_app_service_plan.svcplan` | module | Assert that the service plan module for the az-hello-world CIT is configured for the least expensive S1 tier. | - | `azurerm_resource_group.main` | resource | Assert the resource group contains the datacenter used for test environments. | + | `azurerm_resource_group.main` | resource | Assert the resource group contains the datacenter used for dev environments. | ### **Step 3:** Develop Your Terraform Apply Based Assertions (Integration Tests) -In the previous step we stated, "Values visible at plan time are unit testable, values not visible require integration testing.". After completing the exercise of developing Terraform Plan based assertions, it follows that we must also develop Terraform Apply based assertions. These are test assertions about values that are not resolvable until the `terraform apply` command finishes deploying real infrastructure. +In the previous step we stated, "Values visible at plan time are unit testable, values not visible require integration testing.". After completing the exercise of developing Terraform _Plan_ based assertions, it follows that we must also develop Terraform _Apply_ based assertions. These are test assertions about values that are not resolvable until the `terraform apply` command finishes deploying real infrastructure. -* Map unresolvable Terraform Plan values to CIT outputs for inspection - Visit [Terraform Outputs](https://www.terraform.io/docs/configuration/outputs.html) to learn more. +* Make unresolvable Terraform Plan values testable - Visit [Terraform Outputs](https://www.terraform.io/docs/configuration/outputs.html) to learn more. - Outputs are simply return values for modules, therefore, your CIT also has return values. These outputs are visible in standard out when the `terraform apply` command has finished executing. Reconfigure your CIT by taking unresolvable properties that you care about from your Terraform Plan and mapping them to outputs configured in your CIT. Here's an example of a test assertion we formulated for our az-hello-world CIT's integration tests: + You'll have to map unresolvable Terraform Plan values to CIT outputs. Outputs are simply return values for modules, therefore, your CIT also has return values. These outputs are visible in standard out when the `terraform apply` command has finished executing. Reconfigure your CIT by taking unresolvable properties that you care about from your Terraform Plan and mapping them to outputs configured in your CIT. Here's an example of a test assertion we formulated for our az-hello-world CIT's integration tests: | Unresolvable Terraform Plan Value | Output Var Name | Planned Assertion | |--------|-----------|-----------| | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url (A value that is mapped to the CIT's output.) returns a status of 200. | -### **Step 4:** Choose a Testing Framework +### **Step 4:** Choose Testing Frameworks (Terraform and Golang) -It is our opinion that the job of the testing framework you ultimately choose should be based on how well it lends itself to executing on the assertions you developed in the previous steps. In our case, we chose [TerraTest](https://github.com/gruntwork-io/terratest). TerraTest has a hard dependency on Golang's native testing package, therefore, the decision of which Golang testing framework to use has also been made. Visit [How-terratest-compares-to-other-testing-tools](https://github.com/gruntwork-io/terratest#how-terratest-compares-to-other-testing-tools) for a further explanation. +It is our opinion that the job of the testing framework you ultimately choose should be based on how well it lends itself to executing on the assertions you developed in the previous steps. In our case, we chose [TerraTest](https://github.com/gruntwork-io/terratest) as our Terraform testing framework. A byproduct of using TerraTest is that we must also use Golang's native testing framework ([Go Test](https://golang.org/pkg/testing/)) because Terratest depends on it. Visit [How-terratest-compares-to-other-testing-tools](https://github.com/gruntwork-io/terratest#how-terratest-compares-to-other-testing-tools) to examine the trade-offs. ### **Step 5:** Write Unit Tests -When running unit tests in Cobalt, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to effectively use [TerraTest](https://github.com/gruntwork-io/terratest) in order to test CITs. This will in turn reduce the effort required to write robust unit tests for CITs in Cobalt. +When writing unit tests for Cobalt CITs, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to wire-up [TerraTest](https://github.com/gruntwork-io/terratest) and [Go Test](https://golang.org/pkg/testing/) for test automation. + +The test harness implementation automates the command line execution of the Cobalt Developer Workflow (i.e. *create/choose/extend* a template/module ---> init ---> select workspace ---> plan ---> **test** ---> apply ---> **test** ---> destroy). The test harness is packaged and is exposed in way that allows you to provide your custom testing context as inputs to the harness. + +> **NOTE:** Golang compiles to a static library, therefore, it's important that all types are defined ahead of time. With a little familiarity on how the [Golang interface](https://tour.golang.org/methods/14) type behaves, you'll understand how the below examples initialize variables to provide context to the testing harness hooks for running unit tests within the az-hello-world CIT. + +1. Initialize testharness variables for dev isolation + + The code snippet below comes from the unit tests within the az-hello-world CIT. It initializes a Terratest object called *Options* with the values that we preselected for achieving test isolation. This *Options* object acts as the first step for setting up the context needed to run unit tests within the test harness. + + ```go + var workspace = fmt.Sprintf("az-hello-world-%s", random.UniqueId()) + var prefix = fmt.Sprintf("az-hw-unit-tst-%s", random.UniqueId()) + var datacenter = "eastus" + + var tfOptions = &terraform.Options{ + TerraformDir: "../../", + Upgrade: true, + Vars: map[string]interface{}{ + "name": prefix, + "resource_group_location": datacenter, + }, + BackendConfig: map[string]interface{}{ + "storage_account_name": os.Getenv("TF_VAR_remote_state_account"), + "container_name": os.Getenv("TF_VAR_remote_state_container"), + }, + } + ``` + +2. Add your Terraform Plan based assertions + + The code snippet below also comes from the unit tests within the az-hello-world CIT. It initializes a custom test harness object called *UnitTestFixture* with the values that will be later passed to Terratest and other logic needed for test automation. The *ResourceDescription* object is where you plug in your formulated test assertions as code. The test harness will run assertions for you by parsing the Terraform Plan as long as you provide the expected values and their respective data types. + + ```go + func TestAzureSimple(t *testing.T) { + testFixture := infratests.UnitTestFixture{ + GoTest: t, + TfOptions: tfOptions, + ExpectedResourceCount: 10, + ... + ExpectedResourceAttributeValues: infratests.ResourceDescription{ + "module.app_service.azurerm_app_service.appsvc[0]": map[string]interface{}{ + ... + "site_config": []interface{}{ + map[string]interface{}{"linux_fx_version": "DOCKER|docker.io/appsvcsample/static-site:latest"}, + }, + }, + "module.service_plan.azurerm_app_service_plan.svcplan": map[string]interface{}{ + ... + "sku": []interface{}{ + map[string]interface{}{"size": "S1", "tier": "Standard"}, + }, + }, + "azurerm_resource_group.main": map[string]interface{}{ + "location": datacenter, + }, + }, + } + infratests.RunUnitTests(&testFixture) + } + ``` + +### **Step 6:** Run Unit Tests (Local vs Docker) + +Tests are run from the command line using Go Test. + +* **RUN TESTS LOCALLY** + +1. From your existing unit test directory, execute your tests by running the following command: + + ```bash + go test + ``` + +* **RUN TESTS FROM DOCKER** + +1. pending -1. Pending +### **Step 7:** Write Integration Tests -### **Step 6:** Run Unit Tests +When running integration tests in Cobalt, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to wire-up [TerraTest](https://github.com/gruntwork-io/terratest) and [Go Test](https://golang.org/pkg/testing/) for test automation. -1. Pending +1. Run Terraform Apply against a dev workspace environment -### **Step 7:** Write Integration Tests + Integration tests compare the existing output of a CIT's workspace state against expected values provided in your tests. The integration test **does not** run `terraform apply` against your CIT to create infrastructure. These tests assume that the infrastructure already exists. + + 1. Setup Local Environment Variables + + 2. See step 3 of the [quick start guide](./2_QUICK_START_GUIDE.md) for guidance on how to setup your environment variables. + + 3. Initialize the default Terraform Remote Workspace + + 4. See step 4 of the [quick start guide](./2_QUICK_START_GUIDE.md) for guidance on how to initalize a Terraform remote workspace. + + 5. From the az-function-hw directory, execute the following commands to run a template and execute a deployment. + + ```bash + # Select your existing dev workspace + terraform workspace select az-function-hw-$USER + # Ensure that the current workspace is az-function-hw-$USER. + terraform workspace show + # See what terraform will try to deploy without actually deploying. + terraform plan + # Run Azure Function Hello World CIT to execute a deployment. + terraform apply + ``` + +3. Initialize test harness variables for dev isolation + + 1. Create directory called `integration` within your test's sub-directory. + 2. From within `integration` sub-directory , create a file named `az_function_hw_test`. This file will be your integration script entry point. + 3. Copy the following into your script: + + pending + +4. Add your Terraform Apply based output assertions + + pending + +### **Step 8:** Run Integration Tests (Local vs Docker) + +* **RUN TESTS LOCALLY** + +1. Integration tests must be run from the CIT's main directory, **not the test sub-directory**. This allows the integration test to run against the proper workspace context. + + ```bash + go test ./tests/integration + ``` -When running integration tests in Cobalt, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to effectively use [TerraTest](https://github.com/gruntwork-io/terratest) in order to test CITs. This will in turn reduce the effort required to write robust integrations tests for CITs in Cobalt. +* **RUN TESTS FROM DOCKER** -### **Step 8:** Run Integration Tests +1. pending \ No newline at end of file From 2c51d3b762be79ca17040ed750996a63af78b739 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Tue, 19 Nov 2019 14:38:50 -0600 Subject: [PATCH 14/16] testing a template - recontextualize docs to build test on top of az-function-hw cit --- docs/4_TEMPLATE_TESTING.md | 216 ++++++++++++++++++++++++++----------- 1 file changed, 153 insertions(+), 63 deletions(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index cd839a56..a7b17dc6 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -4,15 +4,17 @@ As software developers, we seek out opportunities to improve and grow projects via code contributions. Code contributions, whether major or gradual, require changes that unfortunately bring with them a chance of unexpected behavior to occur. Fortunately, automated testing grants us a chance to gain more predictability and control over the changes that occur from our contributions. By running pre-existing automated tests every time new code changes are introduced, we can catch problems before they are deliverd into a production environment. -With that being said, the task of building automated tests for infrastructure can feel a bit different then the kinds of tests you are used to. For this project, let's further build on what you've learned so far by introducing the approach Cobalt has taken to infrastructure testing and how that will impact what you know about the Cobalt Developer Workflow: +With that being said, the task of building automated tests for infrastructure can feel a bit different then the kinds of tests you are used to. For this project, let's further build on what you've learned so far by introducing the approach Cobalt has taken to infrastructure testing and how that will impact what you know about the Cobalt Developer Workflow. Happy testing! 😄 > *Have yet to create a Cobalt Infrastructure Template or CIT (/kɪt/) of your own? Design and create your first infrastructure template with Cobalt by completing our [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) walkthrough.* ## 4.2 Goals and Objectives -🔲 Understand how testing fits into the SDLC for writing *CIT*s +🔲 Design and build automated tests for the az-function-hw CIT -🔲 Avoid regressions with robust unit and integration tests +🔲 Demonstrate how testing fits into the SDLC for writing *CIT*s + +🔲 Demonstrate how to avoid regressions with robust unit and integration tests 🔲 Feel confident in moving forward to our next recommended section: *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* @@ -21,13 +23,12 @@ With that being said, the task of building automated tests for infrastructure ca | Prereqs | Description | |----------|--------------| | [Quickstart Guide](./2_QUICK_START_GUIDE.md) | This should have served as your first Cobalt Infrastructure deployment. | -| [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | Completing this prequisite leaves you with a CIT and prior knowledge needed for this walkthrough. | +| [Cobalt Templating From Scratch](./3_NEW_TEMPLATE.md) | Completing this prequisite leaves you with a CIT that you can test in this walkthrough. | | [Terraform Modules](https://www.terraform.io/docs/configuration/modules.html) | An introductory understanding of Terraform modules.| | [Golang](https://golang.org/dl/) (1.12.5 +) | Our testing strategy depends on Golang. Install it and gain an introductory understanding. | | [Go Modules](https://blog.golang.org/using-go-modules) | An introductory understanding of Golang's latest dependency management system. | | [Go Test](https://golang.org/pkg/testing/) | An introductory understanding of Golang's native testing package. | | [TerraTest](https://github.com/gruntwork-io/terratest) | An introductory understanding of TerraTest. | -| [Docker](https://docs.docker.com/install/) (18.09 +) | A secondary option for running our tests. An introductory understanding of Docker. | ## 4.4 Walkthrough - Testing a Cobalt Infrastructure Template (CIT) @@ -43,25 +44,25 @@ You'll want to make sure that your tests are using actual test values before pro > **NOTE:** Due to the beginning of the Cobalt Developer Workflow having a hard dependency on a remote back-end state file, both unit testing and integration testing will need these environment variables. Therefore, we highly recommend completing this step. -1. Prepare your environment variables. +1. Prepare your environment variables for dev - Ensure that your .env file is using non-production values as you will be referencing these when writing your automated tests in Golang. These values are usually specific to the provider your CIT is targeting. Here are a few examples: + See step 3 of the [quick start guide](./2_QUICK_START_GUIDE.md) for guidance on how to setup your environment variables. Ensure that your .env file is using non-production values as you will be referencing these when writing your automated tests for the az-function-hw CIT in Golang. These values are specific to the provider your CIT is targeting. In this case, it's Azure. Here are a few examples: | Env Variables | Test Value | Description | |--------|----------|-----------| | `storage_account_name` | Value from "TF_VAR_remote_state_account" | This value should be the storage account dedicated to the dev environment. The value lives within your .env file. | | `container_name` | Value from "TF_VAR_remote_state_container" | This value will be the dedicated remote container that will hold multiple Terraform backend workspace state files. The value lives within your .env file. | -2. Locate the type of input variables from your CIT that further help you achieve test isolation and prepare values for them. +2. Locate the input variables from your CIT that further help you achieve test isolation then prepare values for them. > **NOTE:** Some CITs rely on a commons.tf file that resolves input names into unique names for cloud infrastructure resources like fqdns. The commons.tf file is an answer to the inherit naming constraints that come with some cloud resources. Therefore, some of the names you provide as inputs for your CIT will be further sanitized by the commons.tf file. Another feature of this file is that it improves work isolation. - Here are all the input variables and values we used to help further prepare the az-hello-world CIT for test isolation in Golang: + Here are all the input variables and values you'll use to help further prepare the az-function-hw CIT for test isolation in Golang: | Input Var Name | Test Value | Description | |--------|----------|-----------| - | `workspace` | "az-hello-world-" + guid | Teams share cloud provider resources and accounts. A workspace name with a random guid may be your only solution to working in isolation from other devs on your team. | - | `name` | "az-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests that create actual infrastructure.| + | `workspace` | "az-func-hw-" + guid | Teams share cloud provider resources and accounts. A workspace name with a random guid may be your only solution to working in isolation from other devs on your team. | + | `name` | "az-func-hw-unit-tst-" + guid | A prefix name for appending unique values to resources that require a unique name. Helpful for integration tests that create actual infrastructure.| | `resource_group_location` | "eastus" | The geo-location of the Azure datacenters inside of which you want the resource group that will contain your dev infrastructure to live. | ### **Step 2:** Develop Your Terraform Plan Based Assertions (Unit Testing) @@ -78,25 +79,25 @@ The previous step properly prepared you for test isolation. Now you can concern > **TIP:** A good rule of thumb is, "values visible at plan time are unit testable, values not visible require integration testing.". Make full use of your unit tests because integration tests require spinning up and tearing down real infrastructure and that takes time. - * Run `terraform plan` and inspect the Terraform Plan to formulate test assertions. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. Here are a few examples of test assertions we formulated for our az-hello-world CIT's unit tests: + * Run `terraform plan` and inspect the Terraform Plan to formulate test assertions. It helps to consider each of the common properties that make up modules and their possible impact on the resource addresses that make up the plan. Here are the test assertions we formulated your az-function-hw CIT's unit tests. Use the following: | Terraform Plan Resource Address | Module Property Type | Test Assertion | |--------|-----------|-----------| - | `module.app_service.azurerm_app_service.appsvc[0]` | module | Assert the presence of a docker image in the app service module's configuration. | - | `module.service_plan.azurerm_app_service_plan.svcplan` | module | Assert that the service plan module for the az-hello-world CIT is configured for the least expensive S1 tier. | - | `azurerm_resource_group.main` | resource | Assert the resource group contains the datacenter used for dev environments. | + | `module.function_app.azurerm_function_app.walkthrough` | module | Assert that the value "az-function-hw-walkthrough" is present in the azure function module's configuration. | + | `module.service_plan.azurerm_app_service_plan.svcplan` | module | Assert that the service plan module is configured as a "FunctionApp" and that the Function App sku is "Dynamic". | ### **Step 3:** Develop Your Terraform Apply Based Assertions (Integration Tests) In the previous step we stated, "Values visible at plan time are unit testable, values not visible require integration testing.". After completing the exercise of developing Terraform _Plan_ based assertions, it follows that we must also develop Terraform _Apply_ based assertions. These are test assertions about values that are not resolvable until the `terraform apply` command finishes deploying real infrastructure. -* Make unresolvable Terraform Plan values testable - Visit [Terraform Outputs](https://www.terraform.io/docs/configuration/outputs.html) to learn more. +* Make unresolvable Terraform Plan values testable. - Visit [Terraform Outputs](https://www.terraform.io/docs/configuration/outputs.html) to learn more. - You'll have to map unresolvable Terraform Plan values to CIT outputs. Outputs are simply return values for modules, therefore, your CIT also has return values. These outputs are visible in standard out when the `terraform apply` command has finished executing. Reconfigure your CIT by taking unresolvable properties that you care about from your Terraform Plan and mapping them to outputs configured in your CIT. Here's an example of a test assertion we formulated for our az-hello-world CIT's integration tests: + You'll have to map unresolvable Terraform Plan values to CIT outputs. Outputs are simply return values for modules, therefore, your CIT can also have return values. These outputs are visible in standard out when the `terraform apply` command has finished executing. Reconfigure your CIT by taking unresolvable properties that you care about from your Terraform Plan and mapping them to outputs configured in your CIT. Here's an example of an integration test assertion we formulated for your az-function-hw CIT: | Unresolvable Terraform Plan Value | Output Var Name | Planned Assertion | |--------|-----------|-----------| - | `module.app_service.app_service_uris` | `app_service_default_hostname` | Assert that the app service module's app service url (A value that is mapped to the CIT's output.) returns a status of 200. | + | `module.app_service.azure_function_url` | `azure_function_default_hostname` | Assert that the azure function module's app service url (A value that is mapped to the CIT's output.) returns a status of 200. | + ### **Step 4:** Choose Testing Frameworks (Terraform and Golang) @@ -106,17 +107,44 @@ It is our opinion that the job of the testing framework you ultimately choose sh When writing unit tests for Cobalt CITs, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to wire-up [TerraTest](https://github.com/gruntwork-io/terratest) and [Go Test](https://golang.org/pkg/testing/) for test automation. -The test harness implementation automates the command line execution of the Cobalt Developer Workflow (i.e. *create/choose/extend* a template/module ---> init ---> select workspace ---> plan ---> **test** ---> apply ---> **test** ---> destroy). The test harness is packaged and is exposed in way that allows you to provide your custom testing context as inputs to the harness. +The test harness implementation partially automates the command line execution of the Cobalt Developer Workflow (i.e. *create/choose/extend* a template/module ---> init ---> select workspace ---> plan ---> **test** ---> destroy). The test harness is packaged and is exposed in way that allows you to provide your custom testing context as inputs to the harness. + +1. Navigate to the az-function-hw CIT directory (i.e. ./infra/templates/az-function-hw) and execute the following commands to wire up your new test repository: + + ```bash + # Create a directory called "tests" + mkdir -p ./tests + # Navigate to the tests directory + cd tests + # Create a directory called "unit" + mkdir -p ./unit + # Navigate to the unit directory + cd unit + # Create a test file + touch az_function_hw_test.go + ``` -> **NOTE:** Golang compiles to a static library, therefore, it's important that all types are defined ahead of time. With a little familiarity on how the [Golang interface](https://tour.golang.org/methods/14) type behaves, you'll understand how the below examples initialize variables to provide context to the testing harness hooks for running unit tests within the az-hello-world CIT. +1. Initialize the test file and it's test harness variables for dev isolation with the following go code snippet: -1. Initialize testharness variables for dev isolation + The code snippet below initializes a Terratest object called *Options* with the values from an earlier step that we pre-selected for achieving test isolation. This *Options* object act as the first step for setting up the context needed to run unit tests within the test harness. - The code snippet below comes from the unit tests within the az-hello-world CIT. It initializes a Terratest object called *Options* with the values that we preselected for achieving test isolation. This *Options* object acts as the first step for setting up the context needed to run unit tests within the test harness. + > **NOTE:** Golang compiles to a static library, therefore, it's important that all types are defined ahead of time. With a little familiarity on how the [Golang interface](https://tour.golang.org/methods/14) type behaves, you'll understand how the below examples initialize variables to provide context to the testing harness hooks for running unit tests within the az-function-hw CIT. ```go - var workspace = fmt.Sprintf("az-hello-world-%s", random.UniqueId()) - var prefix = fmt.Sprintf("az-hw-unit-tst-%s", random.UniqueId()) + package test + + import ( + "fmt" + "os" + "testing" + + "github.com/gruntwork-io/terratest/modules/random" + "github.com/gruntwork-io/terratest/modules/terraform" + "github.com/microsoft/cobalt/test-harness/infratests" + ) + + var workspace = fmt.Sprintf("az-func-hw-%s", random.UniqueId()) + var prefix = fmt.Sprintf("az-func-hw-unit-tst-%s", random.UniqueId()) var datacenter = "eastus" var tfOptions = &terraform.Options{ @@ -133,72 +161,66 @@ The test harness implementation automates the command line execution of the Coba } ``` -2. Add your Terraform Plan based assertions +2. Add your Terraform Plan based assertions to the test file using the following go code snippet: - The code snippet below also comes from the unit tests within the az-hello-world CIT. It initializes a custom test harness object called *UnitTestFixture* with the values that will be later passed to Terratest and other logic needed for test automation. The *ResourceDescription* object is where you plug in your formulated test assertions as code. The test harness will run assertions for you by parsing the Terraform Plan as long as you provide the expected values and their respective data types. + The code snippet below initializes a custom test harness object called *UnitTestFixture* with the values that will be later passed to Terratest and other logic needed for test automation. The *ResourceDescription* object is where you plug in your formulated test assertions as code. The test harness will run assertions for you by parsing the Terraform Plan as long as you provide the expected values and their respective data types. ```go func TestAzureSimple(t *testing.T) { testFixture := infratests.UnitTestFixture{ GoTest: t, TfOptions: tfOptions, - ExpectedResourceCount: 10, - ... + ExpectedResourceCount: 6, + PlanAssertions: nil, + Workspace: workspace, ExpectedResourceAttributeValues: infratests.ResourceDescription{ - "module.app_service.azurerm_app_service.appsvc[0]": map[string]interface{}{ - ... - "site_config": []interface{}{ - map[string]interface{}{"linux_fx_version": "DOCKER|docker.io/appsvcsample/static-site:latest"}, + "module.function_app.azurerm_function_app.walkthrough": map[string]interface{}{ + "app_settings": map[string]interface{}{ + "environment": "az-function-hw-walkthrough", }, }, "module.service_plan.azurerm_app_service_plan.svcplan": map[string]interface{}{ - ... + "kind": "FunctionApp", "sku": []interface{}{ - map[string]interface{}{"size": "S1", "tier": "Standard"}, + map[string]interface{}{"size": "Y1", "tier": "Dynamic"}, }, }, - "azurerm_resource_group.main": map[string]interface{}{ - "location": datacenter, - }, }, } + infratests.RunUnitTests(&testFixture) } ``` -### **Step 6:** Run Unit Tests (Local vs Docker) +### **Step 6:** Run Unit Tests -Tests are run from the command line using Go Test. +Tests are run from the command line using Golang's testing framework Go Test. * **RUN TESTS LOCALLY** -1. From your existing unit test directory, execute your tests by running the following command: +1. From your existing unit test directory, execute your tests by running the following go command: ```bash go test ``` -* **RUN TESTS FROM DOCKER** - -1. pending - ### **Step 7:** Write Integration Tests When running integration tests in Cobalt, we suggest coding against our provided [test harness](./../test-harness/README.md). Our test harness minimizes the boiler plate code required to wire-up [TerraTest](https://github.com/gruntwork-io/terratest) and [Go Test](https://golang.org/pkg/testing/) for test automation. 1. Run Terraform Apply against a dev workspace environment - Integration tests compare the existing output of a CIT's workspace state against expected values provided in your tests. The integration test **does not** run `terraform apply` against your CIT to create infrastructure. These tests assume that the infrastructure already exists. + The test harness integration test automates the comparison of a CIT's resolved outputs against expected values that you provide in your tests. The integration test **does not** run `terraform apply` against your CIT to create infrastructure. This is a step you must do yourself in order for your CIT to have resolved outputs. These outputs are written to the CIT's workspace state. - 1. Setup Local Environment Variables + 1. Setup Environment Variables for your dev workspace - 2. See step 3 of the [quick start guide](./2_QUICK_START_GUIDE.md) for guidance on how to setup your environment variables. + 1. See step 3 of the [quick start guide](./2_QUICK_START_GUIDE.md) for guidance on how to setup your environment variables. - 3. Initialize the default Terraform Remote Workspace + 1. Initialize the default Terraform Remote Workspace - 4. See step 4 of the [quick start guide](./2_QUICK_START_GUIDE.md) for guidance on how to initalize a Terraform remote workspace. + 1. See step 4 of the [quick start guide](./2_QUICK_START_GUIDE.md) for guidance on how to initalize a Terraform remote workspace. - 5. From the az-function-hw directory, execute the following commands to run a template and execute a deployment. + 1. From the az-function-hw directory, execute the following commands to run a template and execute a deployment. ```bash # Select your existing dev workspace @@ -211,28 +233,96 @@ When running integration tests in Cobalt, we suggest coding against our provided terraform apply ``` -3. Initialize test harness variables for dev isolation +2. Navigate to the az-function-hw tests directory (i.e. ./infra/templates/az-function-hw/tests) and execute the following commands to wire up your new integration test repository: + + ```bash + # Create a directory called "integration" + mkdir -p ./integration + # Navigate to the integration directory + cd integration + # Create a test file + touch az_function_hw_test.go + ``` + +3. Initialize the test file and it's test harness variables for dev isolation with the following go code snippet: - 1. Create directory called `integration` within your test's sub-directory. - 2. From within `integration` sub-directory , create a file named `az_function_hw_test`. This file will be your integration script entry point. - 3. Copy the following into your script: + ```go + package test - pending + import ( + "os" + "strings" + "testing" + "time" + + httpClient "github.com/gruntwork-io/terratest/modules/http-helper" + "github.com/gruntwork-io/terratest/modules/terraform" + "github.com/microsoft/cobalt/test-harness/infratests" + ) + + var tfOptions = &terraform.Options{ + TerraformDir: "../../", + Upgrade: true, + BackendConfig: map[string]interface{}{ + "storage_account_name": os.Getenv("TF_VAR_remote_state_account"), + "container_name": os.Getenv("TF_VAR_remote_state_container"), + }, + } + ``` 4. Add your Terraform Apply based output assertions - pending + The code snippet below initializes a custom test harness object called *IntegrationTestFixture* with the context you provide for the Terratest framework. The *TfOutputAssertions* object is where you plug in your formulated test assertions as code. In this example, the test assertion is in the form of a function pointer that will make an http call when executed. The test harness will execute this function for you. -### **Step 8:** Run Integration Tests (Local vs Docker) + ```go + // Validates that the service responds with HTTP 200 status code. A retry strategy + // is used because it may take some time for the application to finish standing up. + func httpGetRespondsWith200(goTest *testing.T, output infratests.TerraformOutput) { + hostname := output["azure_function_default_hostname"].(string) + maxRetries := 20 + timeBetweenRetries := 2 * time.Second + expectedResponse := "Hello App Service!" + + err := httpClient.HttpGetWithRetryWithCustomValidationE( + goTest, + hostname, + maxRetries, + timeBetweenRetries, + func(status int, content string) bool { + return status == 200 && strings.Contains(content, expectedResponse) + }, + ) + if err != nil { + goTest.Fatal(err) + } + } + + func TestAzureSimple(t *testing.T) { + testFixture := infratests.IntegrationTestFixture{ + GoTest: t, + TfOptions: tfOptions, + ExpectedTfOutputCount: 1, + TfOutputAssertions: []infratests.TerraformOutputValidation{ + httpGetRespondsWith200, + }, + } + infratests.RunIntegrationTests(&testFixture) + } + ``` + +### **Step 8:** Run Integration Tests * **RUN TESTS LOCALLY** -1. Integration tests must be run from the CIT's main directory, **not the test sub-directory**. This allows the integration test to run against the proper workspace context. +1. Navigate back to the az-function-hw directory (i.e. ./infra/templates/az-function-hw) and run the following commands to execute your integration tests: + + > **NOTE:** Integration tests must be run from the CIT's main directory, **not the test sub-directory**. This allows the integration test to run against the proper workspace context. ```bash + # Select your existing dev workspace + terraform workspace select az-function-hw-$USER + # Ensure that the current workspace is az-function-hw-$USER. + terraform workspace show + # Use Golang to execute your integration test go test ./tests/integration - ``` - -* **RUN TESTS FROM DOCKER** - -1. pending \ No newline at end of file + ``` \ No newline at end of file From fde2ee65cbde65773c81b567d134286054752b22 Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Mon, 9 Dec 2019 16:28:21 -0600 Subject: [PATCH 15/16] docs/4_TEMPLATE_TESTING.md --- docs/4_TEMPLATE_TESTING.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index a7b17dc6..1934df51 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -325,4 +325,10 @@ When running integration tests in Cobalt, we suggest coding against our provided terraform workspace show # Use Golang to execute your integration test go test ./tests/integration - ``` \ No newline at end of file + ``` + +## Conclusion + +Completion of this walkthrough means that you have used the az-function-hw CIT to fully exercise the Cobalt Developer Workflow (i.e. create/choose a template ---> init ---> select workspace ---> plan ---> test ---> apply ---> test ---> destroy). Please continue to the recommended next step for guidance on operationalizing a CIT. + +### **Recommended Next Step:** *[Operationalizing CITs - A CICD Approach](./5_OPERATIONALIZE_TEMPLATE.md)* \ No newline at end of file From d3a6610cde033b1ddf5dface7109abb54fd47fec Mon Sep 17 00:00:00 2001 From: Dexter Williams Date: Fri, 13 Dec 2019 12:36:29 -0600 Subject: [PATCH 16/16] add notes to prepare user for running go test --- docs/4_TEMPLATE_TESTING.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/4_TEMPLATE_TESTING.md b/docs/4_TEMPLATE_TESTING.md index 1934df51..99f17828 100644 --- a/docs/4_TEMPLATE_TESTING.md +++ b/docs/4_TEMPLATE_TESTING.md @@ -198,9 +198,19 @@ Tests are run from the command line using Golang's testing framework Go Test. * **RUN TESTS LOCALLY** -1. From your existing unit test directory, execute your tests by running the following go command: +1. Ensure that the go.mod file is present in the root directory of the project. If it is not, run the following commands from the root directory ```bash + # creates default go dependencies + go mod init github.com/microsoft/cobalt + # crawls the directory for more go dependencies to be added to the go.mod file + go build ./... + ``` + +2. From your existing unit test directory, execute your tests by running the following go command: + + ```bash + # Run this command from the unit test directory of the template go test ```