Skip to content

Commit

Permalink
DEV-628 - Add a new page discussing developer self service (#2209)
Browse files Browse the repository at this point in the history
* DEV-628 - Add a new page discussing developer self service

* missing file

* more placeholders

* Fill in more self service documentation

* Remaining todos on dev self service

---------

Co-authored-by: Lewis Christie <[email protected]>
  • Loading branch information
ZachGoldberg and Resonance1584 authored Nov 23, 2024
1 parent d58dc63 commit f6c69fc
Show file tree
Hide file tree
Showing 10 changed files with 109 additions and 22 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The following options are particularly relevant for delegated repositories and y

#### Catalog Repositories

Catalog Repositories are what your developer teams will use when running `terragrunt catalog` within their delegated repository. This defaults to your `infrastructure-modules` repository but you can customize this list and the values will be vended into the newly created repository.
Catalog Repositories are what your developer teams will use when running `terragrunt catalog` within their delegated repository. This defaults to your `infrastructure-catalog` repository but you can customize this list and the values will be vended into the newly created repository.

[catalog-repositories](/2.0/reference/accountfactory/configurations#catalog-repositories)

Expand Down
14 changes: 8 additions & 6 deletions docs/2.0/docs/library/guides/integrate-tfc.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,14 @@

This section will cover how to use Gruntwork in conjunction with two popular HashiCorp products: [Terraform Cloud (TFC)](https://www.terraform.io/docs/cloud/index.html) and [Terraform Enterprise (TFE)](https://www.terraform.io/docs/enterprise/index.html). Although the open source edition of Terraform is quite powerful and flexible as a standalone project, many organizations turn to TFC/TFE for the CLI/UI integration, approval-based workflow capabilities, Sentinel policy framework, and more. At its core, Terraform Enterprise is basically Terraform Cloud repackaged for a self-hosted environment. We’ll use "TFC" as short hand for both Terraform Cloud and Enterprise throughout this guide.


In our guide on [Customizing Modules](/2.0/docs/library/tutorials/customizing-modules) we describe how to use Gruntwork with two VCS repositories: `infrastructure-modules`, containing your OpenTofu/Terraform code that wraps the modules from Gruntwork IaC Library, and `infrastructure-live`, containing Terragrunt configurations that enable you to manage Terraform easily across multiple accounts and environments. When using Gruntwork with TFC, you have two choices regarding these repositories:

<div className="dlist">

#### Use TFC without Terragrunt

Using this approach, Terraform modules are still defined in an `infrastructure-modules` repository as discussed above.
Using this approach, Terraform modules are still defined in an `infrastructure-catalog` repository as discussed above.
Each module has a dedicated [TFC workspace](https://www.terraform.io/docs/state/workspaces.html). You create the
workspace in advance, and then you can run the Terraform modules either from the `terraform` CLI or from the TFC UI.
In essence, TFC replaces Terragrunt and `infrastructure-live`. You’ll be able to use the full TFC feature set, but you
Expand All @@ -19,8 +20,8 @@ args, etc DRY, and adding support for applying changes across multiple modules /

#### Use TFC with Terragrunt

Alternatively, you can use both `infrastructure-modules` and `infrastructure-live` repositories as described above,
storing the wrapper modules in `infrastructure-modules`, and using `infrastructure-live` and Terragrunt for
Alternatively, you can use both `infrastructure-catalog` and `infrastructure-live` repositories as described above,
storing the wrapper modules in `infrastructure-catalog`, and using `infrastructure-live` and Terragrunt for
deployments. In this approach, TFC is used as a [remote backend](https://www.terraform.io/docs/backends/types/remote.html)
for Terraform. You use Terragrunt to run deployments from the CLI, which in turn invokes Terraform on the TFC backend.
The TFC UI is used for audit and tracking capabilities, but not for executing Terraform runs.
Expand Down Expand Up @@ -97,7 +98,7 @@ build an end-to-end infrastructure.

In our guide on [using Gruntwork modules](/2.0/docs/library/tutorials/customizing-modules), we discuss the wrapper module
pattern in which multiple Terraform modules are contained in a hierarchy of directories located under
`infrastructure-modules/modules`. Using such a hierarchy, each workspace will use the same `infrastructure-modules` repository, but pointed at different subdirectories within the repository.
`infrastructure-catalog/modules`. Using such a hierarchy, each workspace will use the same `infrastructure-catalog` repository, but pointed at different subdirectories within the repository.

:::note

Expand All @@ -111,7 +112,7 @@ We’ll demonstrate how to set up a workspace for a simple SQS module. To get st

### Connect to a version control provider

Connect the workspace to the version control system of your choice. For example, GitHub, GitLab, or Bitbucket. This allows TFC to access your `infrastructure-modules` repository. Once your VCS is connected, select your `infrastructure-modules` repository from the list of repositories presented.
Connect the workspace to the version control system of your choice. For example, GitHub, GitLab, or Bitbucket. This allows TFC to access your `infrastructure-catalog` repository. Once your VCS is connected, select your `infrastructure-catalog` repository from the list of repositories presented.

![Connect a workspace to a VCS](/img/guides/working-with-code/tfc/tfc-create-workspace.png)

Expand Down Expand Up @@ -150,7 +151,7 @@ With all the configuration complete, it’s time to kick off the plan and apply.
commit to a file in the working directory that you set up when configuring workspace settings (in our case, in
`/modules/networking/sqs`), or by manually triggering the run using the _Queue plan_ button in the TFC UI. The run will:

- Clone your `infrastructure-modules` repository using the VCS connection
- Clone your `infrastructure-catalog` repository using the VCS connection
- Download the AWS provider and set credentials using the environment variables
- Download the Gruntwork SQS module using the SSH key
- Run a `terraform plan`
Expand All @@ -162,6 +163,7 @@ commit to a file in the working directory that you set up when configuring works
### Final thoughts on integrating TFC with the Gruntwork IaC Library

It’s easy to use TFC with the Gruntwork IaC Library. When using the `infrastructure-modules` approach outlined in this

guide, all of your Terraform wrapper modules will be in one place. You can configure one workspace per module, and you
can link modules together with the [`remote_state`
data source](https://www.terraform.io/docs/providers/terraform/d/remote_state.html). Note that you’ll need to set up the AWS credentials and SSH key within each workspace.
Expand Down
2 changes: 1 addition & 1 deletion docs/2.0/docs/library/tutorials/customizing-modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ In some instances, you may need to extend a Gruntwork module, set default values

For example, if you were creating a wrapper module for an AWS Lambda function, your repository would have a file structure like the following:
```
infrastructure-modules
infrastructure-catalog
└ lambda
└ main.tf
└ outputs.tf
Expand Down
79 changes: 79 additions & 0 deletions docs/2.0/docs/overview/concepts/developer-self-service.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Developer Self Service

Developer Self-Service lets developers deploy and manage their apps and infrastructure themselves, without always needing help from the platform team.

Effective developer self-service then requires a set of patterns, practices, rules, constraints and governance pre-baked into the modules that engineers use to represent their infrastructure. Some teams call these modules "approved blueprints" or "blessed modules". At Gruntwork we generally refer to this as the **Infrastructure Catalog** that lives in a repository we call `infrastructure-catalog` (previously `infrastructure-modules`).

Inside the `infrastructure-catalog` we recommend following the [infrastructure-live pattern](https://docs.gruntwork.io/2.0/docs/overview/concepts/infrastructure-live#separating-modules-from-live-infrastructure) and storing catalog module code in a folder called `modules`.

## Why self-service

### Benefits to Developers

Developers benefit by avoiding "reinventing the wheel". They're able to leverage pre-built, tested, and secure modules that have already been approved for use at your organization. Handoffs between platform and development teams are reduced resulting in faster delivery and fewer bottlenecks.

### Benefits to Platform Teams

Platform teams are able to operate effectively at scale. They retain centralized control of reusable modules, and are able to focus on standardizing testing, security, compliance, and governance. The day to day workload of assisting Developers with infrastructure code is reduced, while retaining oversight.

## Terragrunt Catalog

Terragrunt has [native support](https://terragrunt.gruntwork.io/docs/features/catalog/) for an interface that allows developers to browse modules using a terminal based UI, and then [scaffold](https://terragrunt.gruntwork.io/docs/features/scaffold/) out new modules using [boilerplate](https://github.com/gruntwork-io/boilerplate). All repositories vended as part of Devops Foundations include a `catalog` configuration in the root `terragrunt.hcl` file that points to a starter `infrastructure-catalog` repository that provides examples on how to further build out the catalog.

### Using Catalog

Within a DevOps Foundations repository, create a new directory to contain your new terragrunt unit, then navigate to this directory.

Running `terragrunt catalog` will open an interactive terminal UI allowing you to browse the available units from your `infrastructure-catalog`.

When browsing your catalog, you can select a unit, and hit **S** to scaffold it out.
The scaffold command does the following automatically:

- Download and template the unit into the current directory.

- Figures out the module URL and the latest version (tag) available, and fills that into the source URL.

- Parse all of the module’s input variables and generate placeholders in the inputs = { } block to make it easy to fill those variables in.

Once you've replaced these placeholders, commit the new unit, and push it to a new Pull Request for [Gruntwork Pipelines](/2.0/docs/pipelines/concepts/overview) to being planning the changes.

## Self-Service Best Practices

### Versioning

Implement a consistent versioning strategy, such as [semantic versioning](https://semver.org/), for all modules in your infrastructure catalog. This allows you to:

* **Track Changes:** Maintain a clear history of module updates and modifications.
* **Enable Rollbacks:** Revert to previous versions in case of issues or errors.
* **Ensure Clarity:** Use semantic versioning (e.g., v1.2.3) to communicate the nature of changes (major, minor, patch).

Use [Gruntwork Patcher](/2.0/docs/patcher/concepts/) to efficiently roll out breaking changes across your infrastructure and minimize disruption.

### Infrastructure Tagging and Labeling

A central catalog allows core teams to implement a consistent tagging and labeling strategy across all infrastructure components. This enables:

* **Cost Tracking and Allocation:** Assign costs to specific teams or projects based on tags.
* **Access Control:** Use tags to manage and control access to resources.
* **Automation:** Automate tasks like provisioning, termination, and reporting based on tags.
* **Monitoring and Alerting:** Filter and group resources for monitoring and alerting purposes.


### Wrapper Modules / Services

Create wrapper modules or services to simplify complex deployments and configurations. These wrappers can:

* **Encapsulate Best Practices:** Bundle security, compliance, and operational best practices into reusable units.
* **Reduce Boilerplate:** Minimize repetitive code and configuration for developers.
* **Standardize Deployments:** Ensure consistency and reduce errors across different deployments.
* **Abstract Complexity:** Hide the underlying complexity of infrastructure from developers, allowing them to focus on application logic.


### Using Stacks to keep developer code DRY

Organize infrastructure into stacks to promote code reusability and maintainability. Terragrunt Stacks allow you to:

* **Group Related Resources:** Manage interconnected units as a single entity.
* **Reduce Code Duplication:** Avoid repeating the same code across different environments or projects.
* **Promote Modularity:** Break down complex infrastructure into smaller, manageable components.
* **Simplify Management:** Deploy, update, and destroy entire stacks with ease.
10 changes: 5 additions & 5 deletions docs/2.0/docs/pipelines/architecture/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,25 +78,25 @@ It also doesn't have to be the only access control repository in your organizati

While the permissions for this workflow are largely read-only when proposing access control changes, the workflow also has the ability to make changes to relevant access control infrastructure when the changes are merged.

## `infrastructure-modules`
## `infrastructure-catalog`

The `infrastructure-modules` repository is used to store OpenTofu/Terraform modules that your organization has authored and intends to use within your organization. It's provisioned when the "Bootstrap the infrastructure-modules repository" checkbox is checked when running the Bootstrap workflow in the `infrastructure-live-root` repository.
The `infrastructure-catalog` repository is used to store OpenTofu/Terraform modules that your organization has authored and intends to use within your organization. It's provisioned when the "Bootstrap the infrastructure-catalog repository" checkbox is checked when running the Bootstrap workflow in the `infrastructure-live-root` repository.

This repository is optional, but is recommended for all customers. It's where you'll store reusable infrastructure code that can be shared across your organization.

A common pattern used by customers is to leverage the Gruntwork IaC library whenever possible, and to author custom modules here if there's ever a need to extend or modify the behavior of Gruntwork modules.

:::tip

The `infrastructure-modules` repository does not have to be named "infrastructure-modules". You can customize the name during the bootstrap process. It is highly recommended that the repository is named something similar to `infrastructure-modules` to make it clear what it is when reading Gruntwork documentation, however.
The `infrastructure-catalog` repository does not have to be named "infrastructure-catalog". You can customize the name during the bootstrap process. It is highly recommended that the repository is named something similar to `infrastructure-catalog` to make it clear what it is when reading Gruntwork documentation, however.

It also doesn't have to be the only modules repository in your organization. You can have multiple modules repositories if you have a complex organization structure that requires it. Make sure to evaluate the trade-offs of having multiple modules repositories before doing so. It can be advantageous to have one repository for all modules to make it easier to find and share modules across your organization. However, it can also be advantageous to have multiple repositories if you have different teams that need to manage their own modules, or want to have different modules available to different teams within your organization.

:::

### Workflows

- **Tests** - This workflow will be used by the `infrastructure-modules` repository to run tests on the modules in the repository. This workflow is typically run on every push to ensure that the modules are functioning as expected by provisioning them in real environments, running [Terratests](https://github.com/gruntwork-io/terratest) against them, then tearing them down.
- **Tests** - This workflow will be used by the `infrastructure-catalog` repository to run tests on the modules in the repository. This workflow is typically run on every push to ensure that the modules are functioning as expected by provisioning them in real environments, running [Terratests](https://github.com/gruntwork-io/terratest) against them, then tearing them down.

The configurations for this workflow are configured out of the box to run tests on the examples written in the repository.

Expand Down Expand Up @@ -132,4 +132,4 @@ If users opt-in to delegating access control to the `infrastructure-live-access-

If users opt-in to delegating infrastructure management to the `infrastructure-live-delegated` repositories, they can provision the relevant `pipelines-delegated` roles in AWS accounts where they want to delegate infrastructure management using the `infrastructure-live-access-control` repository, then manage infrastructure using those roles in the `infrastructure-live-delegated` repositories.

If users opt-in to creating and managing modules centrally, they can create and manage modules in the `infrastructure-modules` repository. This repository can be used by any repository in the organization to share reusable, vetted modules.
If users opt-in to creating and managing modules centrally, they can create and manage modules in the `infrastructure-catalog` repository. This repository can be used by any repository in the organization to share reusable, vetted modules.
6 changes: 3 additions & 3 deletions docs/2.0/docs/pipelines/installation/addingnewrepo.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ To set up IaC Foundations, we use a pre-configured git repository template that

This template creates an infrastructure-live-root repository with a bootstrap workflow that can be run to create scaffolding for a best practices Terragrunt configuration, including patterns for module defaults, global variables, and account baselines. It also configures Gruntwork Pipelines, which is easy to remove if you don't want it.

The workflow also optionally creates and scaffolds your `infrastructure-live-access-control` and `infrastructure-modules` repositories.
The workflow also optionally creates and scaffolds your `infrastructure-live-access-control` and `infrastructure-catalog` repositories.

Navigate to the template repository and select **Use this template** -> **Create a new Repository**. This will initiate repository creation. You should select your org as the owner, add a description if you like, make sure you are creating a **private** repo, and click **Create repository**.

Expand All @@ -31,15 +31,15 @@ Return to your `infrastructure-live-root` repository and follow the instructions

## Run The Workflow

Follow the instructions in your `infrastructure-live-root` to run the Bootstrap workflow. Gruntwork is available to assist with questions around other patterns as they arise. When running the workflow you can select options to create `infrastructure-live-access-control` and `infrastructure-modules` repositories. These will be created in your GitHub organization using values defined in the workflow file.
Follow the instructions in your `infrastructure-live-root` to run the Bootstrap workflow. Gruntwork is available to assist with questions around other patterns as they arise. When running the workflow you can select options to create `infrastructure-live-access-control` and `infrastructure-catalog` repositories. These will be created in your GitHub organization using values defined in the workflow file.

### Infrastructure Live Access Control

This repository is only necessary for Enterprise customers, but is recommended for all customers. When running the Bootstrap workflow in your `infrastructure-live-root` account, select the option to "Bootstrap the infrastructure-access-control repository".

### Infrastructure Modules

The Bootstrap workflow creates an empty infrastructure-modules repository that will be used to store Terraform/OpenTofu modules that your organization has authored and intends to use within your organization. When running the Bootstrap workflow in your `infrastructure-live-root` account, select the option to "Bootstrap the infrastructure-modules repository".
The Bootstrap workflow creates an empty infrastructure-catalog repository that will be used to store Terraform/OpenTofu modules that your organization has authored and intends to use within your organization. When running the Bootstrap workflow in your `infrastructure-live-root` account, select the option to "Bootstrap the infrastructure-catalog repository".

## Complete Instructions In The Bootstrap Pull Requests

Expand Down
Loading

0 comments on commit f6c69fc

Please sign in to comment.