Skip to content

Commit

Permalink
Content Updates from Edify - First 10 files (#2237)
Browse files Browse the repository at this point in the history 
* Update index.md

* Update logging.md

* Update network-topology.md

* Update opentofu-terraform-compatibility.md

* Update overview.md

* Update module-defaults.md

* Update modules.md

* Update overview.md

* Update be-judicious-with-new-features.md

* Update index.md

* Update index.md

* Update index.md

* Update logging.md

* Update network-topology.md

* Update network-topology.md

* Update network-topology.md

* Update opentofu-terraform-compatibility.md

* Update overview.md

* Update overview.md

* Update overview.md

* Update overview.md

* Update overview.md

* Update be-judicious-with-new-features.md

* Update be-judicious-with-new-features.md

* Update be-judicious-with-new-features.md

* Update module-defaults.md

* Update docs/2.0/docs/accountfactory/architecture/index.md

Co-authored-by: Yousif Akbar <[email protected]>

* Maintain spacing in mermaid diagram defintions

* Update docs/2.0/docs/accountfactory/architecture/logging.md

* Update docs/2.0/docs/accountfactory/architecture/logging.md

* Update docs/2.0/docs/library/architecture/opentofu-terraform-compatibility.md

* Update docs/2.0/docs/library/architecture/opentofu-terraform-compatibility.md

* Update docs/2.0/docs/library/architecture/overview.md

* Update docs/2.0/docs/library/concepts/modules.md

* Update docs/2.0/docs/library/concepts/modules.md

* Update docs/2.0/docs/library/architecture/opentofu-terraform-compatibility.md

Co-authored-by: Yousif Akbar <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yousif Akbar <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yousif Akbar <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yousif Akbar <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yousif Akbar <[email protected]>

* Apply suggestions from code review

Co-authored-by: Yousif Akbar <[email protected]>

---------

Co-authored-by: EdifyContent <[email protected]>
Co-authored-by: Yousif Akbar <[email protected]>
  • Loading branch information
@ZachGoldberg @EdifyContent @yhakbar
3 people authored Dec 16, 2024
1 parent f4b28a9 commit 5b9b958
Show file tree
Hide file tree
Showing 9 changed files with 104 additions and 129 deletions.
31 changes: 15 additions & 16 deletions docs/2.0/docs/accountfactory/architecture/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,37 +4,36 @@

Account Factory builds upon Gruntwork's [AWS Control Tower Multi Account Factory](/reference/modules/terraform-aws-control-tower/control-tower-multi-account-factory/) and Pipelines to provide automated account creation, baselining, and managed IAM policies.

Within your `infrastructure-live-root` account, the `_new-account-requests` directory serves as an input to the Gruntwork Control Tower Module. This module runs within your management account and uses AWS Control Tower to provision new accounts.
In your `infrastructure-live-root` repository, the `_new-account-requests` directory acts as input for the Gruntwork Control Tower Module. This module runs within your management account and uses AWS Control Tower to provision new accounts and manage existing ones.

Each provisioned account is tracked in your `infrastructure-live-root` repository as a new base directory containing Terragrunt units that are automatically tracked by Pipelines.
Pipelines tracks each provisioned account as a new base directory containing Terragrunt units in your `infrastructure-live-root` repository.

![Architecture Overview Diagram](/img/accountfactory/architecture.png)

## Account Vending

Account Vending begins by using the Account Factory Workflow to generate a Pull Request against `infrastructure-live-root` that adds a file to the `_new-account-requests` directory. Pipelines detects these new account requests and begins executing terragrunt plan/apply on this module in the management account.
Account Vending starts when the Account Factory Workflow generates a Pull Request against `infrastructure-live-root`, adding a file to the `_new-account-requests` directory. Pipelines detects these new account requests and runs terragrunt plan/apply commands on the `control-tower-multi-account-factory` unit in the management account.

Once the account has been created Pipelines can begin provisioning resources into the account, this includes the IaC controlled OIDC authentication Pipelines will use to deploy infrastructure changes within the account, and IAM policies used to restrict the scope of changes that Pipelines can deploy.
After creating the account(s), Pipelines provisions resources, including IaC-controlled OIDC authenticated roles, which Pipelines can later use to deploy infrastructure changes within the account, and IAM policies that define the scope of changes Pipelines can deploy.

Once this infrastructure has been added to the repository Pipelines deploys it into the AWS account, and runs account baselines in the logs, security, and shared accounts to finish provisioning the new account.
After adding this infrastructure to the repository, Pipelines deploys the resources into the AWS account and runs account baselines in the logs, security, and shared accounts to complete the provisioning process.

```mermaid
sequenceDiagram
Account Factory Workflow ->> Infra Live Repository: Create account request file;
Infra Live Repository ->> Pipelines: Trigger Account Requested;
Pipelines ->> AWS Control Tower Module: Execute terragrunt to create account
AWS Control Tower Module ->> Pipelines: Account Created
Pipelines ->> Infra Live Repository: Add Account Infrastructure
Infra Live Repository ->> Pipelines: Trigger Account Added
Pipelines ->> Core Accounts: Execute terragrunt to baseline account
Account Factory Workflow ->> Infra Live Repository: Create account request file;
Infra Live Repository ->> Pipelines: Trigger Account Requested;
Pipelines ->> AWS Control Tower Module: Execute terragrunt to create account
AWS Control Tower Module ->> Pipelines: Account Created
Pipelines ->> Infra Live Repository: Add Account Infrastructure
Infra Live Repository ->> Pipelines: Trigger Account Added
Pipelines ->> Core Accounts: Execute terragrunt to baseline account
```

## IAM Roles

Each new account has a set of IAM policies that determine the scope of changes Pipelines can plan/apply within AWS. Pipelines will automatically assume the appropriate roles for each account when changes are detected. Read about the [roles in full here](/2.0/docs/pipelines/architecture/security-controls#roles-provisioned-by-devops-foundations).
Each new account includes IAM policies that define the scope of changes Pipelines can make within AWS. Pipelines automatically assumes the appropriate roles for each account when changes are detected. Read about the [roles in full here](/2.0/docs/pipelines/architecture/security-controls#roles-provisioned-by-devops-foundations).

## Delegated Repositories

Delegated repositories provide additional control over your infrastructure by expanding on the above architecture. When vending delegated repositories new account security baselines are still tracked in your `infrastructure-live-root` repository, however other infrastructure is tracked in a new repository specific to this account(s). New IAM roles are added to your `infrastructure-live-access-control` repository that are inherited by pipelines when deploying infrastructure in the delegated repositories, allowing the central platform team to control what changes can be implemented via Pipelines in the delegated repository.
Delegated repositories expand the architecture of your infrastructure estate management and provide additional access control for your infrastructure. When vending delegated repositories, Pipelines continues tracking new account security baselines in your `infrastructure-live-root` repository, while other infrastructure is tracked in a new repository specific to the account(s). Pipelines inherits new IAM roles from your `infrastructure-live-access-control` repository when deploying infrastructure in delegated repositories. This setup allows the central platform team to control what changes individual teams can make via Pipelines in the delegated repository.

![Delegated Architecture Overview Diagram](/img/accountfactory/delegated-architecture.png)
![Delegated Architecture Overview Diagram](/img/accountfactory/delegated-architecture.png)
36 changes: 18 additions & 18 deletions docs/2.0/docs/accountfactory/architecture/logging.md
View file
Original file line number Diff line number Diff line change
@@ -1,40 +1,40 @@
# Logging

Gruntwork Account Factory sets up [AWS CloudTrail](https://aws.amazon.com/cloudtrail/) for all accounts in your [AWS Organization](https://aws.amazon.com/organizations/). CloudTrail allows you to answer the question of _who_ did _what_ and _when_ in each of your AWS accounts.
Gruntwork Account Factory configures [AWS CloudTrail](https://aws.amazon.com/cloudtrail/) for all accounts in your [AWS Organization](https://aws.amazon.com/organizations/). CloudTrail helps you determine _who_ did _what_ and _when_ in each of your AWS accounts.

## Where you can find logs

AWS CloudTrail is automatically configured to log all operations in your AWS accounts when you use Gruntwork Account Factory. By default, CloudTrail maintains your data for 90 days and is queryable using CloudTrail UI.
Gruntwork Account Factory automatically configures AWS CloudTrail to log all operations in your AWS accounts. By default, CloudTrail maintains your data for 90 days and is queryable using the AWS Console CloudTrail UI.

Account Factory sets up CloudTrail to output all events from all of your AWS accounts to an S3 bucket in your `logs` AWS account with a default rule to expire objects after 1 year. Once logs are in S3, you may set up an additional tool for [querying the logs](#querying-data).
Account Factory sets up CloudTrail to forward all events from all of your AWS accounts to an S3 bucket in your `logs` AWS account with a default rule to expire objects after 1 year. After logs reach S3, you can set up an additional tool for [querying the logs](#querying-data).

### CloudTrail

Logs can be viewed in the CloudTrail UI in each of your AWS accounts. To access the CloudTrail UI, navigate to the AWS Console, search `CloudTrail` in the search bar, select CloudTrail from the search results, then select **Event History** from the left side panel.
The CloudTrail UI in each AWS account provides access to logs. To access the CloudTrail UI, navigate to the AWS Console, search `CloudTrail` in the search bar, select CloudTrail from the search results, and then select **Event History** from the left side panel.

### S3

CloudTrail logs are delivered to S3 approximately every 5 minutes. If you are using an S3 bucket that was created by AWS Control Tower, the bucket will be named `aws-controltower-logs-<logs account id>-<primary Control Tower region>`. At the top level of the bucket is a single prefix with a random id, which contains additional prefixes to distinguish between logs for CloudTrail and AWS Config. CloudTrail logs for each account can be found in the prefix `<random id>/AWSLogs/<random id>/`.
S3 receives CloudTrail logs approximately every 5 minutes. If AWS Control Tower created your S3 bucket, it will be named `aws-controltower-logs-<logs account id>-<primary Control Tower region>`. At the top level of the bucket is a single prefix with a random ID, which contains additional prefixes to distinguish between logs for CloudTrail and AWS Config. Find CloudTrail logs for each account in the prefix `<random id>/AWSLogs/<random id>/`.

For each account, CloudTrail delivers logs to region, year, month, and day specific prefixes in the bucket. For example, logs for an account with the id `123456789012` on September 26th, 2023 in the `us-west-2` region, would be in a prefix named `123456789012/us-west-2/2023/09/26`.
For each account, CloudTrail delivers logs to region, year, month, and day-specific prefixes in the bucket. For example, logs for an account with the id `123456789012` on September 26th, 2023 in the `us-west-2` region would be in a prefix named `123456789012/us-west-2/2023/09/26`.

If you configured your logs bucket while setting up AWS Control Tower, you will need access to the KMS key you created to encrypt the objects to download any objects. See [Logs bucket access](#logs-bucket-access) for more information.
If you configured your logs bucket while setting up AWS Control Tower, you will need access to the KMS key you created to encrypt the objects before you download any objects. See [Logs bucket access](#logs-bucket-access) for more information.

For more information about querying data in S3, see [querying in S3](#querying-in-s3).

## Data access
## Data access

Granting access to the audit logs requires security configurations in both the originating account (e.g., the account in which the events are occurring) and the `logs` account. The originating account contains the CloudTrail trail itself, which should only be viewable by account administrators. The `logs` account contains the AWS S3 bucket that contains synchronized CloudTrail logs from all logs.
Granting access to the audit logs requires security configurations in the originating account (e.g., the account in which the events are occurring) and the `logs` account. The originating account contains the CloudTrail trail itself, which should only be viewable by account administrators. The `logs` account contains the AWS S3 bucket that contains synchronized CloudTrail logs from all logs.

### CloudTrail access

Access to CloudTrail is controlled by AWS IAM policies that are assigned to individual IAM users (not recommended) or IAM roles than can be assumed by users (recommended) in AWS accounts.
Access to CloudTrail is controlled by AWS IAM policies that are assigned to individual IAM users (not recommended) or IAM roles that can be assumed by users (recommended) in AWS accounts.

:::tip
Gruntwork recommends that only those with administrative access to an AWS account have access to view CloudTrail logs, as they contain a record of every single API operation that was performed in the account, which may expose the name or configuration of resources an individual user may otherwise not have access to.
:::

Further, the configuration of CloudTrail trails should be defined as code, with all changes reviewed in a pull request before being applied automatically by [Gruntwork Pipelines](/2.0/docs/pipelines/concepts/overview).
Furthermore, you should define the configuration of CloudTrail trails as code, with all changes reviewed in a pull request before being applied automatically by [Gruntwork Pipelines](/2.0/docs/pipelines/concepts/overview).

See [Identity-based policy examples for AWS CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/security_iam_id-based-policy-examples.html) to learn more about granting access to CloudTrail.

Expand All @@ -48,22 +48,22 @@ Access to the objects containing CloudTrail events in S3 is controlled by IAM po
Gruntwork recommends that only a select group of trusted individuals on your security team have direct access to objects in the S3 bucket. Whenever possible, the data should be accessed by [querying](#querying-data) it using the CloudTrail UI or a query service such as [Amazon Athena](https://aws.amazon.com/athena/).
:::

## Querying data
## Querying data

You can query CloudTrail data in two ways - in the originating account or from the `logs` account. Querying in the originating account is done using the CloudTrail UI and is useful for quick checks that do not require in-depth analysis of usage and trends. If you require support for performing analytics to observe usage and trends, Gruntwork recommends querying the data in the S3 bucket in the `logs` account using a query service like [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html).
You can query CloudTrail data in two ways - in the originating account or from the `logs` account. Querying in the originating account is done using the CloudTrail UI, which is helpful for quick checks that do not require in-depth analysis of usage and trends. If you need support for performing analytics to observe usage and trends, Gruntwork recommends querying the data in the S3 bucket in the `logs` account using a query service like [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html).

### Querying in CloudTrail

CloudTrail supports simple queries based on a pre-set lookup attributes, including the event source, event name, user name, and resource type. A full list of filters can be found in [filtering CloudTrail events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-console.html#filtering-cloudtrail-events). The filters in the CloudTrail allow you to perform coarse grained queries over a single attribute filter and time range and view details on individual events. Using the CloudTrail UI can be a quick way to retrieve a lot of information, such as all the users that have performed a certain API call (e.g., ListBuckets), however it is ineffective when trying analyze data to understand usage patterns across multiple attributes, such as the usage of Gruntwork Pipelines by all users in your GitHub organization.
CloudTrail supports simple queries based on pre-set lookup attributes, including the event source, event name, user name, and resource type. You can find a complete list of filters in [filtering CloudTrail events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-console.html#filtering-cloudtrail-events). The filters in CloudTrail allow you to perform coarse-grained queries over a single attribute filter and time range and view details on individual events. Using the CloudTrail UI can be a quick way to retrieve a lot of information, such as all the users that have performed a specific API call (e.g., ListBuckets). However, it is ineffective when analyzing data to understand usage patterns across multiple attributes, such as the usage of Gruntwork Pipelines by all users in your GitHub organization.

You can also [download events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events-console.html#downloading-events) from CloudTrail in CSV or JSON format and perform more in-depth analysis of events in another system such as a query service or using a script on your local machine.

### Querying in S3

If CloudTrail is configured to output all logs to an S3 bucket, there are two approaches that can be taken to perform queries on the data - downloading the data directly (not recommended) and setting up a query service like [Amazon Athena](https://aws.amazon.com/athena/) to allow for more in-depth analysis of your data (recommended).
If you configure CloudTrail to output all logs to an S3 bucket, you can take two approaches to perform queries on the data - downloading the data directly (not recommended) and setting up a query service like [Amazon Athena](https://aws.amazon.com/athena/) to allow for more in-depth analysis of your data (recommended).

Amazon Athena is a popular choice for a query service because it is directly integrated in the AWS Console. Further, because CloudTrail logs have a known structure and prefix scheme in S3, you can set up [Athena with partition projection](https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html#create-cloudtrail-table-partition-projection), which will automatically create new partitions in Athena, reducing the work required to ensure the data is partitioned for optimal query support. While Athena is recommended because of its convenience, you may use any query service of your choosing to analyze the data, so long as the tool can pull data out of S3. See [example queries](https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html#query-examples-cloudtrail-logs) and [tips for querying CloudTrail logs](https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html#tips-for-querying-cloudtrail-logs) for more information on analyzing CloudTrail data using Athena.
Amazon Athena is a popular query service because it is integrated it into the AWS Console, and lets you perform queries on data in S3 directly. Furthermore, because CloudTrail logs have a known structure and prefix scheme in S3, you can set up [Athena with partition projection](https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html#create-cloudtrail-table-partition-projection), which will automatically create new partitions in Athena, reducing the work required to ensure data partitioning for optimal query support. We recommend Athena because of its convenience; you can use any query service you choose to analyze the data as long as the tool can pull data out of S3. See [example queries](https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html#query-examples-cloudtrail-logs) and [tips for querying CloudTrail logs](https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html#tips-for-querying-cloudtrail-logs) for more information on analyzing CloudTrail data using Athena.

:::warning
Downloading CloudTrail event data from S3, while possible, is generally not recommended. Finding data requires downloading potentially many objects and writing scripts to parse an analyze them. Once the data is outside of S3, it is not possible to know what analysis is being performed. Query services like AWS Athena or similar allow you to see the history of queries performed and who performed the query.
:::
While it is possible to download CloudTrail event data from S3, it is generally not recommended. Finding data requires downloading potentially many objects and writing scripts to parse and analyze them. Once the data is outside S3, it is impossible to know what analysis is performed. Query services like AWS Athena or similar allow you to see the history of queries performed and who performed the query.
:::
Loading

0 comments on commit 5b9b958

Please sign in to comment.