Skip to content

Conversation

@jameslaneovermind
Copy link
Contributor

No description provided.

This change simulates the AWS DNS outage scenario by:

1. Creating a blackhole target group with no registered targets
2. Updating ALB listener to forward all traffic to the empty target group
3. Adding Route53 DNS record with no failover capability and disabled health checks

This mimics the scenario where DNS endpoint resolves but has no healthy
backends, causing immediate 503 errors and service unavailability. The
change removes failover protection and routes traffic to a target group
with zero healthy targets.

Impact: High - complete service outage, no automatic failover capability
@github-actions
Copy link

github-actions bot commented Oct 31, 2025

Overmind

Open in Overmind ↗


model|risks_v6

🔴 Change Signals

Routine 🔴 ▇▅▃▂▁ AWS Route 53 and LB target group resources showing first ever modifications across multiple attributes, which is unusual compared to typical patterns.
Policies 🔴 ▃▂▁ Multiple S3 buckets and security groups are missing required tags and server-side encryption, and security groups allow SSH access from anywhere, which is unusual compared to typical patterns.

View signals ↗


🔥 Risks

Removing target group from ALB listener default action will stop all routing and cause outage ‼️High Open Risk ↗
The ALB listener arn:aws:elasticloadbalancing:eu-west-2:540044833068:listener/app/scenarios--a3ec77f7-alb/a0268d67b29039c7/af5e943a0e473e91 currently forwards to target group scenarios--a3ec77f7-tg. The proposed change removes the target group ARN from the listener’s default_action while keeping type: forward.

When applied, the listener will have no valid forwarding destination for its default rule, so requests reaching scenarios--a3ec77f7-alb will not be routed to any targets. This will cause immediate traffic loss and application downtime; alarms tied to the target group may no longer reflect live routing conditions while the service is unreachable.

EC2 replacement will delete root EBS volume (DeleteOnTermination=true), causing loss of instance state/data ❗Medium Open Risk ↗
The EC2 instance 540044833068.eu-west-2.ec2-instance.i-02f292e8a0766d313 is being replaced. Its attached root volume 540044833068.eu-west-2.ec2-volume.vol-0bc99e157a38768b6 is configured with DeleteOnTermination=true. When the old instance is terminated during the replacement, AWS will delete this root volume.

Any OS-level changes, logs, or application state stored on the root disk will be lost unless they are baked into the new AMI or otherwise preserved, which can cause service disruption or irrecoverable data loss for workloads that depend on files on the root filesystem.


🟣 Expected Changes

+/- ec2-instance › i-01dcfcab37ba22689
--- current
+++ proposed
@@ -2,72 +2,76 @@
 id: github.com/overmindtech/terraform-example.ec2-instance.module.scenarios[0].aws_instance.app_server
 attributes:
-  ami: ami-0f802dc0fc1809acd
-  arn: arn:aws:ec2:eu-west-2:540044833068:instance/i-01dcfcab37ba22689
+  ami: (sensitive value)
+  arn: (known after apply)
   associate_public_ip_address: true
-  availability_zone: eu-west-2b
-  capacity_reservation_specification:
-    - capacity_reservation_preference: open
-  cpu_core_count: 1
-  cpu_options:
-    - core_count: 1
-      threads_per_core: 2
-  cpu_threads_per_core: 2
-  credit_specification:
-    - cpu_credits: unlimited
-  disable_api_stop: false
-  disable_api_termination: false
-  ebs_optimized: false
-  enable_primary_ipv6: null
-  enclave_options:
-    - enabled: false
+  availability_zone: (known after apply)
+  capacity_reservation_specification: (known after apply)
+  cpu_core_count: (known after apply)
+  cpu_options: (known after apply)
+  cpu_threads_per_core: (known after apply)
+  disable_api_stop: (known after apply)
+  disable_api_termination: (known after apply)
+  ebs_block_device: (known after apply)
+  ebs_optimized: (known after apply)
+  enable_primary_ipv6: (known after apply)
+  enclave_options: (known after apply)
+  ephemeral_block_device: (known after apply)
   get_password_data: false
-  hibernation: false
-  host_resource_group_arn: null
-  id: i-01dcfcab37ba22689
-  instance_initiated_shutdown_behavior: stop
-  instance_state: running
+  hibernation: null
+  host_id: (known after apply)
+  host_resource_group_arn: (known after apply)
+  iam_instance_profile: (known after apply)
+  id: (known after apply)
+  instance_initiated_shutdown_behavior: (known after apply)
+  instance_lifecycle: (known after apply)
+  instance_market_options: (known after apply)
+  instance_state: (known after apply)
   instance_type: t3.small
-  ipv6_address_count: 0
+  ipv6_address_count: (known after apply)
+  ipv6_addresses: (known after apply)
   key_name: Demo Key Pair
-  maintenance_options:
-    - auto_recovery: default
-  metadata_options:
-    - http_endpoint: enabled
-      http_protocol_ipv6: disabled
-      http_put_response_hop_limit: 1
-      http_tokens: optional
-      instance_metadata_tags: disabled
-  monitoring: false
-  placement_partition_number: 0
-  primary_network_interface_id: eni-0501ad33e98bb6f8c
-  private_dns: ip-10-0-10-239.eu-west-2.compute.internal
-  private_dns_name_options:
-    - enable_resource_name_dns_a_record: false
-      enable_resource_name_dns_aaaa_record: false
-      hostname_type: ip-name
-  private_ip: 10.0.10.239
-  public_dns: ec2-13-41-66-30.eu-west-2.compute.amazonaws.com
-  public_ip: 13.41.66.30
+  maintenance_options: (known after apply)
+  metadata_options: (known after apply)
+  monitoring: (known after apply)
+  network_interface: (known after apply)
+  outpost_arn: (known after apply)
+  password_data: (known after apply)
+  placement_group: (known after apply)
+  placement_partition_number: (known after apply)
+  primary_network_interface_id: (known after apply)
+  private_dns: (known after apply)
+  private_dns_name_options: (known after apply)
+  private_ip: (known after apply)
+  public_dns: (known after apply)
+  public_ip: (known after apply)
   root_block_device:
     - delete_on_termination: true
-      device_name: /dev/xvda
-      encrypted: false
-      iops: 0
-      throughput: 0
-      volume_id: vol-011ca7661217b5823
-      volume_size: 8
-      volume_type: standard
+      device_name: (known after apply)
+      encrypted: (known after apply)
+      iops: (known after apply)
+      kms_key_id: (known after apply)
+      tags: null
+      tags_all: (known after apply)
+      throughput: (known after apply)
+      volume_id: (known after apply)
+      volume_size: (known after apply)
+      volume_type: (known after apply)
+  secondary_private_ips: (known after apply)
+  security_groups: (known after apply)
   source_dest_check: true
+  spot_instance_request_id: (known after apply)
   subnet_id: subnet-036704734045071f9
   tags:
+    Environment: dev
     Name: App Server
   tags_all:
+    Environment: dev
     Name: App Server
-  tenancy: default
+  tenancy: (known after apply)
   terraform_address: module.scenarios[0].aws_instance.app_server
   terraform_name: module.scenarios[0].aws_instance.app_server
   timeouts: null
-  user_data: null
-  user_data_base64: null
+  user_data: (known after apply)
+  user_data_base64: (known after apply)
   user_data_replace_on_change: false
   volume_tags: null
+/- ec2-instance › i-02f292e8a0766d313
--- current
+++ proposed
@@ -2,72 +2,76 @@
 id: github.com/overmindtech/terraform-example.ec2-instance.module.scenarios[0].aws_instance.webserver
 attributes:
-  ami: ami-0f802dc0fc1809acd
-  arn: arn:aws:ec2:eu-west-2:540044833068:instance/i-02f292e8a0766d313
+  ami: (sensitive value)
+  arn: (known after apply)
   associate_public_ip_address: true
-  availability_zone: eu-west-2a
-  capacity_reservation_specification:
-    - capacity_reservation_preference: open
-  cpu_core_count: 1
-  cpu_options:
-    - core_count: 1
-      threads_per_core: 2
-  cpu_threads_per_core: 2
-  credit_specification:
-    - cpu_credits: unlimited
-  disable_api_stop: false
-  disable_api_termination: false
-  ebs_optimized: false
-  enable_primary_ipv6: null
-  enclave_options:
-    - enabled: false
+  availability_zone: (known after apply)
+  capacity_reservation_specification: (known after apply)
+  cpu_core_count: (known after apply)
+  cpu_options: (known after apply)
+  cpu_threads_per_core: (known after apply)
+  disable_api_stop: (known after apply)
+  disable_api_termination: (known after apply)
+  ebs_block_device: (known after apply)
+  ebs_optimized: (known after apply)
+  enable_primary_ipv6: (known after apply)
+  enclave_options: (known after apply)
+  ephemeral_block_device: (known after apply)
   get_password_data: false
-  hibernation: false
-  host_resource_group_arn: null
-  id: i-02f292e8a0766d313
-  instance_initiated_shutdown_behavior: stop
-  instance_state: running
+  hibernation: null
+  host_id: (known after apply)
+  host_resource_group_arn: (known after apply)
+  iam_instance_profile: (known after apply)
+  id: (known after apply)
+  instance_initiated_shutdown_behavior: (known after apply)
+  instance_lifecycle: (known after apply)
+  instance_market_options: (known after apply)
+  instance_state: (known after apply)
   instance_type: t3.small
-  ipv6_address_count: 0
+  ipv6_address_count: (known after apply)
+  ipv6_addresses: (known after apply)
   key_name: Demo Key Pair
-  maintenance_options:
-    - auto_recovery: default
-  metadata_options:
-    - http_endpoint: enabled
-      http_protocol_ipv6: disabled
-      http_put_response_hop_limit: 1
-      http_tokens: optional
-      instance_metadata_tags: disabled
-  monitoring: false
-  placement_partition_number: 0
-  primary_network_interface_id: eni-0784f95b7ff052c6b
-  private_dns: ip-10-0-9-25.eu-west-2.compute.internal
-  private_dns_name_options:
-    - enable_resource_name_dns_a_record: false
-      enable_resource_name_dns_aaaa_record: false
-      hostname_type: ip-name
-  private_ip: 10.0.9.25
-  public_dns: ec2-13-40-28-149.eu-west-2.compute.amazonaws.com
-  public_ip: 13.40.28.149
+  maintenance_options: (known after apply)
+  metadata_options: (known after apply)
+  monitoring: (known after apply)
+  network_interface: (known after apply)
+  outpost_arn: (known after apply)
+  password_data: (known after apply)
+  placement_group: (known after apply)
+  placement_partition_number: (known after apply)
+  primary_network_interface_id: (known after apply)
+  private_dns: (known after apply)
+  private_dns_name_options: (known after apply)
+  private_ip: (known after apply)
+  public_dns: (known after apply)
+  public_ip: (known after apply)
   root_block_device:
     - delete_on_termination: true
-      device_name: /dev/xvda
-      encrypted: false
-      iops: 0
-      throughput: 0
-      volume_id: vol-0bc99e157a38768b6
-      volume_size: 8
-      volume_type: standard
+      device_name: (known after apply)
+      encrypted: (known after apply)
+      iops: (known after apply)
+      kms_key_id: (known after apply)
+      tags: null
+      tags_all: (known after apply)
+      throughput: (known after apply)
+      volume_id: (known after apply)
+      volume_size: (known after apply)
+      volume_type: (known after apply)
+  secondary_private_ips: (known after apply)
+  security_groups: (known after apply)
   source_dest_check: true
+  spot_instance_request_id: (known after apply)
   subnet_id: subnet-06302fc5a50644cd9
   tags:
+    Environment: dev
     Name: Webserver
   tags_all:
+    Environment: dev
     Name: Webserver
-  tenancy: default
+  tenancy: (known after apply)
   terraform_address: module.scenarios[0].aws_instance.webserver
   terraform_name: module.scenarios[0].aws_instance.webserver
   timeouts: null
-  user_data: null
-  user_data_base64: null
+  user_data: (known after apply)
+  user_data_base64: (known after apply)
   user_data_replace_on_change: false
   volume_tags: null
~ ec2-launch-template › lt-0731f767e6be2ab94
--- current
+++ proposed
@@ -7,7 +7,7 @@
   disable_api_termination: false
   id: lt-0731f767e6be2ab94
-  image_id: ami-0f802dc0fc1809acd
+  image_id: ami-0ca1753a2af8d9bbf
   instance_type: t3.micro
-  latest_version: 20
+  latest_version: (known after apply)
   name: asg-change-launch-template-terraform-example20240827194210168200000007
   name_prefix: asg-change-launch-template-terraform-example
~ elbv2-listener › arn:aws:elasticloadbalancing:eu-west-2:540044833068:listener/app/scenarios--a3ec77f7-alb/a0268d67b29039c7/af5e943a0e473e91
--- current
+++ proposed
@@ -6,12 +6,6 @@
   certificate_arn: null
   default_action:
-    - forward:
-        - stickiness:
-            - duration: 0
-              enabled: false
-          target_group:
-            - arn: arn:aws:elasticloadbalancing:eu-west-2:540044833068:targetgroup/scenarios--a3ec77f7-tg/202d88113aa8b778
-              weight: 1
-      order: 1
+    - order: 1
+      target_group_arn: (known after apply)
       type: forward
   id: arn:aws:elasticloadbalancing:eu-west-2:540044833068:listener/app/scenarios--a3ec77f7-alb/a0268d67b29039c7/af5e943a0e473e91
~ ec2-route-table › rtb-0c52db7871965c5a1
--- current
+++ proposed
@@ -7,9 +7,9 @@
   owner_id: "540044833068"
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
~ ec2-address › 3.11.31.83
--- current
+++ proposed
@@ -18,9 +18,9 @@
   public_ipv4_pool: amazon
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
~ ec2-address › 18.134.176.13
--- current
+++ proposed
@@ -18,9 +18,9 @@
   public_ipv4_pool: amazon
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
~ ec2-internet-gateway › igw-0b7151f8472d03c8a
--- current
+++ proposed
@@ -6,9 +6,9 @@
   owner_id: "540044833068"
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"
~ ec2-nat-gateway › nat-0f789c96969ec0dd1
--- current
+++ proposed
@@ -12,9 +12,9 @@
   subnet_id: subnet-0b805a32f5d7f0c7b
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2a
     Terraform: "true"
~ ec2-nat-gateway › nat-06036dc6d716438e0
--- current
+++ proposed
@@ -12,9 +12,9 @@
   subnet_id: subnet-016bfadacc9c60bfc
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-eu-west-2b
     Terraform: "true"
~ ec2-route-table › rtb-07f5933d73ceaab99
--- current
+++ proposed
@@ -9,9 +9,9 @@
       nat_gateway_id: nat-0f789c96969ec0dd1
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
~ ec2-route-table › rtb-09d0b7c0ce1121c2d
--- current
+++ proposed
@@ -9,9 +9,9 @@
       nat_gateway_id: nat-06036dc6d716438e0
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
~ ec2-route-table › rtb-0536cdbeadfe92efa
--- current
+++ proposed
@@ -9,9 +9,9 @@
       gateway_id: igw-0b7151f8472d03c8a
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public
     Terraform: "true"
~ ec2-subnet › subnet-0d0a1aff83bd8a460
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2a
     Terraform: "true"
~ ec2-subnet › subnet-0303f6ca155877094
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-private-eu-west-2b
     Terraform: "true"
~ ec2-subnet › subnet-0b805a32f5d7f0c7b
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2a
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2a
     Terraform: "true"
~ ec2-subnet › subnet-016bfadacc9c60bfc
--- current
+++ proposed
@@ -18,9 +18,9 @@
   private_dns_hostname_type_on_launch: ip-name
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2b
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-public-eu-west-2b
     Terraform: "true"
~ ec2-vpc › vpc-0f4ddbf8c33e5c725
--- current
+++ proposed
@@ -20,9 +20,9 @@
   owner_id: "540044833068"
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example
     Terraform: "true"

🟠 Unmapped Changes

+ aws_route53_record › module.scenarios[0].aws_route53_record.blackhole[0]
--- current
+++ proposed
@@ -0,0 +1,21 @@
+type: aws_route53_record
+id: github.com/overmindtech/terraform-example.aws_route53_record.module.scenarios[0].aws_route53_record.blackhole[0]
+attributes:
+  alias:
+    - evaluate_target_health: false
+      name: scenarios--a3ec77f7-alb-491363430.eu-west-2.elb.amazonaws.com
+      zone_id: ZHURV8PSTC4K8
+  allow_overwrite: (known after apply)
+  fqdn: (known after apply)
+  health_check_id: null
+  id: (known after apply)
+  multivalue_answer_routing_policy: null
+  name: blackhole-terraform-example.overmind-terraform-example.com
+  records: null
+  set_identifier: null
+  terraform_address: module.scenarios[0].aws_route53_record.blackhole[0]
+  terraform_name: module.scenarios[0].aws_route53_record.blackhole[0]
+  timeouts: null
+  ttl: null
+  type: A
+  zone_id: Z01381333G7W1ZLUTENL1
+ aws_lb_target_group › module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
--- current
+++ proposed
@@ -0,0 +1,68 @@
+type: aws_lb_target_group
+id: github.com/overmindtech/terraform-example.aws_lb_target_group.module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+attributes:
+  arn: (known after apply)
+  arn_suffix: (known after apply)
+  connection_termination: (known after apply)
+  deregistration_delay: "300"
+  health_check:
+    - enabled: true
+      healthy_threshold: 5
+      interval: 60
+      matcher: "200"
+      path: /
+      port: traffic-port
+      protocol: HTTP
+      timeout: 5
+      unhealthy_threshold: 2
+  id: (known after apply)
+  ip_address_type: (known after apply)
+  lambda_multi_value_headers_enabled: false
+  load_balancer_arns: (known after apply)
+  load_balancing_algorithm_type: (known after apply)
+  load_balancing_anomaly_mitigation: (known after apply)
+  load_balancing_cross_zone_enabled: (known after apply)
+  name: scenarios--a3ec77f7-tg-blackhole
+  name_prefix: (known after apply)
+  port: 8080
+  preserve_client_ip: (known after apply)
+  protocol: HTTP
+  protocol_version: (known after apply)
+  proxy_protocol_v2: false
+  slow_start: 0
+  stickiness: (known after apply)
+  tags:
+    CreatedBy: terraform
+    DaysUntilBF: "7"
+    Environment: demo
+    JavaHeapMB: "1536"
+    MemoryMB: "2048"
+    Mode: blackhole
+    Name: scenarios--a3ec77f7-tg-blackhole
+    OptimizationWorks: "true"
+    Project: memory-optimization
+    Purpose: risk-test
+    RequiredMemoryMB: "1792"
+    RiskLevel: low
+    Scenario: cost-reduction
+  tags_all:
+    CreatedBy: terraform
+    DaysUntilBF: "7"
+    Environment: demo
+    JavaHeapMB: "1536"
+    MemoryMB: "2048"
+    Mode: blackhole
+    Name: scenarios--a3ec77f7-tg-blackhole
+    OptimizationWorks: "true"
+    Project: memory-optimization
+    Purpose: risk-test
+    RequiredMemoryMB: "1792"
+    RiskLevel: low
+    Scenario: cost-reduction
+  target_failover: (known after apply)
+  target_group_health: (known after apply)
+  target_health_state: (known after apply)
+  target_type: ip
+  terraform_address: module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+  terraform_name: module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+  vpc_id: vpc-0f4ddbf8c33e5c725
~ aws_default_network_acl › module.scenarios[0].module.vpc.aws_default_network_acl.this[0]
--- current
+++ proposed
@@ -46,9 +46,9 @@
     - subnet-0d0a1aff83bd8a460
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
~ aws_default_security_group › module.scenarios[0].module.vpc.aws_default_security_group.this[0]
--- current
+++ proposed
@@ -35,9 +35,9 @@
   revoke_rules_on_delete: false
   tags:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"
   tags_all:
-    Environment: dev
+    Environment: development
     Name: workloads-terraform-example-default
     Terraform: "true"

💥 Blast Radius

Items 76

Edges 204

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants