-
Notifications
You must be signed in to change notification settings - Fork 1
Resilience/dns blackhole clean #345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This change simulates the AWS DNS outage scenario by: 1. Creating a blackhole target group with no registered targets 2. Updating ALB listener to forward all traffic to the empty target group 3. Adding Route53 DNS record with no failover capability and disabled health checks This mimics the scenario where DNS endpoint resolves but has no healthy backends, causing immediate 503 errors and service unavailability. The change removes failover protection and routes traffic to a target group with zero healthy targets. Impact: High - complete service outage, no automatic failover capability
Open in Overmind ↗
🔴 Change SignalsRoutine 🔴 🔥 RisksRemoving target group from ALB listener default action will stop all routing and cause outage When applied, the listener will have no valid forwarding destination for its default rule, so requests reaching scenarios--a3ec77f7-alb will not be routed to any targets. This will cause immediate traffic loss and application downtime; alarms tied to the target group may no longer reflect live routing conditions while the service is unreachable. EC2 replacement will delete root EBS volume (DeleteOnTermination=true), causing loss of instance state/data Any OS-level changes, logs, or application state stored on the root disk will be lost unless they are baked into the new AMI or otherwise preserved, which can cause service disruption or irrecoverable data loss for workloads that depend on files on the root filesystem. 🟣 Expected Changes+/- ec2-instance › i-01dcfcab37ba22689--- current
+++ proposed
@@ -2,72 +2,76 @@
id: github.com/overmindtech/terraform-example.ec2-instance.module.scenarios[0].aws_instance.app_server
attributes:
- ami: ami-0f802dc0fc1809acd
- arn: arn:aws:ec2:eu-west-2:540044833068:instance/i-01dcfcab37ba22689
+ ami: (sensitive value)
+ arn: (known after apply)
associate_public_ip_address: true
- availability_zone: eu-west-2b
- capacity_reservation_specification:
- - capacity_reservation_preference: open
- cpu_core_count: 1
- cpu_options:
- - core_count: 1
- threads_per_core: 2
- cpu_threads_per_core: 2
- credit_specification:
- - cpu_credits: unlimited
- disable_api_stop: false
- disable_api_termination: false
- ebs_optimized: false
- enable_primary_ipv6: null
- enclave_options:
- - enabled: false
+ availability_zone: (known after apply)
+ capacity_reservation_specification: (known after apply)
+ cpu_core_count: (known after apply)
+ cpu_options: (known after apply)
+ cpu_threads_per_core: (known after apply)
+ disable_api_stop: (known after apply)
+ disable_api_termination: (known after apply)
+ ebs_block_device: (known after apply)
+ ebs_optimized: (known after apply)
+ enable_primary_ipv6: (known after apply)
+ enclave_options: (known after apply)
+ ephemeral_block_device: (known after apply)
get_password_data: false
- hibernation: false
- host_resource_group_arn: null
- id: i-01dcfcab37ba22689
- instance_initiated_shutdown_behavior: stop
- instance_state: running
+ hibernation: null
+ host_id: (known after apply)
+ host_resource_group_arn: (known after apply)
+ iam_instance_profile: (known after apply)
+ id: (known after apply)
+ instance_initiated_shutdown_behavior: (known after apply)
+ instance_lifecycle: (known after apply)
+ instance_market_options: (known after apply)
+ instance_state: (known after apply)
instance_type: t3.small
- ipv6_address_count: 0
+ ipv6_address_count: (known after apply)
+ ipv6_addresses: (known after apply)
key_name: Demo Key Pair
- maintenance_options:
- - auto_recovery: default
- metadata_options:
- - http_endpoint: enabled
- http_protocol_ipv6: disabled
- http_put_response_hop_limit: 1
- http_tokens: optional
- instance_metadata_tags: disabled
- monitoring: false
- placement_partition_number: 0
- primary_network_interface_id: eni-0501ad33e98bb6f8c
- private_dns: ip-10-0-10-239.eu-west-2.compute.internal
- private_dns_name_options:
- - enable_resource_name_dns_a_record: false
- enable_resource_name_dns_aaaa_record: false
- hostname_type: ip-name
- private_ip: 10.0.10.239
- public_dns: ec2-13-41-66-30.eu-west-2.compute.amazonaws.com
- public_ip: 13.41.66.30
+ maintenance_options: (known after apply)
+ metadata_options: (known after apply)
+ monitoring: (known after apply)
+ network_interface: (known after apply)
+ outpost_arn: (known after apply)
+ password_data: (known after apply)
+ placement_group: (known after apply)
+ placement_partition_number: (known after apply)
+ primary_network_interface_id: (known after apply)
+ private_dns: (known after apply)
+ private_dns_name_options: (known after apply)
+ private_ip: (known after apply)
+ public_dns: (known after apply)
+ public_ip: (known after apply)
root_block_device:
- delete_on_termination: true
- device_name: /dev/xvda
- encrypted: false
- iops: 0
- throughput: 0
- volume_id: vol-011ca7661217b5823
- volume_size: 8
- volume_type: standard
+ device_name: (known after apply)
+ encrypted: (known after apply)
+ iops: (known after apply)
+ kms_key_id: (known after apply)
+ tags: null
+ tags_all: (known after apply)
+ throughput: (known after apply)
+ volume_id: (known after apply)
+ volume_size: (known after apply)
+ volume_type: (known after apply)
+ secondary_private_ips: (known after apply)
+ security_groups: (known after apply)
source_dest_check: true
+ spot_instance_request_id: (known after apply)
subnet_id: subnet-036704734045071f9
tags:
+ Environment: dev
Name: App Server
tags_all:
+ Environment: dev
Name: App Server
- tenancy: default
+ tenancy: (known after apply)
terraform_address: module.scenarios[0].aws_instance.app_server
terraform_name: module.scenarios[0].aws_instance.app_server
timeouts: null
- user_data: null
- user_data_base64: null
+ user_data: (known after apply)
+ user_data_base64: (known after apply)
user_data_replace_on_change: false
volume_tags: null
+/- ec2-instance › i-02f292e8a0766d313--- current
+++ proposed
@@ -2,72 +2,76 @@
id: github.com/overmindtech/terraform-example.ec2-instance.module.scenarios[0].aws_instance.webserver
attributes:
- ami: ami-0f802dc0fc1809acd
- arn: arn:aws:ec2:eu-west-2:540044833068:instance/i-02f292e8a0766d313
+ ami: (sensitive value)
+ arn: (known after apply)
associate_public_ip_address: true
- availability_zone: eu-west-2a
- capacity_reservation_specification:
- - capacity_reservation_preference: open
- cpu_core_count: 1
- cpu_options:
- - core_count: 1
- threads_per_core: 2
- cpu_threads_per_core: 2
- credit_specification:
- - cpu_credits: unlimited
- disable_api_stop: false
- disable_api_termination: false
- ebs_optimized: false
- enable_primary_ipv6: null
- enclave_options:
- - enabled: false
+ availability_zone: (known after apply)
+ capacity_reservation_specification: (known after apply)
+ cpu_core_count: (known after apply)
+ cpu_options: (known after apply)
+ cpu_threads_per_core: (known after apply)
+ disable_api_stop: (known after apply)
+ disable_api_termination: (known after apply)
+ ebs_block_device: (known after apply)
+ ebs_optimized: (known after apply)
+ enable_primary_ipv6: (known after apply)
+ enclave_options: (known after apply)
+ ephemeral_block_device: (known after apply)
get_password_data: false
- hibernation: false
- host_resource_group_arn: null
- id: i-02f292e8a0766d313
- instance_initiated_shutdown_behavior: stop
- instance_state: running
+ hibernation: null
+ host_id: (known after apply)
+ host_resource_group_arn: (known after apply)
+ iam_instance_profile: (known after apply)
+ id: (known after apply)
+ instance_initiated_shutdown_behavior: (known after apply)
+ instance_lifecycle: (known after apply)
+ instance_market_options: (known after apply)
+ instance_state: (known after apply)
instance_type: t3.small
- ipv6_address_count: 0
+ ipv6_address_count: (known after apply)
+ ipv6_addresses: (known after apply)
key_name: Demo Key Pair
- maintenance_options:
- - auto_recovery: default
- metadata_options:
- - http_endpoint: enabled
- http_protocol_ipv6: disabled
- http_put_response_hop_limit: 1
- http_tokens: optional
- instance_metadata_tags: disabled
- monitoring: false
- placement_partition_number: 0
- primary_network_interface_id: eni-0784f95b7ff052c6b
- private_dns: ip-10-0-9-25.eu-west-2.compute.internal
- private_dns_name_options:
- - enable_resource_name_dns_a_record: false
- enable_resource_name_dns_aaaa_record: false
- hostname_type: ip-name
- private_ip: 10.0.9.25
- public_dns: ec2-13-40-28-149.eu-west-2.compute.amazonaws.com
- public_ip: 13.40.28.149
+ maintenance_options: (known after apply)
+ metadata_options: (known after apply)
+ monitoring: (known after apply)
+ network_interface: (known after apply)
+ outpost_arn: (known after apply)
+ password_data: (known after apply)
+ placement_group: (known after apply)
+ placement_partition_number: (known after apply)
+ primary_network_interface_id: (known after apply)
+ private_dns: (known after apply)
+ private_dns_name_options: (known after apply)
+ private_ip: (known after apply)
+ public_dns: (known after apply)
+ public_ip: (known after apply)
root_block_device:
- delete_on_termination: true
- device_name: /dev/xvda
- encrypted: false
- iops: 0
- throughput: 0
- volume_id: vol-0bc99e157a38768b6
- volume_size: 8
- volume_type: standard
+ device_name: (known after apply)
+ encrypted: (known after apply)
+ iops: (known after apply)
+ kms_key_id: (known after apply)
+ tags: null
+ tags_all: (known after apply)
+ throughput: (known after apply)
+ volume_id: (known after apply)
+ volume_size: (known after apply)
+ volume_type: (known after apply)
+ secondary_private_ips: (known after apply)
+ security_groups: (known after apply)
source_dest_check: true
+ spot_instance_request_id: (known after apply)
subnet_id: subnet-06302fc5a50644cd9
tags:
+ Environment: dev
Name: Webserver
tags_all:
+ Environment: dev
Name: Webserver
- tenancy: default
+ tenancy: (known after apply)
terraform_address: module.scenarios[0].aws_instance.webserver
terraform_name: module.scenarios[0].aws_instance.webserver
timeouts: null
- user_data: null
- user_data_base64: null
+ user_data: (known after apply)
+ user_data_base64: (known after apply)
user_data_replace_on_change: false
volume_tags: null
~ ec2-launch-template › lt-0731f767e6be2ab94--- current
+++ proposed
@@ -7,7 +7,7 @@
disable_api_termination: false
id: lt-0731f767e6be2ab94
- image_id: ami-0f802dc0fc1809acd
+ image_id: ami-0ca1753a2af8d9bbf
instance_type: t3.micro
- latest_version: 20
+ latest_version: (known after apply)
name: asg-change-launch-template-terraform-example20240827194210168200000007
name_prefix: asg-change-launch-template-terraform-example
~ elbv2-listener › arn:aws:elasticloadbalancing:eu-west-2:540044833068:listener/app/scenarios--a3ec77f7-alb/a0268d67b29039c7/af5e943a0e473e91--- current
+++ proposed
@@ -6,12 +6,6 @@
certificate_arn: null
default_action:
- - forward:
- - stickiness:
- - duration: 0
- enabled: false
- target_group:
- - arn: arn:aws:elasticloadbalancing:eu-west-2:540044833068:targetgroup/scenarios--a3ec77f7-tg/202d88113aa8b778
- weight: 1
- order: 1
+ - order: 1
+ target_group_arn: (known after apply)
type: forward
id: arn:aws:elasticloadbalancing:eu-west-2:540044833068:listener/app/scenarios--a3ec77f7-alb/a0268d67b29039c7/af5e943a0e473e91
~ ec2-route-table › rtb-0c52db7871965c5a1--- current
+++ proposed
@@ -7,9 +7,9 @@
owner_id: "540044833068"
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-default
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-default
Terraform: "true"
~ ec2-address › 3.11.31.83--- current
+++ proposed
@@ -18,9 +18,9 @@
public_ipv4_pool: amazon
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2a
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2a
Terraform: "true"
~ ec2-address › 18.134.176.13--- current
+++ proposed
@@ -18,9 +18,9 @@
public_ipv4_pool: amazon
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2b
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2b
Terraform: "true"
~ ec2-internet-gateway › igw-0b7151f8472d03c8a--- current
+++ proposed
@@ -6,9 +6,9 @@
owner_id: "540044833068"
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example
Terraform: "true"
~ ec2-nat-gateway › nat-0f789c96969ec0dd1--- current
+++ proposed
@@ -12,9 +12,9 @@
subnet_id: subnet-0b805a32f5d7f0c7b
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2a
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2a
Terraform: "true"
~ ec2-nat-gateway › nat-06036dc6d716438e0--- current
+++ proposed
@@ -12,9 +12,9 @@
subnet_id: subnet-016bfadacc9c60bfc
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2b
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-eu-west-2b
Terraform: "true"
~ ec2-route-table › rtb-07f5933d73ceaab99--- current
+++ proposed
@@ -9,9 +9,9 @@
nat_gateway_id: nat-0f789c96969ec0dd1
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2a
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2a
Terraform: "true"
~ ec2-route-table › rtb-09d0b7c0ce1121c2d--- current
+++ proposed
@@ -9,9 +9,9 @@
nat_gateway_id: nat-06036dc6d716438e0
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2b
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2b
Terraform: "true"
~ ec2-route-table › rtb-0536cdbeadfe92efa--- current
+++ proposed
@@ -9,9 +9,9 @@
gateway_id: igw-0b7151f8472d03c8a
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-public
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-public
Terraform: "true"
~ ec2-subnet › subnet-0d0a1aff83bd8a460--- current
+++ proposed
@@ -18,9 +18,9 @@
private_dns_hostname_type_on_launch: ip-name
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2a
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2a
Terraform: "true"
~ ec2-subnet › subnet-0303f6ca155877094--- current
+++ proposed
@@ -18,9 +18,9 @@
private_dns_hostname_type_on_launch: ip-name
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2b
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-private-eu-west-2b
Terraform: "true"
~ ec2-subnet › subnet-0b805a32f5d7f0c7b--- current
+++ proposed
@@ -18,9 +18,9 @@
private_dns_hostname_type_on_launch: ip-name
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-public-eu-west-2a
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-public-eu-west-2a
Terraform: "true"
~ ec2-subnet › subnet-016bfadacc9c60bfc--- current
+++ proposed
@@ -18,9 +18,9 @@
private_dns_hostname_type_on_launch: ip-name
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-public-eu-west-2b
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-public-eu-west-2b
Terraform: "true"
~ ec2-vpc › vpc-0f4ddbf8c33e5c725--- current
+++ proposed
@@ -20,9 +20,9 @@
owner_id: "540044833068"
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example
Terraform: "true"
🟠 Unmapped Changes+ aws_route53_record › module.scenarios[0].aws_route53_record.blackhole[0]--- current
+++ proposed
@@ -0,0 +1,21 @@
+type: aws_route53_record
+id: github.com/overmindtech/terraform-example.aws_route53_record.module.scenarios[0].aws_route53_record.blackhole[0]
+attributes:
+ alias:
+ - evaluate_target_health: false
+ name: scenarios--a3ec77f7-alb-491363430.eu-west-2.elb.amazonaws.com
+ zone_id: ZHURV8PSTC4K8
+ allow_overwrite: (known after apply)
+ fqdn: (known after apply)
+ health_check_id: null
+ id: (known after apply)
+ multivalue_answer_routing_policy: null
+ name: blackhole-terraform-example.overmind-terraform-example.com
+ records: null
+ set_identifier: null
+ terraform_address: module.scenarios[0].aws_route53_record.blackhole[0]
+ terraform_name: module.scenarios[0].aws_route53_record.blackhole[0]
+ timeouts: null
+ ttl: null
+ type: A
+ zone_id: Z01381333G7W1ZLUTENL1
+ aws_lb_target_group › module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]--- current
+++ proposed
@@ -0,0 +1,68 @@
+type: aws_lb_target_group
+id: github.com/overmindtech/terraform-example.aws_lb_target_group.module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+attributes:
+ arn: (known after apply)
+ arn_suffix: (known after apply)
+ connection_termination: (known after apply)
+ deregistration_delay: "300"
+ health_check:
+ - enabled: true
+ healthy_threshold: 5
+ interval: 60
+ matcher: "200"
+ path: /
+ port: traffic-port
+ protocol: HTTP
+ timeout: 5
+ unhealthy_threshold: 2
+ id: (known after apply)
+ ip_address_type: (known after apply)
+ lambda_multi_value_headers_enabled: false
+ load_balancer_arns: (known after apply)
+ load_balancing_algorithm_type: (known after apply)
+ load_balancing_anomaly_mitigation: (known after apply)
+ load_balancing_cross_zone_enabled: (known after apply)
+ name: scenarios--a3ec77f7-tg-blackhole
+ name_prefix: (known after apply)
+ port: 8080
+ preserve_client_ip: (known after apply)
+ protocol: HTTP
+ protocol_version: (known after apply)
+ proxy_protocol_v2: false
+ slow_start: 0
+ stickiness: (known after apply)
+ tags:
+ CreatedBy: terraform
+ DaysUntilBF: "7"
+ Environment: demo
+ JavaHeapMB: "1536"
+ MemoryMB: "2048"
+ Mode: blackhole
+ Name: scenarios--a3ec77f7-tg-blackhole
+ OptimizationWorks: "true"
+ Project: memory-optimization
+ Purpose: risk-test
+ RequiredMemoryMB: "1792"
+ RiskLevel: low
+ Scenario: cost-reduction
+ tags_all:
+ CreatedBy: terraform
+ DaysUntilBF: "7"
+ Environment: demo
+ JavaHeapMB: "1536"
+ MemoryMB: "2048"
+ Mode: blackhole
+ Name: scenarios--a3ec77f7-tg-blackhole
+ OptimizationWorks: "true"
+ Project: memory-optimization
+ Purpose: risk-test
+ RequiredMemoryMB: "1792"
+ RiskLevel: low
+ Scenario: cost-reduction
+ target_failover: (known after apply)
+ target_group_health: (known after apply)
+ target_health_state: (known after apply)
+ target_type: ip
+ terraform_address: module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+ terraform_name: module.scenarios[0].module.memory_optimization.aws_lb_target_group.blackhole[0]
+ vpc_id: vpc-0f4ddbf8c33e5c725
~ aws_default_network_acl › module.scenarios[0].module.vpc.aws_default_network_acl.this[0]--- current
+++ proposed
@@ -46,9 +46,9 @@
- subnet-0d0a1aff83bd8a460
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-default
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-default
Terraform: "true"
~ aws_default_security_group › module.scenarios[0].module.vpc.aws_default_security_group.this[0]--- current
+++ proposed
@@ -35,9 +35,9 @@
revoke_rules_on_delete: false
tags:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-default
Terraform: "true"
tags_all:
- Environment: dev
+ Environment: development
Name: workloads-terraform-example-default
Terraform: "true"
💥 Blast RadiusItems Edges |

No description provided.