[Bug]: Provider produced inconsistent final plan - changed the planned action from Update to DeleteThenCreate

[tcdev00]

Terraform Core Version

v1.5.5

AWS Provider Version

v5.16.2

Affected Resource(s)

• aws_instance

Expected Behavior

Successful apply

Actual Behavior

Error. Retrying on the same state succeeds.

Relevant Error/Panic Output Snippet

│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cluster.module.instances.aws_instance.instance_container[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" changed the planned action from Update to DeleteThenCreate.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cluster.module.instances.aws_instance.instance_container[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .outpost_arn: was known, but now unknown.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cluster.module.instances.aws_instance.instance_container[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .public_ip: was known, but now unknown.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cluster.module.instances.aws_instance.instance_container[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .spot_instance_request_id: was known, but now unknown.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│
│ When expanding the plan for module.cluster.module.instances.aws_instance.instance_container[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .placement_group: was known, but now unknown.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Configuration Files

resource "aws_instance" "instance_container" {
  count                  = var.instance_count
  ami                    = var.ami
  instance_type          = var.instance_type
  subnet_id              = var.subnet_ids[count.index]
  vpc_security_group_ids = var.vpc_security_group_ids
  user_data_base64       = data.cloudinit_config.config[count.index].rendered
  iam_instance_profile   = var.iam_instance_profile
  private_ip             = var.private_ips[cound.index]

  user_data_replace_on_change = true
...

Plan

  # module.cluster.module.instances.aws_instance.instance_container[0] will be updated in-place
  ~ resource "aws_instance" "instance_container" {
        id                                   = "i-1kk4r644q9020c846"
        ...
      ~ user_data_base64                     = "H5a0ABCDE...=" -> (known after apply)

      # (20 unchanged attributes hidden)

Steps to Reproduce

• Make a change in code that creates new user_data for the AWS instance
• Make sure that resource "aws_instance" contains the attribute user_data_replace_on_change = true
• terraform apply

Debug Output

None

Panic Output

None

Important Factoids

Whenever we are going to deploy a new version, the aws_instance resources are marked for update in the plan (unless there are changes in other attributes like ami. In that case, it forces a replacement and works). When applying, it crashes with:

When expanding the plan for module.cluster.module.instances.aws_instance.instance_container[0] to include 
new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" 
changed the planned action from Update to DeleteThenCreate

If we re-apply after the failure, the aws_instance resource is marked for replacement and the deployment just works fine.

Workaround

If we deploy a new version but this time forcing the replacement with -replace or -taint, the instances are marked for replace on the first attempt and the deployment just works fine.

The most accurate description of the cause can be found here: #29012 (comment)

References

Similar/related issues:
#29012
#23315
hashicorp/terraform-provider-azurerm#21402

---

The issue #29012 is the most closely related to this current issue. Unfortunately, it's closed although the issue is still happening in the newest versions of the provider and wrecking our deployments every day.

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[microbioticajon]

I have just started running into something similar to this:

• Provider hashicorp/aws v5.19.0
• terragrunt version 0.51.7
• terraform version 1.5.7

When modifying an existing state, I get errors similar to the above.

I can destroy the resources with no problem

I can create a fresh set of resources

Specifically this seems to be affecting aws_batch_compute_environment

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.main to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" changed the planned action from
│ Update to DeleteThenCreate.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.main to include
│ new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .id: was known, but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
...

[TheSAS]

I see the same issue.

│ Error: Provider produced inconsistent final plan                                                                                                                                                                                             
│                                                                                                                                                                                                                                              
│ When expanding the plan for aws_batch_compute_environment.compute_environment to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" changed the planned action  
│ from Update to DeleteThenCreate.                                                                                                                                                                                                             
│                                                                                                                                                                                                                                              
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.                                                                                                                                                 
╵                       ```

[wrbooth]

We were running into the same issue with the aws_batch_compute_environment resource, specifically when the launch_template.version field was changed.

In some cases for our aws_batch_compute_environment resources we would see a "forces replacement" with the above change. However, in some cases, that same launch_template.version change would would instead be an "update in place" operation. All "update" operations would ultimately manifest into the inconsistent plan highlighted in this bug.

We worked around this by always forcing a replacement when the launch_template.version was changed. via:

lifecycle {
  replace_triggered_by  = [aws_launch_template.this[each.key].latest_version]
}

There are likely more elegant ways to define replace_triggered_by, but this got us out of the inconsistent plan spiral.

[bruceadowns]

I am unable to reproduce the error.

Here is one example (of many) that I tried:

resource "aws_instance" "sample" {
  ami                         = local.ami
  instance_type               = "t2.micro"
  subnet_id                   = local.subnet
  vpc_security_group_ids      = local.security_group_ids
  user_data                   = "foobar"
  user_data_replace_on_change = true
}

The original issue for aws_instance may be resolved, I recommend closing unless a more detailed failure scenario is provided.

[bruceadowns]

The PR #38628 fixes the issue noted by various comments regarding aws_batch_compute_environment.