[Bug]: Adding tags to root_block_device in aws_instance caused Error: Provider produced inconsistent final plan

[hendolim]

Terraform Core Version

1.9.8

AWS Provider Version

5.85.0

Affected Resource(s)

I added new tags in root_block_device for existing ec2 instances. They were applied correctly for some and not for the others.

Expected Behavior

TF should have applied successfully and added the tags to my instance.

Actual Behavior

Error: Provider produced inconsistent final plan
When expanding the plan for aws_instance.my_instance to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .root_block_device[0].tags: new element "Date" has appeared.
This is a bug in the provider, which should be reported in the provider's own issue tracker.

Error: Provider produced inconsistent final plan
When expanding the plan for aws_instance.my_instance to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .root_block_device[0].tags: new element "Name" has appeared.
This is a bug in the provider, which should be reported in the provider's own issue tracker.

Error: Provider produced inconsistent final plan
When expanding the plan for aws_instance.my_instance to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .root_block_device[0].tags: new element "Owner" has appeared.
This is a bug in the provider, which should be reported in the provider's own issue tracker.

Relevant Error/Panic Output Snippet

Terraform Configuration Files

resource "aws_instance" "my_instance" {
  ami               = data.aws_ami.windows.id
  availability_zone = local.availability_zone
  instance_type     = local.instance_type
  subnet_id              = var.subnet_map[local.availability_zone]
  vpc_security_group_ids = local.security_group
  iam_instance_profile = "my_instance_profile"

  user_data = local.user_data_start
 
  tags = {
    Name = "my_instance"
  }

  root_block_device {
    encrypted = true
    tags = {
      Name = "my_instance"
      Owner = "me"
      Date = "${formatdate("YYYY-MM-DD-hh-mm", timestamp())}"
    }
  }
}

Steps to Reproduce

Create an instance using the template above, but without the tags, under root_block_device.
Now update the template by adding the tags, and apply. You will see the error. Note that I have not been able to reproduce this consistently

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @hendolim 👋 Thank you for taking the time to raise this! While looking at this during triage, my instinct is that this is related to the use of the timestamp() function in the Date tag, given the caveats mentioned in that function's documentation. I'm happy to leave this open for further review, but it may be worth testing without the use of timestamp() to see if that resolves the issue in the meantime.

[hendolim]

Hey @hendolim 👋 Thank you for taking the time to raise this! While looking at this during triage, my instinct is that this is related to the use of the timestamp() function in the Date tag, given the caveats mentioned in that function's documentation. I'm happy to leave this open for further review, but it may be worth testing without the use of timestamp() to see if that resolves the issue in the meantime.

Thanks Justin, removing the Date tag seem to solve it for some instances but not for the others. This issue has been very challenging to reproduce consistently. Another thing worth mentioning is that I have been using the exact same resource definition and tags for linux machines, and they did not run into such issue. Although the initial tags updates were done a while back, using an earlier TF and AWS provider versions. But they do go through instance recreation very regularly since I have regular AMI updates, so the root block device creation has been working fine for a while.

[hendolim]

Upon further investigation, it seems this is only happening when updating volumes with no tags at all. It doesn't happen when creating a fresh volume or when updating a volume with some other existing tags.

I wonder if not only the timestamp() was evaluated during apply, but the entire tags block, which caused that inconsistency between plan and apply. But for now, using plantimestamp() seems to have fixed it