[Bug]: S3 bucket lifecycle configuration error after updating provider to 5.86.0

[chelseaharalson]

Hi,

We are getting this error after the terraform-provider-aws update to 5.86.0:

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to
│ aws_s3_bucket_lifecycle_configuration.availability_service_s3_bucket_lifecycle_configuration,
│ provider "provider["registry.terraform.io/hashicorp/aws"]" produced an
│ unexpected new value: .rule[0].expiration[0].expired_object_delete_marker:
│ was cty.True, but now cty.False.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[baer95]

Could this be related to or a duplicate of #41268?

[chelseaharalson]

@baer95 it might be, the error is slightly different though, but has to do with the lifecycle.

[gdavison]

It's related, but a separate problem

[tinuz74]

We have exactly the same issue.

Error: Provider produced inconsistent result after apply
│
│ When applying changes to
│ module.gateway.aws_s3_bucket_lifecycle_configuration.tia-nfs-bucket-lifecycle_rule,
│ provider "provider["registry.terraform.io/hashicorp/aws"]" produced an
│ unexpected new value: .rule[4].expiration[0].expired_object_delete_marker:
│ was cty.True, but now cty.False.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[tinuz74]

This bug is now since friday. Is there any forecast when we can expect a fix?

[gdavison]

Hi @chelseaharalson and @tinuz74 can you please share your the full configuration of your aws_s3_bucket_lifecycle_configuration both before and after the upgrade, as well as the error message that you're getting? Debug logs generated using the instructions at https://developer.hashicorp.com/terraform/internals/debugging as well as setting the environment variable TF_LOG_AWS_AUTOFLEX to DEBUG would also be useful.

There are several tests for both expired_object_delete_marker = true and = false, but I haven't come across this behaviour

Without a way to reproduce the problem that you're reporting, we won't be able to fix it.

[chelseaharalson]

Hi @chelseaharalson and @tinuz74 can you please share your the full configuration of your aws_s3_bucket_lifecycle_configuration both before and after the upgrade, as well as the error message that you're getting? Debug logs generated using the instructions at https://developer.hashicorp.com/terraform/internals/debugging as well as setting the environment variable TF_LOG_AWS_AUTOFLEX to DEBUG would also be useful.

There are several tests for both expired_object_delete_marker = true and = false, but I haven't come across this behaviour

Without a way to reproduce the problem that you're reporting, we won't be able to fix it.

resource "aws_s3_bucket_lifecycle_configuration" "as_s3_bucket_lifecycle_configuration" {
  bucket = module.as_s3_upload_bucket.s3_bucket.id

  rule {
    id = "${module.as_s3_upload_bucket.s3_bucket.id} S3 bucket lifecycle rule for objects"

    filter {
      prefix = "${var.as_folder}/"
    }

    expiration {
      days = var.expiration_days
      expired_object_delete_marker = true
    }

    noncurrent_version_expiration {
      noncurrent_days = var.expiration_days
    }

    status = "Enabled"
  }

  depends_on = [aws_s3_bucket_versioning.as_main_s3_bucket_versioning]
}

I did not change the configuration before or after the upgrade. This is the error message:
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to
│ aws_s3_bucket_lifecycle_configuration.availability_service_s3_bucket_lifecycle_configuration,
│ provider "provider["registry.terraform.io/hashicorp/aws"]" produced an
│ unexpected new value: .rule[0].expiration[0].expired_object_delete_marker:
│ was cty.True, but now cty.False.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[tinuz74]

Hi, See config below. Also nothing changed with the code before or after the upgrade.
##############################################################
resource "aws_s3_bucket_lifecycle_configuration" "xxxxx-nfs-bucket-lifecycle_rule" {
bucket = aws_s3_bucket.xxxxx-nfs-bucket.id
rule {
id = "xxxxxxxxxxxxxxx"
status = "Enabled"

filter {
  prefix = "xxxxxxx${var.environment}/path/path/"
}

abort_incomplete_multipart_upload {
  days_after_initiation = 8
}

expiration {
  days = 8
}

noncurrent_version_expiration {
  noncurrent_days = 1
}

}

dynamic "rule" {
for_each = toset(var.environments)
content {
id = "${rule.value}/path/path/"
status = "Enabled"
filter {
prefix = "${rule.value}/path/path/"
}
abort_incomplete_multipart_upload {
days_after_initiation = 2
}
expiration {
days = 30
expired_object_delete_marker = true
}
# Add 3 years expiration policy to delete objects permanently (Non-Current version)
noncurrent_version_expiration {
noncurrent_days = 1
}
}
}
dynamic "rule" {
for_each = [for object in local.output_dir_archive_list : {
directory = object.directory
environment = object.environment
inttierdays = object.inttierdays
expirationdays = object.expirationdays
}
]

content {
  id     = "archival_removal_${rule.value.environment}_${replace(rule.value.directory, "/", "_")}_objects"
  status = "Enabled"

  filter {
    prefix = rule.value.directory == "" ? "${rule.value.environment}/" : "${rule.value.environment}/${rule.value.directory}/"
  }

  abort_incomplete_multipart_upload {
    days_after_initiation = 2
  }

  transition {
    days          = rule.value.inttierdays
    storage_class = "INTELLIGENT_TIERING"
  }

  expiration {
    days                         = rule.value.expirationdays
    expired_object_delete_marker = true
  }
  # Add 3 years expiration policy to delete objects permanently (Non-Current version)
  noncurrent_version_expiration {
    noncurrent_days = 1
  }
}

}
}
###################################################

[mschultz-aofl]

@gdavison This is also affecting our org, with similar issues as the other posters above - no changes to config, applied 5.85.0, then 5.86.0, and now on 5.86.1. It's now giving the same error as above - I've attached some debug logs below, considering the full output is massive and there's some concerns about data leaks by posting the full debug log; let me know if there's anything in particular you're looking for. I can also grab all logs for the tf_req_id if you'd like, but again, probably more than I can paste here.

Plan logs:

2025-02-11T23:25:00.040Z [DEBUG] provider.terraform-provider-aws_v5.86.1_x5: HTTP Response Received: aws.region=us-west-2 http.response.header.server=AmazonS3 http.response.header.x_amz_request_id=STN9 tf_resource_type=aws_s3_bucket http.response.header.date="Tue, 11 Feb 2025 23:25:00 GMT" rpc.method=GetBucketLifecycleConfiguration rpc.system=aws-api @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:45 @module=aws http.response.body="<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>delete-old-versions-after-7-days</ID><Filter/><Status>Enabled</Status><Expiration><ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker></Expiration><NoncurrentVersionExpiration><NoncurrentDays>7</NoncurrentDays></NoncurrentVersionExpiration></Rule></LifecycleConfiguration>
2025-02-11T23:25:01.090Z [INFO]  provider.terraform-provider-aws_v5.86.1_x5: Converting: autoflex.source.type=*bool tf_resource_type=aws_s3_bucket_lifecycle_configuration @module=aws.autoflex autoflex.target.path=ExpiredObjectDeleteMarker tf_mux_provider=*proto5server.Server autoflex.source.path=ExpiredObjectDeleteMarker autoflex.target.type=github.com/hashicorp/terraform-plugin-framework/types/basetypes.BoolValue tf_rpc=ReadResource @caller=github.com/hashicorp/terraform-provider-aws/internal/framework/flex/auto_flatten.go:146 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=aacd2c11-5d1a-9824-d714-12b1befe5696 timestamp=2025-02-11T23:25:01.090Z
2025-02-11T23:25:01.090Z [DEBUG] provider.terraform-provider-aws_v5.86.1_x5: Using legacy flattener: @module=aws.autoflex autoflex.target.path=ExpiredObjectDeleteMarker tf_mux_provider=*proto5server.Server tf_rpc=ReadResource autoflex.target.type=github.com/hashicorp/terraform-plugin-framework/types/basetypes.BoolValue tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=aacd2c11-5d1a-9824-d714-12b1befe5696 @caller=github.com/hashicorp/terraform-provider-aws/internal/framework/flex/auto_flatten.go:216 autoflex.source.path=ExpiredObjectDeleteMarker autoflex.source.type=*bool tf_resource_type=aws_s3_bucket_lifecycle_configuration timestamp=2025-02-11T23:25:01.090Z
2025-02-11T23:25:01.093Z [DEBUG] provider.terraform-provider-aws_v5.86.1_x5: Value switched to prior value due to semantic equality logic: tf_mux_provider=*proto5server.Server tf_req_id=aacd2c11-5d1a-9824-d714-12b1befe5696 tf_resource_type=aws_s3_bucket_lifecycle_configuration @module=sdk.framework tf_attribute_path=rule[0].expiration[0].expired_object_delete_marker tf_provider_addr=registry.terraform.io/hashicorp/aws tf_rpc=ReadResource @caller=github.com/hashicorp/[email protected]/internal/fwschemadata/value_semantic_equality.go:91 timestamp=2025-02-11T23:25:01.093Z
 ~ resource "aws_s3_bucket_lifecycle_configuration" "this" {
        id                                     = "test-automation-dev-assets"
        # (2 unchanged attributes hidden)
      ~ rule {
            id     = "cleanup"
            # (1 unchanged attribute hidden)
          ~ expiration {
              ~ expired_object_delete_marker = false -> true
                # (1 unchanged attribute hidden)
            }
          ~ filter {
              + object_size_greater_than = (known after apply)
              + object_size_less_than    = (known after apply)
            }
        }
    }

Apply logs:

2025-02-11T23:26:28.843Z [INFO]  provider.terraform-provider-aws_v5.86.1_x5: Converting: @caller=github.com/hashicorp/terraform-provider-aws/internal/framework/flex/auto_flatten.go:146 autoflex.target.path=ExpiredObjectDeleteMarker autoflex.target.type=github.com/hashicorp/terraform-plugin-framework/types/basetypes.BoolValue tf_mux_provider=*proto5server.Server tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=debc5226-87cb-8eb1-0608-b55479177341 tf_resource_type=aws_s3_bucket_lifecycle_configuration tf_rpc=ApplyResourceChange @module=aws.autoflex autoflex.source.path=ExpiredObjectDeleteMarker autoflex.source.type=*bool timestamp=2025-02-11T23:26:28.843Z
2025-02-11T23:26:28.843Z [DEBUG] provider.terraform-provider-aws_v5.86.1_x5: Using legacy flattener: autoflex.source.path=ExpiredObjectDeleteMarker autoflex.source.type=*bool autoflex.target.type=github.com/hashicorp/terraform-plugin-framework/types/basetypes.BoolValue tf_provider_addr=registry.terraform.io/hashicorp/aws @caller=github.com/hashicorp/terraform-provider-aws/internal/framework/flex/auto_flatten.go:216 autoflex.target.path=ExpiredObjectDeleteMarker tf_mux_provider=*proto5server.Server tf_req_id=debc5226-87cb-8eb1-0608-b55479177341 @module=aws.autoflex tf_resource_type=aws_s3_bucket_lifecycle_configuration tf_rpc=ApplyResourceChange timestamp=2025-02-11T23:26:28.843Z

2025-02-11T23:26:08.550Z [DEBUG] provider.terraform-provider-aws_v5.86.1_x5: HTTP Response Received: http.status_code=200 rpc.method=GetBucketLifecycleConfiguration tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=debc57341 tf_resource_type=aws_s3_bucket_lifecycle_configuration @caller=github.com/hashicorp/aws-sdk-go-base/[email protected]/logging/tf_logger.go:45 aws.region=us-west-2 http.response.header.server=AmazonS3 http.response.header.x_amz_id_2=d57cpIO25NE= http.response_content_length=263 tf_rpc=ApplyResourceChange @module=aws http.duration=46 http.response.header.date="Tue, 11 Feb 2025 23:26:09 GMT" http.response.header.x_amz_request_id=B3HGN rpc.service=S3 http.response.body="<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>cleanup</ID><Filter><Prefix></Prefix></Filter><Status>Enabled</Status><Expiration><Days>60</Days></Expiration></Rule></LifecycleConfiguration>

Full source:

module "test_automation_dev_assets" {
  version = "4.5.0"
  source = "terraform-aws-modules/s3-bucket/aws"

  bucket = "test-automation-dev-assets"

  server_side_encryption_configuration  = {
    rule = {
      apply_server_side_encryption_by_default = {
        sse_algorithm     = "AES256"
      }
    }
  }

  attach_policy = true
  policy = <<EOF
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudFrontServicePrincipalReadOnly",
      "Effect": "Allow",
      "Principal": {
          "Service": "cloudfront.amazonaws.com"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::test-automation-dev-assets/*",
      "Condition": {
          "StringEquals": {
              "AWS:SourceArn": "XYZ"
          }
      }
    }
  ]
}
  EOF
  
    block_public_acls       = true
    block_public_policy     = true
    ignore_public_acls      = true
    restrict_public_buckets = true
    lifecycle_rule = [
    {
      id      = "cleanup"
      enabled = true
  
      expiration = {
        days = 60
        expired_object_delete_marker = true
      }
    }]
  }