plans with very large number of cloudflare_records take too much time to complete

[jficz]

Confirmation

• This is a bug with an existing resource and is not a feature request or enhancement. Feature requests should be submitted with Cloudflare Support or your account team.
• I have searched the issue tracker and my issue isn't already found.
• I have replicated my issue using the latest version of the provider and it is still present.

Terraform and Cloudflare provider version

Terraform 1.5.7 (we're limited to the last OSS version) / OpenTofu 1.6.2

Provider version 4.39 (because of #4280) but tried with 4.50 with similar results.

Affected resource(s)

cloudflare_record

Terraform configuration files

resource "cloudflare_record" "record0001" {
  content = "192.0.2.1"
  name    = "www"
  proxied = false
  ttl     = 1
  type    = "A"
  zone_id = "<zone_id>
}

# ...
# repeat ^^^ many times
# ...

resource "cloudflare_record" "record4356" {
  content = "www"
  name    = "web"
  proxied = false
  ttl     = 1
  type    = "CNAME"
  zone_id = "<zone_id>
}

Link to debug output

https://gist.github.com/jficz/a0c393bef69720d882dbec8bacba32c2

Panic output

No response

Expected output

Much faster plan execution.

Actual output

takes about an hour to execute a full import of ~5000 records

takes about half that time to update just a single record

Steps to reproduce

1. create many records (3000+)
2. run terraform plan/apply
3. wait....

Additional factoids

Debug log just for a refresh of ~3500 records with very few changes has 19MB. Attached debug log only has two imported records and one change but the information in it is otherwise the same as in the large one (sans the other 3500 records).

Due to API rate limiting, zones with thousands of DNS records take ages to properly refresh and apply which makes the provider highly impractical for large deployments.

Such long runs cause issues with CI, block resources and bandwidth for very long time even for small changes.

The API provides batch operation endpoins for both record list and record change.

It would be great if the provider used these endpoints for refresh and update instead of iterating through the records one by one. That would likely speed up operations noticeably even for small deployments but would be several order of magnitude improvement for large deployments.

-refresh=false is a possible workaround for some use cases but introduces other problems like configuration drift

We in fact use a for_each loop in generating the resources but the example above causes the same issues.

References

No response

[github-actions]

Terraform debug log detected ✅

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[jacobbednarz]

this is expected behaviour from the provider when it runs into the rate limits. it will back off with jitter until the operation completes (even if it is hours).

there are a few options here:

• structure your Terraform configuration to separate them into smaller focus areas per the recommendation at https://developers.cloudflare.com/terraform/advanced-topics/best-practices/#directory-structure; or
• configure the max RPS of the client to be lower but will still complete faster as it will stay under the threshold (https://registry.terraform.io/providers/cloudflare/cloudflare/4.50.0/docs#rps-1); or
• talk to your account team about rate limit increase for these particular endpoints.

[jficz]

Unfortunately, restructuring won't help in our case, about 3500 records are in a single zone.

I understand this is expected with the current code base, that's why I suggested changing the approach to use batch processing API endpoints instead of doing each record separately and I already optimized the rate.

[jacobbednarz]

we won't swap to batch endpoints as terraform operates on a single resource model and it would be working against the intended design of terraform.