Error: Error deleting default network in project *****: googleapi: Error 403: This API method requires billing to be enabled.

[andrew-j-hagner]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.8.2
on ubuntu-latest

• provider registry.terraform.io/hashicorp/google v "~> 6.12.0"
• provider registry.terraform.io/hashicorp/google-beta v "~> 6.12.0"

Affected Resource(s)

• google_project

Terraform Configuration

terraform {
  required_version = ">= 1.8.2"
  backend "gcs" {
    bucket = "REDACTED"
    prefix = "terraform/REDACTED"
  }

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.12.0"
    }

    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 6.12.0"
    }
  }
}

provider "google" {
  region = "us-central1"
}

provider "google-beta" {
  region = "us-central1"
}

module "project_factory" {
  source  = "terraform-google-modules/project-factory/google"
  version = "~> 17.0"

  name                    = "REDACTED"
  auto_create_network     = false
  billing_account         = var.billing_account
  create_project_sa       = false
  default_service_account = "disable"
  folder_id               = REDACTED
  random_project_id       = true
  deletion_policy         = local.deletion_protection_enabled ? "PREVENT" : "DELETE"

  activate_apis = [
    "storage.googleapis.com",
    "run.googleapis.com",
    "workflows.googleapis.com",
    "bigquery.googleapis.com",
    "iap.googleapis.com"
  ]
}

Debug Output

Initial failure:

Error: Error deleting default network in project REDACTED: googleapi: Error 403: This API method requires billing to be enabled. Please enable billing on project #REDACTED by visiting https://console.developers.google.com/billing/enable?project=REDACTED then retry. If you enabled billing for this project recently, wait a few minutes for the action to propagate to our systems and retry.

Subsequent failure on re-try:

Error: error creating project REDACTED (REDACTED): googleapi: Error 409: Requested entity already exists, alreadyExists. If you received a 403 error, make sure you have the `roles/resourcemanager.projectCreator` permission

Expected Behavior

The project factory module should know that there are async tasks that GCP does during project creation and in turn wait for these to finish before trying to attempt any other operation that depends on those tasks (such as billing being enabled to use other GCP APIs).

In this example, it should wait for billing to be enable before attempting to delete the default network.

Actual Behavior

It does not wait and periodically (inconsistently) it will attempt to delete the default network before GCP finishes enabling the billing API for the new project.

Not only does it fail the apply, but it seems like terraform doesn't remember that the project was created successfully and subsequent applies will also fail as the project exists already.

Steps to reproduce

Since this seems to be a timing issue with GCP, there isn't a 100% reproduce procedure...

1. Use project factory and set auto_create_network to false
2. terraform apply
3. Destroy and repeat until you encounter the initial issue
4. Re-run failed GitHub Action/Workflow step to see the subsequent apply fail because it tries to create the project again.

Important Factoids

No response

References

This apparently was "fixed" in 2023:
#11028

b/390459382

[rileykarson]

Sent GoogleCloudPlatform/magic-modules#12779 to tune up the wait here, this is an eventual consistency issue and we're unable to tell for sure when the billing association has settled/propagated. I'm gonna let that close this bug, but if y'all are still seeing issues I'll reopen it. There's a more complete fix we could do- adding a proper retry loop instead of a single attempt- but I could immediately bump the timing so I opted for that for now.