Failing test(s): TestAccSecurityCenterV2ProjectBigQueryExportConfig_basic + others

[SarahFrench]

Impacted tests

• TestAccSecurityCenterV2ProjectBigQueryExportConfig_basic
• TestAccSecurityCenterV2OrganizationBigQueryExportConfig_basic
• TestAccSecurityCenterV2OrganizationBigQueryExportsConfig_basic
• TestAccSecurityCenterOrganizationBigQueryExportConfig_basic
• TestAccSecurityCenterProjectBigQueryExportConfig_basic

Affected Resource(s)

• google_scc_v2_project_scc_big_query_export
• google_scc_v2_organization_scc_big_query_export

Failure rates

• 51% since 2024-08-29
• 93% (53 failed out of last 57 runs) as of 2024-11-08 for TestAccSecurityCenterOrganizationBigQueryExportConfig_basic

Message(s)

------- Stdout: -------
=== RUN   TestAccSecurityCenterV2ProjectBigQueryExportConfig_basic
=== PAUSE TestAccSecurityCenterV2ProjectBigQueryExportConfig_basic
=== CONT  TestAccSecurityCenterV2ProjectBigQueryExportConfig_basic
    testing_new.go:90: Error running post-test destroy, there may be dangling resources: exit status 1
        Error: Error when reading or editing Dataset: googleapi: Error 400: Dataset ci-test-project-nightly-ga:tf_test_fbn2dql0rc is still in use, resourceInUse
--- FAIL: TestAccSecurityCenterV2ProjectBigQueryExportConfig_basic (210.52s)
FAIL

Nightly build test history

https://hashicorp.teamcity.com/test/2715200116441949818?currentProjectId=TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS&expandTestHistoryChartSection=true&expandedTest=build%3A%28id%3A249228%29%2Cid%3A2000000012

b/375653701

[wj-chen]

Hi Sarah, do you have more details about the tests to show? We don't have access to the test history and I can't seem to find the test in the repo by searching by this name.

[SarahFrench]

Here's are the tests in the code:

terraform-provider-google/google/services/securitycenterv2/resource_scc_v2_project_big_query_export_config_test.go

Line 14 in a27d219

func TestAccSecurityCenterV2ProjectBigQueryExportConfig_basic(t *testing.T) {

terraform-provider-google/google/services/securitycenterv2/resource_scc_v2_organization_big_query_export_config_test.go

Line 14 in a27d219

func TestAccSecurityCenterV2OrganizationBigQueryExportConfig_basic(t *testing.T) {

terraform-provider-google/google/services/securitycenterv2/resource_scc_v2_organization_big_query_exports_config_test.go

Line 14 in a27d219

func TestAccSecurityCenterV2OrganizationBigQueryExportsConfig_basic(t *testing.T) {

I don't have more details about the tests to show, but I believe there is a Google-internal way that nightly test logs are shared with people from the service team. I don't know the details myself- it might be worth checking with the Google Terraform team.

[wj-chen]

Thank you Sarah. I took a look and these tests should be about the google_scc_v2_organization_scc_big_query_export resource which is owned by the SCC team. The tests use google_bigquery_dataset in the setup and for some reasons are not cleaning it up consistently. Adding @roaks3 the terraform-team-oncall this week to help reroute the issue here and in our internal issue tracker.

[melinath]

Added a couple more tests impacted by this issue.

[rileykarson]

@vijaykanthm: Do you happen to know the trigger for this error? Is this a lock on the resource from SCC that we need to wait to go away before deletions, are we failing to clean up a pointer/reference to the resource by abandoning the resource?

[vijaykanthm]

@vijaykanthm: Do you happen to know the trigger for this error? Is this a lock on the resource from SCC that we need to wait to go away before deletions, are we failing to clean up a pointer/reference to the resource by abandoning the resource?

I’m not entirely sure about the exact cause, as I don't have access to the TeamCity logs and I've learned that I won’t be able to access them. However, I believe the issue might be due to the dataset still being referenced or locked by SCC, which would prevent its deletion. We might need to add some additional wait time to ensure the lock is released before cleanup.

As I mentioned in our discussion, the reason I wanted to implement the cleanup function was to remove references to existing resources before test execution, in order to avoid this type of issue.

[rileykarson]

Yep! To clarify a bit more- we need to make sure we run generic/broad resource cleanups outside of individual tests so that we can ensure that they don't execute at the same time as other tests that may be using resources that could be affected. So a test can clean up its own resources under test at the end, but not those of other tests- and those other resources need to be cleaned up through a "sweeper". Not sure why they haven't been effective here.