CIF - ServerError: ZonalAllocationFailed #3691

rhamitarora · 2024-07-15T06:07:20Z

Which issue this PR addresses:

What this PR does / why we need it:

Following the pattern established in the old PR #3116 for handling Hive installation failures in the RP, this PR addresses a specific case (Azure ZonalAllocationFailed) by forwarding the policy errors to the requester. The sequence of patterns in the Hive ConfigMap ensures that specific errors take precedence over this general one.

Test plan for issue:

With this code running on e.g. a local RP setup that installs via Hive, with the included additional-install-log-regexes ConfigMap applied in the Hive cluster:

Create a policy that blocks e.g. creation of the bootstrap/master VMs created by the installer
Attempt a cluster creation
Unit tests have been added for the specific Hive install failure cases implemented within the PR.

Is there any documentation that needs to be updated for this PR?

Hive Cluster Provision Pod Not Ready SOP (todo)

How do you know this will function as expected in production?

bitoku · 2024-07-15T10:59:58Z

/azp run ci,e2e

azure-pipelines · 2024-07-15T11:00:12Z

Azure Pipelines successfully started running 2 pipeline(s).

bitoku · 2024-07-15T11:03:39Z

pkg/hive/failure/reasons_test.go

+		{
+			name: "ZonalAllocationFailed",
+			installLog: `Message: level=info msg=creating InstanceMetadata from Azure Instance Metadata Service (AIMS) level=info msg=InstanceMetadata: running on AzurePublicCloud level=info msg=running step [Action github.com/openshift/ARO-Installer/pkg/installer.(*manager).Manifests.func1] level=info msg=running step [Action github.com/openshift/ARO-Installer/pkg/installer.(*manager).Manifests.func2] level=info msg=resolving graph level=info
+						 msg=running step [Action github.com/openshift/ARO-Installer/pkg/installer.(*manager).Manifests.func3] level=info msg=checking if graph exists level=info msg=save graph Generates the Ignition Config asset level=info msg=creating InstanceMetadata from Azure Instance Metadata Service (AIMS) level=info msg=InstanceMetadata: running on AzurePublicCloud level=info
+						 msg=running step [AuthorizationRetryingAction github.com/openshift/ARO-Installer/pkg/installer.(*manager).deployResourceTemplate-fm] level=info msg=load persisted graph level=info msg=deploying resources template level=error msg=step [AuthorizationRetryingAction github.com/openshift/ARO-Installer/pkg/installer.(*manager).deployResourceTemplate-fm]
+						 encountered error: 400: DeploymentFailed: : Deployment failed. Details: : : {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","target":null,"details":
+						 [{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\":
+						 [\r\n {\r\n \"code\": \"ZonalAllocationFailed\",\r\n \"message\": \"Allocation failed. We do not have sufficient capacity for the requested VM size in this zone. Read more about improving likelihood of allocation success at http://aka.ms/allocation-guidance\"\r\n }\r\n ]\r\n }\r\n}"}],"innererror":null,"additionalInfo":null}
+						 level=error msg=400: DeploymentFailed: : Deployment failed. Details: : : {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","target":null,"details":
+						 [{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\":
+						 [\r\n {\r\n \"code\": \"ZonalAllocationFailed\",\r\n \"message\": \"Allocation failed. We do not have sufficient capacity for the requested VM size in this zone. Read more about improving likelihood of allocation success at http://aka.ms/allocation-guidance\"\r\n }\r\n ]\r\n }\r\n}"}],"innererror":null,"additionalInfo":null}`,
+			want: AzureZonalAllocationFailed,
+		},


I wonder why the escaping rule is different from other tests.
Is either one wrong probably?

No other's are not wrong, the Installlog data is what we get from hive pod log I run test case on that raw data for more accuracy. This log data also attached in JIRA.

I think these need to be consistent. The logs will be emitted the same way by Hive regardless of error type. We should confirm that the same regex format applies to any permutation of the logs (whether they have explicit newlines or literal \n or \r\n within them, escaped quotes or not, etc)

Multiple test cases added to confirm regex format with or without \n or \r\n

tsatam

Change looks good, we should validate that the regexes match against the log format we get from Hive.

I'd suggest testing this by applying the Hive ConfigMap to either a dev or full-service Hive cluster, and validating that the behavior is as expected.

tsatam · 2024-07-15T20:16:00Z

pkg/hive/failure/reasons_test.go

+		{
+			name: "ZonalAllocationFailed",
+			installLog: `Message: level=info msg=creating InstanceMetadata from Azure Instance Metadata Service (AIMS) level=info msg=InstanceMetadata: running on AzurePublicCloud level=info msg=running step [Action github.com/openshift/ARO-Installer/pkg/installer.(*manager).Manifests.func1] level=info msg=running step [Action github.com/openshift/ARO-Installer/pkg/installer.(*manager).Manifests.func2] level=info msg=resolving graph level=info
+						 msg=running step [Action github.com/openshift/ARO-Installer/pkg/installer.(*manager).Manifests.func3] level=info msg=checking if graph exists level=info msg=save graph Generates the Ignition Config asset level=info msg=creating InstanceMetadata from Azure Instance Metadata Service (AIMS) level=info msg=InstanceMetadata: running on AzurePublicCloud level=info
+						 msg=running step [AuthorizationRetryingAction github.com/openshift/ARO-Installer/pkg/installer.(*manager).deployResourceTemplate-fm] level=info msg=load persisted graph level=info msg=deploying resources template level=error msg=step [AuthorizationRetryingAction github.com/openshift/ARO-Installer/pkg/installer.(*manager).deployResourceTemplate-fm]
+						 encountered error: 400: DeploymentFailed: : Deployment failed. Details: : : {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","target":null,"details":
+						 [{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\":
+						 [\r\n {\r\n \"code\": \"ZonalAllocationFailed\",\r\n \"message\": \"Allocation failed. We do not have sufficient capacity for the requested VM size in this zone. Read more about improving likelihood of allocation success at http://aka.ms/allocation-guidance\"\r\n }\r\n ]\r\n }\r\n}"}],"innererror":null,"additionalInfo":null}
+						 level=error msg=400: DeploymentFailed: : Deployment failed. Details: : : {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","target":null,"details":
+						 [{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\":
+						 [\r\n {\r\n \"code\": \"ZonalAllocationFailed\",\r\n \"message\": \"Allocation failed. We do not have sufficient capacity for the requested VM size in this zone. Read more about improving likelihood of allocation success at http://aka.ms/allocation-guidance\"\r\n }\r\n ]\r\n }\r\n}"}],"innererror":null,"additionalInfo":null}`,
+			want: AzureZonalAllocationFailed,
+		},


I think these need to be consistent. The logs will be emitted the same way by Hive regardless of error type. We should confirm that the same regex format applies to any permutation of the logs (whether they have explicit newlines or literal \n or \r\n within them, escaped quotes or not, etc)

tsatam · 2024-07-15T20:17:57Z

hack/hive-config/hive-additional-install-log-regexes.yaml

+      installFailingReason: AzureZonalAllocationFailed
+      name: AzureZonalAllocationFailed
+      searchRegexStrings:
+      - '"code\\":\W\\"ZonalAllocationFailed\\"'


Same feedback as on the unit test below - we should ensure this, as well as other search regex strings, are compatible with either "format" of escaped logs we get, rather than building an implementation that assumes just one way. Thankfully, we can pass multiple regex strings in for each failing reason here so if we need to just duplicate the regex strings instead of building a single pattern that matches both, we can.

Regex format updated, added 3 different test cases to match multiple regex strings.

mociarain

Nice changes

mociarain · 2024-07-24T14:08:07Z

/azp run ci

azure-pipelines · 2024-07-24T14:08:16Z

Azure Pipelines successfully started running 1 pipeline(s).

rhamitarora marked this pull request as ready for review July 15, 2024 07:36

rhamitarora added ready-for-review go Pull requests that update Go code firefly Issues or Pull requests owned by Team Firefly labels Jul 15, 2024

bitoku reviewed Jul 15, 2024

View reviewed changes

rhamitarora requested a review from bitoku July 15, 2024 11:30

bitoku previously approved these changes Jul 15, 2024

View reviewed changes

tsatam requested changes Jul 15, 2024

View reviewed changes

tsatam mentioned this pull request Jul 15, 2024

CIF: OSProvisioningTimedOut #3677

Open

CIF - ServerError: ZonalAllocationFailed

49af5c6

rhamitarora dismissed bitoku’s stale review via 49af5c6 July 16, 2024 05:49

rhamitarora force-pushed the zonalallocationfailed branch from daf27cd to 49af5c6 Compare July 16, 2024 05:49

rhamitarora requested review from bitoku and tsatam July 16, 2024 05:55

bitoku approved these changes Jul 17, 2024

View reviewed changes

SudoBrendan approved these changes Jul 17, 2024

View reviewed changes

mociarain approved these changes Jul 22, 2024

View reviewed changes

tsatam approved these changes Jul 24, 2024

View reviewed changes

mociarain merged commit eefe6e8 into master Jul 24, 2024
20 checks passed

mociarain deleted the zonalallocationfailed branch July 24, 2024 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CIF - ServerError: ZonalAllocationFailed #3691

CIF - ServerError: ZonalAllocationFailed #3691

rhamitarora commented Jul 15, 2024 •

edited

Loading

bitoku commented Jul 15, 2024

azure-pipelines bot commented Jul 15, 2024

bitoku Jul 15, 2024

rhamitarora Jul 15, 2024 •

edited

Loading

tsatam Jul 15, 2024

rhamitarora Jul 16, 2024

tsatam left a comment

tsatam Jul 15, 2024

tsatam Jul 15, 2024

rhamitarora Jul 16, 2024

mociarain left a comment

mociarain commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

CIF - ServerError: ZonalAllocationFailed #3691

CIF - ServerError: ZonalAllocationFailed #3691

Conversation

rhamitarora commented Jul 15, 2024 • edited Loading

Which issue this PR addresses:

What this PR does / why we need it:

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

How do you know this will function as expected in production?

bitoku commented Jul 15, 2024

azure-pipelines bot commented Jul 15, 2024

bitoku Jul 15, 2024

Choose a reason for hiding this comment

rhamitarora Jul 15, 2024 • edited Loading

Choose a reason for hiding this comment

tsatam Jul 15, 2024

Choose a reason for hiding this comment

rhamitarora Jul 16, 2024

Choose a reason for hiding this comment

tsatam left a comment

Choose a reason for hiding this comment

tsatam Jul 15, 2024

Choose a reason for hiding this comment

tsatam Jul 15, 2024

Choose a reason for hiding this comment

rhamitarora Jul 16, 2024

Choose a reason for hiding this comment

mociarain left a comment

Choose a reason for hiding this comment

mociarain commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

rhamitarora commented Jul 15, 2024 •

edited

Loading

rhamitarora Jul 15, 2024 •

edited

Loading