Add template for performance testing with custom Kubernetes version #5342

Jont828 · 2024-12-17T01:24:10Z

What type of PR is this?
/kind feature

What this PR does / why we need it:Add template for performance testing with custom Kubernetes version.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

TODOs:

squashed commits
includes documentation
adds unit tests
cherry-pick candidate

Release note:

NONE

codecov · 2024-12-17T02:35:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 52.43%. Comparing base (00dfab1) to head (77cfeb6).
Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5342   +/-   ##
=======================================
  Coverage   52.43%   52.43%           
=======================================
  Files         272      272           
  Lines       29401    29401           
=======================================
  Hits        15417    15417           
  Misses      13178    13178           
  Partials      806      806

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Jont828 · 2024-12-17T02:36:16Z

/test ls

k8s-ci-robot · 2024-12-17T02:36:19Z

@Jont828: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-azure-apiversion-upgrade
/test pull-cluster-api-provider-azure-build
/test pull-cluster-api-provider-azure-ci-entrypoint
/test pull-cluster-api-provider-azure-e2e
/test pull-cluster-api-provider-azure-e2e-aks
/test pull-cluster-api-provider-azure-test
/test pull-cluster-api-provider-azure-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-azure-apidiff
/test pull-cluster-api-provider-azure-apiserver-ilb
/test pull-cluster-api-provider-azure-capi-e2e
/test pull-cluster-api-provider-azure-conformance
/test pull-cluster-api-provider-azure-conformance-custom-builds
/test pull-cluster-api-provider-azure-conformance-dual-stack-with-ci-artifacts
/test pull-cluster-api-provider-azure-conformance-ipv6-with-ci-artifacts
/test pull-cluster-api-provider-azure-conformance-with-ci-artifacts
/test pull-cluster-api-provider-azure-e2e-optional
/test pull-cluster-api-provider-azure-e2e-workload-upgrade
/test pull-cluster-api-provider-azure-load-test-custom-builds
/test pull-cluster-api-provider-azure-windows-custom-builds
/test pull-cluster-api-provider-azure-windows-with-ci-artifacts

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-azure-apidiff
pull-cluster-api-provider-azure-build
pull-cluster-api-provider-azure-ci-entrypoint
pull-cluster-api-provider-azure-conformance
pull-cluster-api-provider-azure-conformance-custom-builds
pull-cluster-api-provider-azure-conformance-dual-stack-with-ci-artifacts
pull-cluster-api-provider-azure-conformance-ipv6-with-ci-artifacts
pull-cluster-api-provider-azure-conformance-with-ci-artifacts
pull-cluster-api-provider-azure-e2e
pull-cluster-api-provider-azure-e2e-aks
pull-cluster-api-provider-azure-test
pull-cluster-api-provider-azure-verify
pull-cluster-api-provider-azure-windows-custom-builds
pull-cluster-api-provider-azure-windows-with-ci-artifacts

In response to this:

/test ls

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Jont828 · 2024-12-17T03:01:31Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2024-12-17T03:59:26Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2024-12-17T04:45:29Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2024-12-17T20:15:10Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2024-12-18T21:06:49Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2024-12-18T21:53:43Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2024-12-18T23:33:12Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2024-12-19T03:39:55Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-03T22:39:28Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-03T23:00:33Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-06T17:10:36Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-06T21:20:52Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

k8s-ci-robot · 2025-01-06T23:36:16Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jont828. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Jont828 · 2025-01-07T00:54:05Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-08T00:45:48Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-08T19:54:49Z

We finally got a passing perf test run!

Jont828 · 2025-01-08T19:55:53Z

Running again to verify that it works during busier hours while other PR jobs are running simultaneously.

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-09T18:34:29Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

Jont828 · 2025-01-09T20:34:16Z

Bump timeout back to 30m and try again.

/test pull-cluster-api-provider-azure-load-test-custom-builds

mboersma · 2025-01-10T15:20:58Z

scripts/aks-as-mgmt.sh

@@ -1,4 +1,4 @@
-#!/usr/bin/env bash
+1#!/usr/bin/env bash


Typo

Suggested change

1#!/usr/bin/env bash

#!/usr/bin/env bash

mboersma · 2025-01-10T15:21:43Z

scripts/aks-as-mgmt.sh

@@ -260,4 +260,4 @@ rm -r combined_contexts.yaml
 rm -f tilt-settings-temp.yaml
 }

-main
+main


Could you add back the trailing newline? That seems to be preferred in this codebase.

mboersma · 2025-01-10T15:25:01Z

Makefile

@@ -377,7 +377,7 @@ create-workload-cluster: $(ENVSUBST) $(KUBECTL) ## Create a workload cluster.
 	timeout --foreground 1800 bash -c "while ! $(KUBECTL) get secrets -n default | grep $(CLUSTER_NAME)-kubeconfig; do sleep 1; done"
 	# Get kubeconfig and store it locally.
 	$(KUBECTL) get secret/$(CLUSTER_NAME)-kubeconfig -n default -o json | jq -r .data.value | base64 --decode > ./kubeconfig
-	$(KUBECTL) -n default wait --for=condition=Ready --timeout=10m cluster "$(CLUSTER_NAME)"
+	$(KUBECTL) -n default wait --for=condition=Ready --timeout=30m cluster "$(CLUSTER_NAME)"


Is it worth parameterizing this, like --timeout=$(CLUSTER_TIMEOUT), so we can leave the default at 10m? This is a popular Makefile target and this may change the overall duration of e2e test failures, or surprise users who were accustomed to failure after 10 minutes.

mboersma · 2025-01-10T15:29:25Z

templates/test/dev/cluster-template-custom-builds-load.yaml

+        # Run the az login command with managed identity
+        if az login --identity > /dev/null 2>&1; then
+          echo "Logged in Azure with managed identity"
+          echo "Use OOT credential provider"


Suggested change

echo "Use OOT credential provider"

echo "Using OOT credential provider"

Maybe? Seems more consistent with other log messages here.

mboersma · 2025-01-10T15:31:16Z

templates/test/dev/cluster-template-custom-builds-load.yaml

+          echo "Logged in Azure with managed identity"
+          echo "Use OOT credential provider"
+          mkdir -p /var/lib/kubelet/credential-provider
+          az storage blob download --blob-url "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_BLOB_CONTAINER_NAME}/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider" -f /var/lib/kubelet/credential-provider/acr-credential-provider --auth-mode login


Maybe set https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_BLOB_CONTAINER_NAME}/${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER} to a local env var so we make the az storage blob and curl commands a bit more readable?

mboersma · 2025-01-10T15:35:50Z

templates/test/dev/cluster-template-custom-builds-load.yaml

      osDisk:
        diskSizeGB: 128
        osType: Linux
      sshPublicKey: ${AZURE_SSH_PUBLIC_KEY_B64:=""}
      userAssignedIdentities:
-      - providerID: /subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${CI_RG:=capz-ci}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${USER_IDENTITY:=cloud-provider-user-identity}


Why do we not want to include default values here?

mboersma · 2025-01-10T15:38:34Z

templates/test/dev/cluster-template-custom-builds-load.yaml

@@ -183,12 +282,18 @@ spec:
  template:
    spec:
      identity: UserAssigned
+      image:
+        marketplace:


The Azure Marketplace images are deprecated and won't be updated. Azure Community Gallery is the new hotness:

image: computeGallery: gallery: "ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019" name: "capi-ubun2-2404" version: "latest"

mboersma · 2025-01-10T15:39:14Z

templates/test/dev/cluster-template-custom-builds-load.yaml

+          set -o nounset
+          set -o pipefail
+          set -o errexit
+          [[ $(id -u) != 0 ]] && SUDO="sudo" || SUDO=""


Same script, same comment: I don't think sudo is used here.

mboersma · 2025-01-10T15:41:30Z

templates/test/dev/cluster-template-custom-builds-load.yaml

@@ -261,14 +420,20 @@ spec:
        windowsServerVersion: ${WINDOWS_SERVER_VERSION:=""}
    spec:
      identity: UserAssigned
+      image:
+        marketplace:


And for Windows images in the Community Gallery:

image: computeGallery: gallery: "ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019" name: "capi-win-2022-containerd" version: "latest"

windows-2019 is now deprecated and we won't be publishing it any more: see #5357.

mboersma · 2025-01-10T15:43:34Z

templates/test/dev/cluster-template-custom-builds-load.yaml

+        path: C:/oot-cred-provider.ps1
+        permissions: "0744"
+      - content: |
+          Write-Host "Installing Azure CLI"


If you're using CAPZ reference images built with --var 'debug_tools=true' (our default), then az should already be installed on Windows and Linux nodes.

jackfrancis · 2025-01-10T19:23:50Z

/test pull-cluster-api-provider-azure-load-test-custom-builds

k8s-ci-robot · 2025-01-10T22:11:19Z

@Jont828: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-azure-verify	`77cfeb6`	link	true	`/test pull-cluster-api-provider-azure-verify`
pull-cluster-api-provider-azure-load-test-custom-builds	`77cfeb6`	link	false	`/test pull-cluster-api-provider-azure-load-test-custom-builds`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot requested review from marosset and nojnhuh December 17, 2024 01:24

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 17, 2024

Jont828 mentioned this pull request Dec 17, 2024

[WIP] Set up ci-entrypoint to work with AKS management cluster #5241

Open

4 tasks

Jont828 force-pushed the entrypoint-custom-builds branch 2 times, most recently from f60ccc6 to 007ef8e Compare December 17, 2024 02:25

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 17, 2024

Jont828 force-pushed the entrypoint-custom-builds branch 2 times, most recently from 5a0efc2 to e05f164 Compare December 17, 2024 02:59

Jont828 force-pushed the entrypoint-custom-builds branch from dd8b409 to d321deb Compare December 17, 2024 18:04

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 3, 2025

Jont828 force-pushed the entrypoint-custom-builds branch from 4e15c40 to 0a0569d Compare January 3, 2025 22:34

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 3, 2025

Jont828 force-pushed the entrypoint-custom-builds branch 2 times, most recently from 7550b03 to 365ccc0 Compare January 9, 2025 18:33

Jont828 changed the title ~~[WIP] Add template for performance testing with custom Kubernetes version~~ Add template for performance testing with custom Kubernetes version Jan 10, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 10, 2025

Jont828 force-pushed the entrypoint-custom-builds branch 2 times, most recently from c865b5c to e1c4aef Compare January 10, 2025 01:54

Add custom K8s template for running clusterloader2 performance test

77cfeb6

Jont828 force-pushed the entrypoint-custom-builds branch from e1c4aef to 77cfeb6 Compare January 10, 2025 01:55

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 10, 2025

mboersma reviewed Jan 10, 2025

View reviewed changes

mboersma requested review from jackfrancis and removed request for nojnhuh and marosset January 10, 2025 15:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add template for performance testing with custom Kubernetes version #5342

Add template for performance testing with custom Kubernetes version #5342

Jont828 commented Dec 17, 2024

codecov bot commented Dec 17, 2024 •

edited

Loading

Jont828 commented Dec 17, 2024

k8s-ci-robot commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 18, 2024

Jont828 commented Dec 18, 2024

Jont828 commented Dec 18, 2024

Jont828 commented Dec 19, 2024

Jont828 commented Jan 3, 2025

Jont828 commented Jan 3, 2025

Jont828 commented Jan 6, 2025

Jont828 commented Jan 6, 2025

k8s-ci-robot commented Jan 6, 2025

Jont828 commented Jan 7, 2025

Jont828 commented Jan 8, 2025

Jont828 commented Jan 8, 2025

Jont828 commented Jan 8, 2025

Jont828 commented Jan 9, 2025

Jont828 commented Jan 9, 2025

mboersma Jan 10, 2025

mboersma Jan 10, 2025

mboersma Jan 10, 2025 •

edited

Loading

mboersma Jan 10, 2025

mboersma Jan 10, 2025

mboersma Jan 10, 2025

mboersma Jan 10, 2025 •

edited

Loading

mboersma Jan 10, 2025

mboersma Jan 10, 2025

mboersma Jan 10, 2025

jackfrancis commented Jan 10, 2025

k8s-ci-robot commented Jan 10, 2025

	echo "Use OOT credential provider"
	echo "Using OOT credential provider"

Add template for performance testing with custom Kubernetes version #5342

Are you sure you want to change the base?

Add template for performance testing with custom Kubernetes version #5342

Conversation

Jont828 commented Dec 17, 2024

codecov bot commented Dec 17, 2024 • edited Loading

Codecov Report

Jont828 commented Dec 17, 2024

k8s-ci-robot commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 17, 2024

Jont828 commented Dec 18, 2024

Jont828 commented Dec 18, 2024

Jont828 commented Dec 18, 2024

Jont828 commented Dec 19, 2024

Jont828 commented Jan 3, 2025

Jont828 commented Jan 3, 2025

Jont828 commented Jan 6, 2025

Jont828 commented Jan 6, 2025

k8s-ci-robot commented Jan 6, 2025

Jont828 commented Jan 7, 2025

Jont828 commented Jan 8, 2025

Jont828 commented Jan 8, 2025

Jont828 commented Jan 8, 2025

Jont828 commented Jan 9, 2025

Jont828 commented Jan 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mboersma Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mboersma Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackfrancis commented Jan 10, 2025

k8s-ci-robot commented Jan 10, 2025

codecov bot commented Dec 17, 2024 •

edited

Loading

mboersma Jan 10, 2025 •

edited

Loading

mboersma Jan 10, 2025 •

edited

Loading