[release-4.16] OCPBUGS-36489: parallelize member health checks #1290
Conversation
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-36301 has been cloned as Jira Issue OCPBUGS-36489. Will retitle bug to link to clone. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/severity-moderate
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-36489, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/invalid-bug
pkg/etcdcli/health.go
Outdated
| // closing here to avoid late replies to panic on resChan, | ||
| // the result will be considered a timeout anyway | ||
| close(resChan) | ||
| resChan <- checkSingleMemberHealth(ctx, member) |
There was a problem hiding this comment.
pkg/etcdcli/health.go:54:44: loop variable member captured by func literal
this one doesn't seem to pass the verify linter somehow, interesting.
https://issues.redhat.com/browse/OCPBUGS-36301 Currently, member health is checked in serial with a 30s timeout per member. 3 out of 4 GetMemberHealth callers had their own default 30s timeout as well for the entire process. Because of this, a slow check on one member could exhaust the timeout for the entire GetMemberHealth function, and thus cause later-checked members to report as unhealthy even though they were fine. With this commit, I am dropping the internal 30s timeout from GetMemberHealth, and instead letting the caller set the timeout. Also, the code now checks the health of all members in parallel. This will prevent a single slow member from affecting the health reporting of other members. I also added a timeout to the context used in IsMemberHealthy which calls GetMemberHealth. Neither Trevor nor I were sure why a default timeout wasn't present there, though one was present in all other callsites.
hasbro17
force-pushed
the
cherry-pick-1286-to-release-4.16
branch
from
8e86f81
to
94d3821
Compare
|
Added the patch to pass the member as an arg to the health check closure to prevent the member variable changing for each closure to be in compliance with the linter.
https://go.dev/doc/faq#closures_and_goroutines 4.17 is on Go 1.22 which has this issue fixed by default which changes loop variable to have per iteration scope so closure won't have the loop variables change on them. /approve |
openshift-ci
bot
added
the
approved
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hasbro17, openshift-cherrypick-robot, wking The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@openshift-cherrypick-robot: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/label backport-risk-assessed |
openshift-ci
bot
added
the
backport-risk-assessed
|
/jira refresh |
|
@tjungblu: This pull request references Jira Issue OCPBUGS-36489, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
This is an automated cherry-pick of #1286
/assign tjungblu
/cherrypick release-4.15 release-4.14 release-4.13 release-4.12