[release-1.20] Define NoSchedule tolerations for tigera-operator

[nojnhuh]

What type of PR is this?
/kind flake

What this PR does / why we need it:

Manual cherry-pick of #5704:

This PR changes the tigera-operator Pod to tolerate only the NoSchedule tolerations that are set when a Node is being bootstrapped instead of all of them (which includes the node.kubernetes.io/unschedulable taint set when a Node is cordoned and drained).

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Ref #5703

Special notes for your reviewer:

TODOs:

• squashed commits
• includes documentation
• adds unit tests
• cherry-pick candidate

Release note:

NONE

[jackfrancis]

this would have been better asked in the original PR, but should we add a comment describing why we want to tolerate all NoExecute taints?

[nojnhuh]

That toleration was already set by default, so I was only trying to change as few things as possible to get around the flake.

[jackfrancis]

is there a bug in the comment

By default, tigera tolerates all NoSchedule taints

should it be

By default, tigera tolerates all NoExecute taints

?

[nojnhuh]

Both are true. I was trying to call out the delta between what tigera sets by default and what we need to change it to.

[jackfrancis]

good talk

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.80%. Comparing base (58587ac) to head (f96822a).
⚠️ Report is 3 commits behind head on release-1.20.

Additional details and impacted files

@@               Coverage Diff                @@
##           release-1.20    #5890      +/-   ##
================================================
- Coverage         52.81%   52.80%   -0.01%     
================================================
  Files               279      279              
  Lines             29602    29602              
================================================
- Hits              15634    15632       -2     
- Misses            13154    13156       +2     
  Partials            814      814              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jackfrancis]

/lgtm
/approve

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: a6ca95b963978bb3d8bc7430cf8f6b1c12d659f7

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nojnhuh]

/retest