Upgrade masters last when upgrading ES clusters

[naemono]

Fixes #8429

What is changing?

This ensure that the master StatefulSets are always upgraded last when doing a version upgrade of Elasticsearch.

Validation

1. Manual testing
2. Unit test for behavior
3. e2e test validating behavior

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[naemono]

buildkite test this -f p=kind,t=TestNonMasterFirstUpgradeComplexTopology -m s=9.1.2

[naemono]

buildkite test this -f p=kind,t=TestHandleUpscaleAndSpecChanges_VersionUpgradeDataFirstFlow -m s=9.1.2

[Copilot]

Pull Request Overview

This PR implements a non-master-first upgrade strategy for Elasticsearch clusters. The key change ensures that during version upgrades, non-master nodes (data, ingest, coordinating nodes) are upgraded before master nodes, which helps maintain cluster stability during upgrades.

• Adds logic to separate master and non-master StatefulSets during version upgrades
• Implements upgrade order validation to ensure non-master nodes complete their upgrades first
• Adds comprehensive unit and e2e tests to verify the upgrade flow

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
pkg/controller/elasticsearch/driver/upgrade.go	Adds check to identify new clusters vs upgrades by checking if status version is empty
pkg/controller/elasticsearch/driver/upscale.go	Implements non-master-first upgrade logic with resource separation and upgrade status checking
pkg/controller/elasticsearch/driver/upscale_test.go	Adds comprehensive unit test for version upgrade flow and minor formatting fixes
test/e2e/es/non_master_first_upgrade_test.go	Adds e2e test that validates non-master-first upgrade behavior with a watcher

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[barkbay]

I will take another look at it today.

Edit: was not able to make it today, will do tomorrow first thing

[barkbay]

I just realized that calling this when len(nonMasterResources) == 0 (or more generally, when all non-master nodesets have already been upgraded?) can be slightly suboptimal.

Assuming that the initial state is:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 9.1.0
  nodeSets:
  - name: default
    config:
      node.roles: ["master", "data", "ingest", "ml"]
      node.store.allow_mmap: false
    count: 3

If we update and upgrade to:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 9.1.2
  nodeSets:
  - name: default
    config:
      node.roles: ["master", "data", "ingest", "ml"]
      node.store.allow_mmap: false
    count: 4

Then we are going to scale up the 9.1.0 statefulset, leading to the creation of elasticsearch-sample-es-default-3, but immediately in the next reconciliation we are going to delete elasticsearch-sample-es-default-3 to upgrade it to 9.1.2

[barkbay]

My previous comment made me wonder if !isVersionUpgrade is actually the only reason we might want to reconcile everything at once.

[naemono]

Is this 4b966db what are were intending from this comment @barkbay ?

[barkbay]

buildkite test this -f p=kind,t=TestNonMasterFirstUpgradeComplexTopology -m s=8.15.2

[barkbay]

Will this test fail if we consistently get an error here? (my feeling is that it's not going to be the case because violations is always empty in that case, but maybe I'm missing something)

[naemono]

No it's not. I wonder what would be the limit we set on errors in the watcher where it would fail? There are 4 instances I can find where we don't fail in these watchers in the e2e tests currently:

cloud-on-k8s/test/e2e/kb/version_upgrade_test.go

Line 130 in b43765b

t.Logf("got error: %v", err)

cloud-on-k8s/test/e2e/kb/version_upgrade_test.go

Line 240 in b43765b

t.Logf("got error: %v", err)

cloud-on-k8s/test/e2e/kb/version_upgrade_test.go

Line 275 in b43765b

t.Logf("failed to list pods: %v", err)

cloud-on-k8s/test/e2e/test/elasticsearch/checks_budget.go

Line 33 in b43765b

t.Logf("got error listing pods: %v", err)

[barkbay]

Almost LGTM, I think we need to adjust the way we scale the master nodes, also the e2e test seems broken (we create the data integrity index with no replicas, which should fail during a rolling upgrade), and may not be accurate in case of errors.

[naemono]

buildkite test this -f p=kind,t=TestNonMasterFirstUpgradeComplexTopology -m s=8.15.2