Bottlerocket Operator always picking up the same node

[landbaychrisburrell]

Hi

I've got an issue where the operator is continually picking up the same node to upgrade, but that node is already upgraded:

2023-12-06T15:10:00.001278Z INFO controller::controller: Found associated bottlerocketshadow name., associated_bottlerocketshadow_name: "brs-ip-10-0-1-2.eu-west-1.compute.internal"
at controller/src/controller.rs:365
in controller::controller::get_associated_bottlerocketshadow_name
in controller::controller::find_and_update_ready_brs

And the same message, with the same host gets repeated each time.

The brs look like this - with the 10-0-1-2 node above already showing upgraded somehow

NAME STATE VERSION TARGET STATE TARGET VERSION CRASH COUNT
brs-ip-10-0-1-1.eu-west-1.compute.internal Idle 1.16.0 Idle 1.16.1 8
brs-ip-10-0-1-2.eu-west-1.compute.internal Idle 1.16.1 Idle 0
brs-ip-10-0-1-3.eu-west-1.compute.internal Idle 1.16.0 Idle 1.16.1 8
brs-ip-10-0-1-4.eu-west-1.compute.internal Idle 1.16.0 Idle 1.16.1 8

Any ideas on how to unblock this? I tried to delete the agents and tried to delete the BRS but neither has helped.

Cheers
Chris

[jpculp]

Thanks for reaching out! It looks like your crash count for the other 3 nodes is 8. The behavior you're seeing on brs-ip-10-0-1-2.eu-west-1.compute.internal is just the normal periodic update check. Would you be able to share some of the logs from one of the nodes where the updater is crashing?

[jpculp]

It might be worth moving this discussion to an issue on the bottlerocket-update-operator repo.