Multirack: On new racks Oximeter should not try to register with Nexus until told to do so

[andrewjstone]

On single rack bringup, as soon as sled-agent starts, Oximeter tries to register with Nexus. This makes sense, as handoff has occured after RSS and Oximeter will be allowed to discover Nexus.

However on a new rack, when multirack join starts sled-agent, sled-agents are not supposed to be doing "active work". Sled-agents don't register themselves and instead wait for Nexus to discover them and only add them to the sled table in CRDB on an operator's command. Oximeter registration should wait for this point.

We can see that this is currently not the case when performing a multirack join command on a new rack in voxel with the code in #10989:

q16:59:57.206Z WARN SledAgent: failed to register with Nexus, will retry
    count = 17
    delay = 1486.727669497s
    error = proto error: io error: No route to host (os error 148)
    file = oximeter/producer/src/lib.rs:374
    sled_id = 964383a1-fbe0-4b6a-9ba3-ce1a4880d5c1

We don't yet have the two racks connected via DDM over the front ports and so Oximeter consistently fails to register. This is fine, as it's just a nuisance in the logs. However, once we connect the racks, Oximeter will be able to find Nexus and register itself even though the sled-agents on the new rack aren't part of the multirack cluster yet.

[jgallagher]

I think this same race exists with RSS too; sled-agent starts trying to register with Nexus very early in RSS (long before Nexus is actually up). I don't know if there's a window where Nexus could allow this to succeed before the sled-agent gets itself added to the sled table in practice, though.