Add the second part of the multirack join service - #10989
Conversation
|
This still needs to be tested. |
6cc81ce to
238e6b3
Compare
|
Tested on voxel. Sled-agents come up after trust quorum. On On On |
238e6b3 to
8ba5f1c
Compare
This builds upon #10894 and adds support for starting sled agents after trust quorum completes. Combined, the two PRs implement 2/3 of the requirements for the first cut of the multirack join service described in #10637. The remaining part is to bring the front ports on line for serving DDM traffic and and announcing the new rack prefix over DDM.
8ba5f1c to
8f9463f
Compare
| match client.start_sled_agent(&info.req).await { | ||
| Ok(_) => Ok(info.baseboard_id), | ||
| Err(err) => { | ||
| // There really aren't any transient errors worth worrying |
There was a problem hiding this comment.
Hah, this comment speaks to the exact question I was going to ask. I'm not sure this is true - it seems like we could spuriously fail to connect for at least a few reasons that could clear up either on their own or with support intervention that wouldn't require clean-slating. (Couple examples we've seen before: network weather, spurious timeouts talking to an SP / RoT). It looks like RSS wraps its client.start_sled_agent() calls in infinite retry_notify() loops; that addresses the "how long do we wait?" question (forever) and seems better than failing on any error.
There was a problem hiding this comment.
Yeah, this is a fair criticism. I think it's reasonable to retry on some of these errors. We could wait indefinitely and report errors in output_rx I suppose.
There was a problem hiding this comment.
The code now retries indefinitely to start sled-agents, reporting its status in output_tx. I tried to use retry_notify, but the lifetime and type checking shenanigans made the code much less clear than the simple loop I ended up with in d5c2a5c
This builds upon #10894 and adds support for starting sled agents after trust quorum completes.
Combined, the two PRs implement 2/3 of the requirements for the first cut of the multirack join service described in #10637.
The remaining part is to bring the front ports on line for serving DDM traffic and and announcing the new rack prefix over DDM.