Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Smart Switch reboot high level design #1699

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Smart Switch reboot high level design #1699

wants to merge 13 commits into from
+400 −0

Conversation

vvolam
Copy link

@vvolam vvolam commented May 16, 2024 •
edited
Loading

This PR is for smart switch reboot high-level design

Repo Pull Request Status
sonic-gnmi sonic-net/sonic-gnmi#286 Open
sonic-host-services sonic-net/sonic-host-services#164 Open
sonic-platform-common sonic-net/sonic-platform-common#501 Merged
sonic-platform-common sonic-net/sonic-platform-common#504 Open
sonic-utilities sonic-net/sonic-utilities#3566 Draft
sonic-platform-daemons sonic-net/sonic-platform-daemons#546 Draft

@vvolam vvolam marked this pull request as ready for review May 16, 2024 23:15
@isabelmsft isabelmsft self-requested a review May 20, 2024 23:31
isabelmsft
isabelmsft reviewed May 21, 2024
View reviewed changes
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
@vvolam vvolam requested review from qiluo-msft and ganglyu May 30, 2024 15:15
rameshraghupathy
rameshraghupathy reviewed May 30, 2024
View reviewed changes
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Show resolved Hide resolved
isabelmsft
isabelmsft reviewed Jun 11, 2024
View reviewed changes
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
doc/smart-switch/reboot/reboot-hld.md Outdated Show resolved Hide resolved
@oleksandrivantsiv oleksandrivantsiv mentioned this pull request Jun 14, 2024
Merged
5 tasks
ganglyu
ganglyu previously approved these changes Jun 17, 2024
View reviewed changes
isabelmsft
isabelmsft previously approved these changes Jun 18, 2024
View reviewed changes
@oleksandrivantsiv oleksandrivantsiv mentioned this pull request Jun 25, 2024
Open
@vvolam vvolam dismissed stale reviews from isabelmsft and ganglyu via 1934915 June 26, 2024 19:00
oleksandrivantsiv
oleksandrivantsiv suggested changes Jun 28, 2024
View reviewed changes
Copy link
Contributor

@oleksandrivantsiv oleksandrivantsiv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As commented

@vvolam vvolam force-pushed the reboot-hld branch 2 times, most recently from 1c9a020 to 7d67e25 Compare July 30, 2024 01:16
@vvolam vvolam mentioned this pull request Sep 11, 2024
Open
@KrisNey-MSFT
Copy link

Discussed in DASH Community call 9/18/2024
If the DPU is unresponsive and we are trying to recover it, is there a way to hard power cycle a DPU w/o having to power cycle the switch?
Via PCIE express lanes, CPLD, or other?
Force-shut or force-reboot the card (w/o forcing the entire switch), and will it be standardized or supplier-specific?
@prgeor

@prgeor
Copy link
Contributor

prgeor commented Sep 18, 2024

@vvolam

Discussed in DASH Community call 9/18/2024 If the DPU is unresponsive and we are trying to recover it, is there a way to hard power cycle a DPU w/o having to power cycle the switch? Via PCIE express lanes, CPLD, or other? Force-shut or force-reboot the card (w/o forcing the entire switch), and will it be standardized or supplier-specific? @prgeor

@vvolam FYI

@prgeor
Copy link
Contributor

prgeor commented Sep 18, 2024

@vvolam please add all the code PRs to this HLD PR description

@vvolam vvolam mentioned this pull request Sep 24, 2024
Merged
@vvolam vvolam mentioned this pull request Oct 5, 2024
Draft
prgeor
prgeor reviewed Oct 6, 2024
View reviewed changes
doc/smart-switch/reboot/reboot-hld.md
| Planned cold reboot of DPU | - | Graceful reboot |
| Planned power-cycle of Smart Switch | Graceful reboot | Graceful reboot |
| Planned power-cycle of DPU | - | Graceful reboot |
| Unplanned DPU power failure | - | Ungraceful reboot |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vvolam how are we planning to induce this failure in sonic-mgmt test?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prgeor I don't have implementation details now. If this cannot be done in sonic-mgmt test, I plan to atleast cover this during unit-test.

prgeor
prgeor reviewed Oct 7, 2024
View reviewed changes
doc/smart-switch/reboot/reboot-hld.md Outdated

* With the DPUs prepared for reboot, the NPU triggers a platform vendor API to initiate the reboot process for the DPUs. Vendor API reboots a single DPU, but the NPU spawns multiple threads to reboot DPUs in parallel. If any of the the DPU is stuck or unresponsive, the DPU reboot platform API should attempt a cold boot or power cycle to recover it.

* DPUs will send an acknowledgment to the NPU and then undergo a reboot. After receiving the acknowledgment from the DPUs, the NPU will proceed to reboot itself to complete the overall reboot procedure. The vendor-specific reboot API should include an error handling mechanism to manage DPU reboot failures. Additionally log all the failures. DPUs will be in DPU_READY state, if the reboot happened successfully.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vvolam can you elaborate on this ack "DPUs will send an acknowledgment to the NPU and then undergo a reboot" ? LIke how is this implemented?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not actually acknowledgment but response to platform vendor API. I have reworded accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@prgeor prgeor prgeor left review comments

@oleksandrivantsiv oleksandrivantsiv oleksandrivantsiv approved these changes

@rameshraghupathy rameshraghupathy rameshraghupathy approved these changes

@isabelmsft isabelmsft isabelmsft left review comments

@ganglyu ganglyu ganglyu left review comments

@r12f r12f Awaiting requested review from r12f

@dgsudharsan dgsudharsan Awaiting requested review from dgsudharsan

@qiluo-msft qiluo-msft Awaiting requested review from qiluo-msft

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@vvolam @KrisNey-MSFT @prgeor @oleksandrivantsiv @rameshraghupathy @isabelmsft @ganglyu