Add debuggability for reboot function by JibinBao · Pull Request #14341 · sonic-net/sonic-mgmt

JibinBao · 2024-08-30T07:16:46Z

Description of PR

Add function to collect console log from starting reboot to dut up
When dut is not up, check if dut is pingable and collect the mgmt interface config

Summary:
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

Enhance the debugability for reboot fucntion

How did you do it?

Add function to collect console log from starting reboot to dut up
When dut is not up, check if dut is pingable and collect the mgmt interface config

How did you verify/test it?

Run reboot cases

Any platform specific information?

Any

Supported testbed topology if it's a new test case?

Documentation

JibinBao · 2024-09-10T01:12:51Z

/azpw run Azure.sonic-mgmt

mssonicbld · 2024-09-10T01:12:53Z

/AzurePipelines run Azure.sonic-mgmt

azure-pipelines · 2024-09-10T01:13:02Z

Azure Pipelines successfully started running 1 pipeline(s).

JibinBao · 2024-09-10T05:53:35Z

/azpw run Azure.sonic-mgmt

mssonicbld · 2024-09-10T05:53:38Z

/AzurePipelines run Azure.sonic-mgmt

azure-pipelines · 2024-09-10T05:53:48Z

Azure Pipelines successfully started running 1 pipeline(s).

JibinBao · 2024-09-11T01:29:59Z

/azpw run Azure.sonic-mgmt

mssonicbld · 2024-09-11T01:30:01Z

/AzurePipelines run Azure.sonic-mgmt

azure-pipelines · 2024-09-11T01:30:12Z

Azure Pipelines successfully started running 1 pipeline(s).

JibinBao · 2024-09-18T02:07:36Z

/azpw run Azure.sonic-mgmt

mssonicbld · 2024-09-18T02:07:39Z

/AzurePipelines run Azure.sonic-mgmt

azure-pipelines · 2024-09-18T02:07:49Z

Azure Pipelines successfully started running 1 pipeline(s).

bingwang-ms · 2024-09-23T17:21:31Z

The concern for this change is we may see test failure if the console connection is not right or not working fine, even though it's not related to the reboot feature.
@prgeor Can you please help review as well?

JibinBao · 2024-09-24T01:01:35Z

When console connection is not right, it will not affect the reboot function, it just does not collect the console logs

bingwang-ms · 2024-10-14T18:32:03Z

Hi @JibinBao, I still have concern on this change. It's because there are various console types and some of them are not stable enough. We don't want to fail the reboot test for a console connection issue.

JibinBao · 2024-10-15T01:22:23Z

Hi @JibinBao, I still have concern on this change. It's because there are various console types and some of them are not stable enough. We don't want to fail the reboot test for a console connection issue.

Hi @bingwang-ms , You are right, I have the same conncern. So, we use try to create the dutconsole. If fail to create dutconsole, we will not collect the logs. so it will not affect the original test logic. You can check the funciton of try_create_dut_console

prgeor · 2024-10-15T06:13:46Z

@JibinBao please resolve the conflicts

JibinBao · 2024-10-15T06:14:56Z

@JibinBao please resolve the conflicts

Will handle it. Thanks

prgeor · 2024-10-15T06:17:14Z

+        logger.warning("dut console is not ready, we can get log by console")
+
+
+def collect_mgmt_config_by_console(duthost, localhost):


@JibinBao what is the objective of collecting this?

Sometimes, when dut is not up, i am not sure if it is the mgmt netowrk issue, so check if the mgmt ip is pingable and
the mgmt config is correct or not.

prgeor · 2024-10-15T06:18:23Z

    return False
+
+
+def try_create_dut_console(duthost, localhost, conn_graph_facts, creds):


@JibinBao How are you ensuring the reboot command is issued via console connection? What if the reboot is issued via SSH connection and there is no console connection?

The function is to create dut_console to collect conosle logs, and it is not related to reboot test logic.
If there is no console connection, it will not collect the console logs, and it does not affect the original reboot test.
It is an enhancement. If there is a console connection and the console connection can be created successfully, it collect the console logs, otherwise, it will do nothing.

1. Add function to collect console log from starting reboot to dut up 2. When dut is not up, check if dut is pingable and collect the mgmt interface config Change-Id: I700340d3fbfc00d61bc6e508ac5be4e4dc0a25e6

JibinBao · 2024-10-16T05:24:08Z

prgeor

hi @prgeor , the Conflict has been fixed. Can you help review it again?

JibinBao · 2024-10-31T01:59:16Z

Hi @bingwang-ms ,
I push a fix for resloving the conflict, but it is stuck on the status(please see picture below). Can you help check it? and What I need to do next?

bingwang-ms

Hi @JibinBao, I don't agree to merge this change.
Unstable console connection can lead to test_reboot failure.
Can you keep the change in your internal repo?

JibinBao · 2024-11-01T01:28:47Z

Hi @JibinBao, I don't agree to merge this change. Unstable console connection can lead to test_reboot failure. Can you keep the change in your internal repo?

Hi @bingwang-ms ,
Because we it will not be affect by the console issue.

If it fails to create a console, it will not collect console logs, and execute the code as the original logic
We have run the test with the fixed for more than tow months, there is not a failure caused by it
If we keep it internal repo, it will alway confict when updating code from communtiy
Can you merge it? if it is not stable, we can revert it.

Thanks

liat-grozovik · 2024-12-03T18:57:09Z

and lets please resolve conflict so we can approve and merge

JibinBao · 2024-12-04T03:16:07Z

cannot reopen

JibinBao · 2024-12-04T03:16:21Z

reopen it

JibinBao · 2024-12-04T03:28:53Z

When pushing the new fix it is hung in "processing-updates" status. So close it and open new PR: #15868

…onic-net#13084) (sonic-net#14341)" (sonic-net#15094) (sonic-net#17367) (sonic-net#17461)

JibinBao force-pushed the add_debugability_reboot branch from 45c9ba3 to 7df7160 Compare August 30, 2024 07:17

congh-nvidia approved these changes Aug 30, 2024

View reviewed changes

nhe-NV added Request for 202311 branch Request for 202405 branch labels Aug 30, 2024

JibinBao force-pushed the add_debugability_reboot branch from 7df7160 to 2653915 Compare September 9, 2024 10:18

prgeor reviewed Oct 15, 2024

View reviewed changes

Add debuggability for reboot function

b5dcc6f

1. Add function to collect console log from starting reboot to dut up 2. When dut is not up, check if dut is pingable and collect the mgmt interface config Change-Id: I700340d3fbfc00d61bc6e508ac5be4e4dc0a25e6

JibinBao force-pushed the add_debugability_reboot branch from 2653915 to b5dcc6f Compare October 15, 2024 06:49

bingwang-ms requested changes Oct 31, 2024

View reviewed changes

congh-nvidia reviewed Nov 26, 2024

View reviewed changes

Comment thread tests/common/reboot.py

JibinBao marked this pull request as draft December 4, 2024 03:13

JibinBao marked this pull request as ready for review December 4, 2024 03:14

JibinBao closed this Dec 4, 2024

JibinBao mentioned this pull request Dec 4, 2024

Add debuggability for reboot function #15868

Merged

8 tasks

kazinator-arista pushed a commit to kazinator-arista/sonic-mgmt that referenced this pull request Mar 4, 2026

Revert "[swss/syncd] remove dependency on interfaces-config.service (s…

688245a

…onic-net#13084) (sonic-net#14341)" (sonic-net#15094) (sonic-net#17367) (sonic-net#17461)

		logger.warning("dut console is not ready, we can get log by console")


		def collect_mgmt_config_by_console(duthost, localhost):

		return False


		def try_create_dut_console(duthost, localhost, conn_graph_facts, creds):

Conversation

JibinBao commented Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

JibinBao commented Sep 10, 2024

Uh oh!

mssonicbld commented Sep 10, 2024

Uh oh!

azure-pipelines Bot commented Sep 10, 2024

Uh oh!

JibinBao commented Sep 10, 2024

Uh oh!

mssonicbld commented Sep 10, 2024

Uh oh!

azure-pipelines Bot commented Sep 10, 2024

Uh oh!

JibinBao commented Sep 11, 2024

Uh oh!

mssonicbld commented Sep 11, 2024

Uh oh!

azure-pipelines Bot commented Sep 11, 2024

Uh oh!

JibinBao commented Sep 18, 2024

Uh oh!

mssonicbld commented Sep 18, 2024

Uh oh!

azure-pipelines Bot commented Sep 18, 2024

Uh oh!

bingwang-ms commented Sep 23, 2024

Uh oh!

JibinBao commented Sep 24, 2024

Uh oh!

bingwang-ms commented Oct 14, 2024

Uh oh!

JibinBao commented Oct 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

prgeor commented Oct 15, 2024

Uh oh!

JibinBao commented Oct 15, 2024

Uh oh!

prgeor Oct 15, 2024

Choose a reason for hiding this comment

Uh oh!

JibinBao Oct 15, 2024

Choose a reason for hiding this comment

Uh oh!

prgeor Oct 15, 2024

Choose a reason for hiding this comment

Uh oh!

JibinBao Oct 15, 2024

Choose a reason for hiding this comment

Uh oh!

JibinBao commented Oct 16, 2024

Uh oh!

JibinBao commented Oct 31, 2024

Uh oh!

bingwang-ms left a comment

Choose a reason for hiding this comment

Uh oh!

JibinBao commented Nov 1, 2024

Uh oh!

Uh oh!

liat-grozovik commented Dec 3, 2024

Uh oh!

JibinBao commented Dec 4, 2024

Uh oh!

JibinBao commented Dec 4, 2024

Uh oh!

JibinBao commented Dec 4, 2024

JibinBao commented Aug 30, 2024 •

edited

Loading

JibinBao commented Oct 15, 2024 •

edited

Loading