Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storm: Network: Topology redeployment: ISL between deactivated switches is in DISCOVERED status (should be FAILED) #5506

Open
yuliiamir opened this issue Dec 4, 2023 · 1 comment
Open

Storm: Network: Topology redeployment: ISL between deactivated switches is in DISCOVERED status (should be FAILED) #5506

yuliiamir opened this issue Dec 4, 2023 · 1 comment

Comments

@yuliiamir
Copy link
Collaborator

yuliiamir commented Dec 4, 2023 •
edited
Loading

The issue occurred when the network topology has been redeployed after killing the network topology with the following switches' deactivation. As a result, after recovering, both switches are marked as DEACTIVATED (expected behavior). BUT ISL between deactivated switches is in DISCOVERED status(incorrect behaviour). The rest of the links that are connected to the DEACTIVATED switches are in FAILED status.
Kibana logs:
ISL that should be in FAILED status(both switches are in DEACTIVATED status):
LogsFowISLThatShouldBeInFailedStatus

ISL is in FAILED status (one of the switches is in DEACTIVATED status):
CorrectLogicAndISLIsInFailedStatus
Topology:
SwitchDetails

Links details after redeploying network topology(switches were switched off after network topology deletion):

[
  {
    "latency_ns": 3086294,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 50,
        "seq_id": 0,
        "segment_latency": 3086294
      },
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 7,
        "seq_id": 1,
        "segment_latency": 3086294
      }
    ]
  },
  {
    "latency_ns": 3659898,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 4,
        "seq_id": 0,
        "segment_latency": 3659898
      },
      {
        "switch_id": "00:00:00:00:00:00:00:04",
        "port_no": 4,
        "seq_id": 1,
        "segment_latency": 3659898
      }
    ]
  },
  {
    "latency_ns": 3786802,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 7,
        "seq_id": 0,
        "segment_latency": 3786802
      },
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 50,
        "seq_id": 1,
        "segment_latency": 3786802
      }
    ]
  },
  {
    "latency_ns": 3197969,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:04",
        "port_no": 4,
        "seq_id": 0,
        "segment_latency": 3197969
      },
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 4,
        "seq_id": 1,
        "segment_latency": 3197969
      }
    ]
  },
  {
    "latency_ns": 3532994,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", // sw_02 is deactivated - OK
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 2,
        "seq_id": 0,
        "segment_latency": 3532994
      },
      {
        "switch_id": "00:00:00:00:00:00:00:04",
        "port_no": 1,
        "seq_id": 1,
        "segment_latency": 3532994
      }
    ]
  },
  {
    "latency_ns": 4992,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 6,
        "seq_id": 0,
        "segment_latency": 4992
      },
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 16,
        "seq_id": 1,
        "segment_latency": 4992
      }
    ]
  },
  {
    "latency_ns": 3340101,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED",  sw_02 is deactivated - OK
    "actual_state": "FAILED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:04",
        "port_no": 1,
        "seq_id": 0,
        "segment_latency": 3340101
      },
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 2,
        "seq_id": 1,
        "segment_latency": 3340101
      }
    ]
  },
  {
    "latency_ns": 5069,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 16,
        "seq_id": 0,
        "segment_latency": 5069
      },
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 6,
        "seq_id": 1,
        "segment_latency": 5069
      }
    ]
  },
  {
    "latency_ns": 5399,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 3,
        "seq_id": 0,
        "segment_latency": 5399
      },
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 1,
        "seq_id": 1,
        "segment_latency": 5399
      }
    ]
  },
  {
    "latency_ns": 3730964,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 49,
        "seq_id": 0,
        "segment_latency": 3730964
      },
      {
        "switch_id": "00:00:00:00:00:00:00:09",
        "port_no": 52,
        "seq_id": 1,
        "segment_latency": 3730964
      }
    ]
  },
  {
    "latency_ns": 4587,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 1,
        "seq_id": 0,
        "segment_latency": 4587
      },
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 2,
        "seq_id": 1,
        "segment_latency": 4587
      }
    ]
  },
  {
    "latency_ns": 4812,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "FAILED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 1,
        "seq_id": 0,
        "segment_latency": 4812
      },
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 3,
        "seq_id": 1,
        "segment_latency": 4812
      }
    ]
  },
  {
    "latency_ns": 3583756,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:09",
        "port_no": 52,
        "seq_id": 0,
        "segment_latency": 3583756
      },
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 49,
        "seq_id": 1,
        "segment_latency": 3583756
      }
    ]
  },
  {
    "latency_ns": 6520,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "FAILED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 2,
        "seq_id": 0,
        "segment_latency": 6520
      },
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 1,
        "seq_id": 1,
        "segment_latency": 6520
      }
    ]
  },
  {
    "latency_ns": 3446700,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_01 is deactivated - OK
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:01",
        "port_no": 2,
        "seq_id": 0,
        "segment_latency": 3446700
      },
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 15,
        "seq_id": 1,
        "segment_latency": 3446700
      }
    ]
  },
  {
    "latency_ns": 3482233,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_01 is deactivated - OK
    "actual_state": "FAILED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 15,
        "seq_id": 0,
        "segment_latency": 3482233
      },
      {
        "switch_id": "00:00:00:00:00:00:00:01",
        "port_no": 2,
        "seq_id": 1,
        "segment_latency": 3482233
      }
    ]
  },
  {
    "latency_ns": 3680203,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 51,
        "seq_id": 0,
        "segment_latency": 3680203
      },
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 47,
        "seq_id": 1,
        "segment_latency": 3680203
      }
    ]
  },
  {
    "latency_ns": 3370558,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 49,
        "seq_id": 0,
        "segment_latency": 3370558
      },
      {
        "switch_id": "00:00:00:00:00:00:00:09",
        "port_no": 48,
        "seq_id": 1,
        "segment_latency": 3370558
      }
    ]
  },
  {
    "latency_ns": 4319796,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 47,
        "seq_id": 0,
        "segment_latency": 4319796
      },
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 51,
        "seq_id": 1,
        "segment_latency": 4319796
      }
    ]
  },
  {
    "latency_ns": 4081218,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:09",
        "port_no": 48,
        "seq_id": 0,
        "segment_latency": 4081218
      },
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 49,
        "seq_id": 1,
        "segment_latency": 4081218
      }
    ]
  },
  {
    "latency_ns": 3228426,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 6,
        "seq_id": 0,
        "segment_latency": 3228426
      },
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 19,
        "seq_id": 1,
        "segment_latency": 3228426
      }
    ]
  },
  {
    "latency_ns": 3370558,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "FAILED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:07",
        "port_no": 19,
        "seq_id": 0,
        "segment_latency": 3370558
      },
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 6,
        "seq_id": 1,
        "segment_latency": 3370558
      }
    ]
  },
  {
    "latency_ns": 4332,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 5,
        "seq_id": 0,
        "segment_latency": 4332
      },
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 1,
        "seq_id": 1,
        "segment_latency": 4332
      }
    ]
  },
  {
    "latency_ns": 3832487,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 5,
        "seq_id": 0,
        "segment_latency": 3832487
      },
      {
        "switch_id": "00:00:00:00:00:00:00:09",
        "port_no": 32,
        "seq_id": 1,
        "segment_latency": 3832487
      }
    ]
  },
  {
    "latency_ns": 4938,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "FAILED", sw_02 is deactivated - OK
    "actual_state": "FAILED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:08",
        "port_no": 1,
        "seq_id": 0,
        "segment_latency": 4938
      },
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 5,
        "seq_id": 1,
        "segment_latency": 4938
      }
    ]
  },
  {
    "latency_ns": 3157360,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:09",
        "port_no": 32,
        "seq_id": 0,
        "segment_latency": 3157360
      },
      {
        "switch_id": "00:00:00:00:00:00:00:03",
        "port_no": 5,
        "seq_id": 1,
        "segment_latency": 3157360
      }
    ]
  },
  {
    "latency_ns": 3487309,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED",  - NOT OK as both switches are DEACTIVATED!!! 
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:01",
        "port_no": 1,
        "seq_id": 0,
        "segment_latency": 3487309
      },
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 4,
        "seq_id": 1,
        "segment_latency": 3487309
      }
    ]
  },
  {
    "latency_ns": 3461928,
    "speed": 10000000,
    "available_bandwidth": 10000000,
    "max_bandwidth": 10000000,
    "default_max_bandwidth": 10000000,
    "state": "DISCOVERED", - NOT OK as both switches are DEACTIVATED!!! 
    "actual_state": "DISCOVERED",
    "round_trip_status": "FAILED",
    "cost": 700,
    "under_maintenance": false,
    "enable_bfd": false,
    "path": [
      {
        "switch_id": "00:00:00:00:00:00:00:02",
        "port_no": 4,
        "seq_id": 0,
        "segment_latency": 3461928
      },
      {
        "switch_id": "00:00:00:00:00:00:00:01",
        "port_no": 1,
        "seq_id": 1,
        "segment_latency": 3461928
      }
    ]
  }
]

Expected result: both switches are deactivated and appropriate ISLs are FAILED.

To reproduce this case, the test System's able to fail an ISL if switches on both ends go offline during restart of network topology can be used.

The investigation of the calling Uni-ISL SETUP request is required.

Useful links:
#2363
#5504
#2364
@yuliiamir yuliiamir changed the title Storm: Network: Topology redeployment: System sends Uni-ISL SETUP request when both ISL-related switches are DEACTIVATED. Storm: Network: Topology redeployment: ISL between deactivated switches is in DISCOVERED status (should be FAILED) Dec 4, 2023
@yuliiamir yuliiamir mentioned this issue Dec 4, 2023
Merged
@yuliiamir
Copy link
Collaborator Author

Minor: need to update test due to the comment https://github.com/telstra/open-kilda/pull/5507/files#r1415444053 when issue is fixed and test can be switched on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yuliiamir