Reboot with mgmt vrf enabled is causing force_mgmt routes to have priority over local table

[vivekrnv]

Description

Reboot with mgmt vrf enabled is causing force_mgmt routes to have priority over local table. I've verified on 202405 and above but likely a day 1 behavior.

Steps to reproduce the issue:

1. Initial Config

root@sonic:/home/admin# redis-cli -n 4 hgetall "MGMT_INTERFACE|eth0|10.245.21.113/22"
1) "forced_mgmt_routes@"
2) "10.245.0.0/16,......"
3) "gwaddr"
4) "10.245.20.1"
 
root@sonic:/home/admin# ip ru sh
1001:        from all lookup local  <Notice local rule has preference over the forced mgmt rule>
32764:        from all to 10.245.0.0/16 lookup default
32765:        from 10.245.21.113 lookup default
32766:        from all lookup main
32767:        from all lookup default

2. Configure mgmt vrf

root@sonic:/home/admin# config vrf add mgmt

root@sonic:/home/admin# ip ru sh
1000:        from all lookup [l3mdev-table] <High priority rule to handle Vrf>
1001:         from all lookup local <Here, local has priority over forced_mgmt route>
32764:        from all to 10.245.0.0/16 lookup mgmt
32765:        from 10.245.21.113 lookup mgmt
32766:        from all lookup main
32767:        from all lookup default

3. Save config and reboot

root@sonic:/home/admin# config save -y 
root@sonic:/home/admin# reboot

<After reboot>

root@sonic:/home/admin# ip ru sh
1000:        from all lookup [l3mdev-table]
32764:        from all to 10.245.0.0/16 lookup mgmt <Forced mgmt rule took priority over local>
32765:        from all lookup local
32765:        from 10.245.21.113 lookup mgmt
32766:        from all lookup main
32767:        from all lookup default

4. Remove mgmt vrf and traffic to eth0 starts dropping

root@sonic:/home/admin#  config vrf del mgmt

root@sonic:/home/admin# ip ru sh
1000:        from all lookup [l3mdev-table]
32764:        from all to 10.245.0.0/16 lookup default
32765:        from all lookup local
32765:        from 10.245.21.113 lookup default
32766:        from all lookup main
32767:        from all lookup default

root@sonic:/home/admin# ping 10.245.21.113
PING 10.245.21.113 (10.245.21.113) 56(84) bytes of data. <ping from inside the device is not working>

ping 10.245.21.113
PING 10.245.21.113 (10.245.21.113) 56(84) bytes of data.
icmp_seq=18 Destination Host Unreachable

Or can be reproduced by running syslog/test_syslog_source_ip.py::test_syslog_config_work_after_reboot setups with mgmt ip being matched with any of the forced_mgmt_route prefixes.

Describe the results you expected:

Traffic should work even after config