Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VR Mising #10574

Open
hstan77 opened this issue Mar 14, 2025 · 18 comments · May be fixed by #10612
Open

VR Mising #10574

hstan77 opened this issue Mar 14, 2025 · 18 comments · May be fixed by #10612

Comments

@hstan77
Copy link

hstan77 commented Mar 14, 2025

I have my VR that deleted during restart network, however it not automatically created back.

Now all the VM that under this GuestNetwork are unable to start, it shows Insufficient Capacity due to no VR . Even though I want to delete this network from the VM and change another one also cannot delete it.

The only thing I can do is destroy the whole vm .

I am using Cloudstack 4.15 and Ubuntu 18.

@weizhouapache
Copy link
Member

please share the logs, thanks

@hstan77
Copy link
Author

hstan77 commented Mar 14, 2025

Starting VM that missing VR
025-03-14 18:57:40,347 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-81:ctx-e1df6ca7 job-487558/job-487559 ctx-ad6c35ae) (logid:ed251ac2) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!
at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:288)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1485)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1840)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1774)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1149)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5464)
at jdk.internal.reflect.GeneratedMethodAccessor424.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5631)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

=======+Restarting network that missing VR ===
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:251)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1392)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1327)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3166)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1994)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2012)
at jdk.internal.reflect.GeneratedMethodAccessor716.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy256.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-14 18:59:52,200 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-100:ctx-310b6a64 job-487561 ctx-99dba8c4) (logid:5b299583) Network id=217 failed to restart.
2025-03-14 18:59:52,223 DEBUG [c.c.a.ApiServlet] (qtp1886491834-1071:ctx-467ab645) (logid:7657

@hstan77
Copy link
Author

hstan77 commented Mar 14, 2025

Any body can let me know to manual get the VR up . or Some Db record need to update it .

@DaanHoogland
Copy link
Contributor

@hstan77 , is there enough capacity to start the VR ?

@weizhouapache
Copy link
Member

@hstan77

I think there are some exceptions before the log you shared. can you have a look ?

also, can you restart the network with cleanup ?

@hstan77
Copy link
Author

hstan77 commented Mar 14, 2025

Hi

It is enaugh to start VR, a lot more

Cleaning up , it will say no VR to clean up

@hstan77
Copy link
Author

hstan77 commented Mar 14, 2025

i grep WARN and ERR

**restart Network

2025-03-14 19:55:30,234 WARN [c.c.a.ApiServer] (qtp1886491834-1084:ctx-584ca2a8 ctx-a2113a51) (logid:6033a140) Unknown API command: listAndSwitchSamlAccount
2025-03-14 19:55:46,110 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-24:ctx-52d1ff6b job-488646 ctx-faec8381) (logid:2394dcae) Failed to implement network Ntwk[217|Guest|6] elements and resources as a part of network restart due to
2025-03-14 19:55:46,112 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-24:ctx-52d1ff6b job-488646 ctx-faec8381) (logid:2394dcae) Network id=217 failed to restart.

2025-03-14 19:57:30,843 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-100:ctx-8c1177ba job-488649 ctx-3c1be356) (logid:7d78035b) Didn't support redundant virtual router without public network!

** restart network and clean up
2025-03-14 19:56:58,533 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-87:ctx-8d18e4fc job-488648 ctx-d509981e) (logid:518baef4) Didn't support redundant virtual router without public network!
2025-03-14 19:56:58,560 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-87:ctx-8d18e4fc job-488648) (logid:518baef4) Unexpected exception while executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd

@hstan77
Copy link
Author

hstan77 commented Mar 14, 2025

This is detail with debug log

2025-03-14 19:58:24,165 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-8c72f876 job-488650) (logid:345a719c) Add job-488650 into job monitoring
2025-03-14 19:58:24,169 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8844, power state: PowerOn
2025-03-14 19:58:24,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1886491834-1113:ctx-cdaaea5e ctx-3c16e021) (logid:a4093f5d) submit async job-488650, details: AsyncJobVO {id:488650, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3079878","id":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","ctxDetails":"{"interface com.cloud.network.Network":"a3cca856-973e-4a6a-aa9d-d4c908938cc7"}","ctxAccountId":"2","uuid":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","cmdEventType":"NETWORK.RESTART"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2130706689, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-03-14 19:58:24,175 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8848, power state: PowerOn
2025-03-14 19:58:24,176 DEBUG [c.c.a.ApiServlet] (qtp1886491834-1113:ctx-cdaaea5e ctx-3c16e021) (logid:a4093f5d) ===END=== 180.75.249.24 -- GET id=a3cca856-973e-4a6a-aa9d-d4c908938cc7&cleanup=false&makeredundant=false&command=restartNetwork&response=json
2025-03-14 19:58:24,180 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1940, power state: PowerOn
2025-03-14 19:58:24,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-50:ctx-8c72f876 job-488650) (logid:3fef167a) Executing AsyncJobVO {id:488650, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3079878","id":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","ctxDetails":"{"interface com.cloud.network.Network":"a3cca856-973e-4a6a-aa9d-d4c908938cc7"}","ctxAccountId":"2","uuid":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","cmdEventType":"NETWORK.RESTART"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2130706689, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-03-14 19:58:24,185 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9621, power state: PowerOn
2025-03-14 19:58:24,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8857, power state: PowerOn
2025-03-14 19:58:24,197 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9628, power state: PowerOn
2025-03-14 19:58:24,204 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1181, power state: PowerOn
2025-03-14 19:58:24,210 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1951, power state: PowerOn
2025-03-14 19:58:24,215 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1445, power state: PowerOn
2025-03-14 19:58:24,221 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9642, power state: PowerOn
2025-03-14 19:58:24,228 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1196, power state: PowerOn
2025-03-14 19:58:24,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1199, power state: PowerOn
2025-03-14 19:58:24,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 433, power state: PowerOn
2025-03-14 19:58:24,249 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8627, power state: PowerOn
2025-03-14 19:58:24,253 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Restarting network 217...
2025-03-14 19:58:24,255 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7352, power state: PowerOn
2025-03-14 19:58:24,258 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Implementing the network Ntwk[217|Guest|6] elements and resources as a part of network restart without cleanup
2025-03-14 19:58:24,260 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1209, power state: PowerOn
2025-03-14 19:58:24,266 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1212, power state: PowerOn
2025-03-14 19:58:24,271 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7358, power state: PowerOn
2025-03-14 19:58:24,277 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8640, power state: PowerOn
2025-03-14 19:58:24,283 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 960, power state: PowerOn
2025-03-14 19:58:24,289 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7620, power state: PowerOn
2025-03-14 19:58:24,295 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 965, power state: PowerOn
2025-03-14 19:58:24,300 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 967, power state: PowerOn
2025-03-14 19:58:24,306 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 970, power state: PowerOn
2025-03-14 19:58:24,311 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 5834, power state: PowerOn
2025-03-14 19:58:24,317 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8139, power state: PowerOn
2025-03-14 19:58:24,323 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2000, power state: PowerOn
2025-03-14 19:58:24,325 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Asking SecurityGroupProvider to implemenet Ntwk[217|Guest|6]
2025-03-14 19:58:24,328 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 980, power state: PowerOn
2025-03-14 19:58:24,333 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 981, power state: PowerOn
2025-03-14 19:58:24,339 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 983, power state: PowerOn
2025-03-14 19:58:24,345 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2007, power state: PowerOn
2025-03-14 19:58:24,351 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 984, power state: PowerOn
2025-03-14 19:58:24,356 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 6106, power state: PowerOn
2025-03-14 19:58:24,361 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2012, power state: PowerOn
2025-03-14 19:58:24,367 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 991, power state: PowerOn
2025-03-14 19:58:24,372 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2017, power state: PowerOn
2025-03-14 19:58:24,375 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Asking VirtualRouter to implemenet Ntwk[217|Guest|6]
2025-03-14 19:58:24,378 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2019, power state: PowerOn
2025-03-14 19:58:24,384 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 996, power state: PowerOn
2025-03-14 19:58:24,388 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Didn't support redundant virtual router without public network!
2025-03-14 19:58:24,389 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7142, power state: PowerOn
2025-03-14 19:58:24,395 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7147, power state: PowerOn
2025-03-14 19:58:24,395 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Failed to implement network Ntwk[217|Guest|6] elements and resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:251)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1392)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1327)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3166)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1994)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2012)
at jdk.internal.reflect.GeneratedMethodAccessor716.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy256.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-14 19:58:24,397 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Network id=217 failed to restart.

@hstan77
Copy link
Author

hstan77 commented Mar 14, 2025

When restart network, it show error cannot find router, of cause it cant find, cause the VR missing.

When restart with Clenaup, It show cannot find VR to clean also. and that is right cause VR is missing.

However my question is the VR not auto created, and how can we create it back ?

@weizhouapache
Copy link
Member

When restart network, it show error cannot find router, of cause it cant find, cause the VR missing.

When restart with Clenaup, It show cannot find VR to clean also. and that is right cause VR is missing.

However my question is the VR not auto created, and how can we create it back ?

Can you upload the full log when start /creatr the VR ?

The logs you uploaded do not give enough information.

@hstan77
Copy link
Author

hstan77 commented Mar 15, 2025

Hi

Issue now is the VR not created automatically , i can only do is retart network as above.

The whole Cloudstack , there is no function to create VR to the GuestNetwork . Or Manual activate Create VR .

The case now is when VR not automatically create, it will be trouble. How to create back VR?

@hstan77
Copy link
Author

hstan77 commented Mar 15, 2025

I do change IP those function:
2025-03-15 03:31:34,288 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (Work-Job-Executor-151:ctx-e7cd657e job-489150/job-489152 ctx-ec97e82e) (logid:8b532c8b) Didn't support redundant virtual router without public network!
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!

it just not auto create back the router.

@hstan77
Copy link
Author

hstan77 commented Mar 15, 2025

Finally i found the issue . This is Bug of Cloudstack.

The GuestNetwork , redundant=1 in Mysql, and when this happen , the VR will not auto create back . It keep say cannot create redundant . And the webinterface is no way to the status of the GuestNetwork is in redundant or not , and there is no way to turn off the redundant.

I edit mysql and set redundant=0, and restart network , and VR created back.

@weizhouapache
Copy link
Member

Finally i found the issue . This is Bug of Cloudstack.

The GuestNetwork , redundant=1 in Mysql, and when this happen , the VR will not auto create back . It keep say cannot create redundant . And the webinterface is no way to the status of the GuestNetwork is in redundant or not , and there is no way to turn off the redundant.

I edit mysql and set redundant=0, and restart network , and VR created back.

a bit strange, why is redundant 1 ?

is it a shared network or isolated network ? what's the network offering ?

@hstan77
Copy link
Author

hstan77 commented Mar 15, 2025

I am in shared network .

Network offering : [DefaultSharedNetworkOfferingWithSGService]

i think it may cause by when restart network time, i clicked "redundant" and "clean up", after VR destroyed, it cannot up back. I guess is this, need to test how to simulate this back.

@weizhouapache
Copy link
Member

I am in shared network .

Network offering : [DefaultSharedNetworkOfferingWithSGService]

i think it may cause by when restart network time, i clicked "redundant" and "clean up", after VR destroyed, it cannot up back. I guess is this, need to test how to simulate this back.

there were some discussion in #10491
we should disallow it

@hstan77
Copy link
Author

hstan77 commented Mar 16, 2025

This issue #491 is only mentioned need or no need the features.

Even now the option is there, the Error Handling seem not handle the issue well. It shall handle the issue of mistake no VR in GuestNetwork. At least have more detail in the log . At begining, I copied all the log , and nobody know what happening.
It is very lucky for me able to "Guess" the issue . Otherwise, this become unknown.

@weizhouapache
Copy link
Member

This issue #10491 is only mentioned need or no need the features.

@hstan77
4.15 is EOL, I suggest you to upgrade to latest versions.

in recent versions, the make redundant option is not visible anymore, even for isolated networks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants