VPC network tier with internal LB issues

[chunkyen]

Hi, I have been testing Cloudstack using the Apache Cloudstack POC guide from ShapeBlue. In one of test case for VPC, it shows how to setup a frontend and backend network tier with internal LB and configuring VM using user data so as to demonstrate the frontend to backend LB and ACL capabilities.
I have performed this test sucessfully. However, when I shutdown my Cloudstack and power it on after sometime, the internal LB will get stuck in a starting state when I try to start the backend VMs. Furthermore, the backend tier guest network will show a shutdown state.
I have tried restarting VPC, restart router, delete router but I cannot get this to unstuck. I am also not able to clean up everything in the VPC.
I have tested this in a nested 1+3 nodes and a single baremetal node on version 4.21 with bridge networking and vlan isolation.
Have anyone else encountered the same issues? Are there any tips to troubleshoot and/or resolved this issue? (I cannot find anything useful from the management and agent log)
Thank you

[DaanHoogland]

@chunkyen , I am not sure but have you considered the order of starting things up again? Your hosts are VMs and after starting they may not start all the nested VMs again, including the systemVMs/VRs. Also where is the DB running? Did the log since restarting look clean or have DB errors?

[chunkyen]

Thanks for your reply. I have always shutdown all the VMs, including virtual routers and system VMs before powering off. Also, I use cmk to trigger management server shutdown before powering it down.
The funny thing is it only seems to affect the network with internal LB only. I also have an all in one physical box which I encountered the same network in shutdown state if I attach an internal LB to it after powering the host off and on.
The DB is running on the same VM/host as the management server and there is nothing specific in the management,DB and agent logs too.
I figure out a way to "clean up" the objects.

1. First delete the internal LB from the backend network tier with internal LB.
2. Disassociate all the VMs nic from this backend network tier.
3. Delete these VMs

After that the backend network tier miraculously turned back into an allocated state from shutdown state and the orphaned internal LB VM is also deleted. Yah!

[DaanHoogland]

After that the backend network tier miraculously turned back into an allocated state

This part is expected, a tier without any VMs “shuts down” and becomes “allocated” after a timeout.

Having to delete the VMs is a bit disturbing, but might give a hint to what is wrong. Can you try not stopping the VR before stopping its host?

When you say, "I have tried restarting VPC”, do you mean restart with cleanup?

[chunkyen]

This part is expected, a tier without any VMs “shuts down” and becomes “allocated” after a timeout.

I am not 100% sure but shouldn't this be a tier without VM running will be in a "allocated" state and become "shutdown" after a timeout? But I have never seen an actual network be in a shutdown state (except for this internal LB network error), only allocated and implemented.

When you say, "I have tried restarting VPC”, do you mean restart with cleanup?

Yes tried with and without cleanup and the result is the same, the internal LB and VM attached to this LB are stuck in starting and failed evetually after the timeout.

[DaanHoogland]

from Network,java, public enum State ,value Shutdown( "Indicates the network configuration is being destroyed”) , meaning it is not just being stopped.

part of the statemachine for it:

            s_fsm.addTransition(State.Implementing, Event.OperationFailed, State.Shutdown);
            s_fsm.addTransition(State.Implemented, Event.DestroyNetwork, State.Shutdown);

so no, only on explicit destroy would it go to state Shutdown. I have to test this, but I think the shutdown only happens if the router is stopped or (as an intermediate state) when the network destroyed.

[chunkyen]

Thank you for the explanation. I observed that the guest network that had the internal LB was in an allocated state upon management and host boot up. But as soon as I tried to boot up a vm in the network tier assigned as an instance to the internalLB, it will go into a shutdown state.
Also for the recovery, I do not need to destroy the VM attached to the internal LB. I just need to detatch the nic to another network, delete the internal LB and I can clean up the internal LB VM that was stucked.

[DaanHoogland]

ok @chunkyen , I’ll try to replay your scenario.

[DaanHoogland]

@chunkyen I think I am not reproducing your exact scenario. Can you expand on it; i.e. what offerings, what order of creation, what order of shutdown, etc...