Disconnect issue with FL5000_Ver 2.0 Dev board

[liquiddrop]

I have taken the code from here and build it into a Android build and flashed it to my phone. On first connect everything seems to be working and I am able to plug usb2 and usb3 devices into my dev board and use them for the phone. But when I disconnect from the board the usb connection on my phone wont re-enumerate with any device (ie I can't plug it back into the board and have it work and I can't plug it into my computer and shell into it using adb). After looking into this more here is a timeline of what is happening:

Disconnect the usb
Hub.c code gets a port change status event which goes on to hub workqueue(hub_wq)
Hub.c handles the event in hub_event()
The port change status kicks off a disconnect in the ehub code
The disconnect in the ehub code calls usb_remove_hcd()
Usb_remove_hcd() calls usb_flush_hub_wq() which tries to flush hub_wq
The flush stalls waiting for hub_wq to finish executing and we get stuck there

So looks like the hub work queue is getting into a dead lock situation. I removed usb_remove_hcd() from the ehub code and I am able to disconnect and reconnect with everything working, but this is leaving virtual hubs attached on my phone and the phone is just assigning new hub numbers every time I reconnect. Any guidance you could give on this would be appreciated.