Option for private deployment

[pearcec]

I just finished a deployment and it is failing to install. It is hanging on healthstatus from a public IP. We don't have public IPs as a workable option in our private environment with our hub/spoke model and User Default Routes forcing tunneling traffic. I am not sure how difficult it would be to eliminate Public IPs and use the Private IPs instead. There should also be an option which would utilized Private DNS with an option to disable it. With have a 3rd party solution. I am happy to work on code. If there are prior thoughts or plans please share.

[pearcec]

Challenges

• LB need to have a type/count solution to pick between the different type of frontend configurations
• a subnet needs to be passed in -- should provide the option of which subnet

[sean-nixon]

One serious challenge we've encountered with trying to accomplish this thing is Azure Internal Load Balancers have issues if backend nodes send traffic to the load balancer and get mapped back to themselves. TCP flows break entirely because of how Azure re-writes source IPs. This causes basically everything to fail because the install process involves nodes calling the load balancer to access the Kubernetes API server. For the first primary VM, this will always get mapped back to itself, stalling the entire installation.

So far we've had to resort to using a public LB with an NSG locking down access to only the public IP of the load balancer (effectively only allowing self-referential access from the backend nodes). There are other solutions such as using a scale set of VMs running something like Nginx or HAProxy to act as a reverse proxy but that adds additional cost and complexity.

Reference: https://docs.microsoft.com/en-us/azure/load-balancer/concepts-limitations#limitations

[pearcec]

AH I see, I will have to take a second look at this from a security perspective given your locking down at a NSG level. I might need to adjust my routes for the load balancer.

[sean-nixon]

A quick disclaimer since I don't think I really made it clear before, I'm not directly involved with Hashicorp or maintaining this module. I'm currently trying to implement a deployment of this for a client. The only other workaround we found so far would be to put a cluster of VMs running nginx or HAProxy behind an internal load balancer and use them as a proxy for self-referential traffic. This would require modifying this module considerably and adds additional cost and management overhead.

[bnferguson]

Hi there! @sean-nixon hit the nail on the head with one of the challenges we've run into with the Azure implementation re: why it's currently a public LB (and the Google one too, but I think in a slightly different way).

There are some options in this area, one of which is HAProxy or similar, and another is something like what is in the refactor of the Google modules where there's a bit of a IPTables "sandwich" playing this role: hashicorp/terraform-google-terraform-enterprise#38 (I believe it's the internal_lb module).

We're looking to do a similar refactor over here very soon which would fix this issue or at very least make it so you could replace it with your own solution in the middle if ours didn't suit the particular situation.

[pearcec]

So far we've had to resort to using a public LB with an NSG locking down access to only the public IP of the load balancer (effectively only allowing self-referential access from the backend nodes). There are other solutions such as using a scale set of VMs running something like Nginx or HAProxy to act as a reverse proxy but that adds additional cost and complexity.

Reference: https://docs.microsoft.com/en-us/azure/load-balancer/concepts-limitations#limitations

How do you access the console then? Assuming you need to have DNS setup for the public IP to point to tfedev.domain do you just setup DNS internally with a different hostname and use subject alternative names to connect? Are you using azure dns?

[sean-nixon]

@pearcec We are using split-horizon DNS to access the console/app over the internal load balancer using internal AD DNS (not automated unfortunately).

For initial testing and debugging, we're also whitelisting our own public IP in the NSG and using the public LB to access the app. That's also our emergency access model if for some reason we can't reach the app over the internal network due to a misconfiguration, outage, etc.

[pearcec]

@pearcec We are using split-horizon DNS to access the console/app over the internal load balancer using internal AD DNS (not automated unfortunately).

Ah, ok so you are using two load balancers?

I am trying to read up on the Load Balancer Floating IP and wondering if there is a mechanism to handle this

[sean-nixon]

@pearcec Can't really advise on your current setup but yes we have two load balancers because it's not possible mix public and private frontends on Azure Load Balancer.