Description

Azure Route Server and BIG-IP using BGP and eCMP for traffic distribution

To Do

Community support only. Template is not F5 supported.
Finish BIG-IP setup with test app
Remove test network virtuals (10.100, 10.101, 10.102)...testing only

Issues

Find an issue? Fork, clone, create branch, fix and PR. I'll review and merge into the main branch. Or submit a GitHub issue with all necessary details and logs.

Diagram

Prerequisites

Azure CLI
Terraform
Azure Subscription
Azure User with 'Owner' role

Login to Azure Environment

# Login
az login

# Show subscriptions
az account show

# Set default
az account set -s <subscriptionId>

Usage example

Clone the repo and open the solution's directory

git clone https://github.com/JeffGiroux/f5_terraform.git
cd f5_terraform/Azure/Route_Server/

Create the tfvars file and update it with your settings

cp admin.auto.tfvars.example admin.auto.tfvars
# MODIFY TO YOUR SETTINGS
vi admin.auto.tfvars

Run the setup script to deploy all of the components into your Azure account (remember that you are responsible for the cost of those components)

./setup.sh

Test your setup

View the created objects in Azure Portal. Choose a VM instance or NIC from a spoke VNet and view "Effective Routes". You will see BIG-IP advertised routes via BGP across the VNet peering. This deployment will launch a single BIG-IP instance, but if you modified 'instanceCountBigIp' then you will see identical network prefixes advertised from multiple BIG-IP devices (aka BGP peers) like the screenshot below. The 10.255.10.4 is BIG-IP #1 and 10.255.10.6 is BIG-IP #2.

View BPG peering on the Azure Route Server object in the portal - https://aka.ms/routeserver

Validate BGP peering on BIG-IP using tmsh

(tmos)# show net routing bgp
------------------------------------------
Net::BGP Instance (route-domain: 0)
------------------------------------------
  Name                               myBGP
  Local AS                           65530

  ----------------------------------------------------------------------------
  | Net::BGP Neighbor - 10.255.255.5 via 10.255.10.4
  ----------------------------------------------------------------------------
  | Remote AS                   0
  | State                       established   0:06:24
  | Notification                Cease/Administratively Shutdown.
  | Address Family              IPv4 Unicast  IPv6 Unicast
  |  Prefix
  |   Accepted                  3
  |   Announced                 6
  |  Table Version
  |   Local                     6
  |   Neighbor                  6
  | Message/Notification/Queue  Sent          Received
  |  Message                    27            26
  |  Notification               0             2
  |  Queued                     0             0
  | Route Refresh               0             0

View running config on BIG-IP using imish

(tmos)# imish
f5vm01.example.com[0]#show running-config
!
service password-encryption
!
bgp extended-asn-cap
!
router bgp 65530
 bgp graceful-restart restart-time 120
 aggregate-address 10.100.0.0/16 summary-only
 aggregate-address 10.101.0.0/16 summary-only
 aggregate-address 10.102.0.0/16 summary-only
 redistribute kernel
 neighbor Neighbor peer-group
 neighbor Neighbor remote-as 65515
 neighbor Neighbor ebgp-multihop 2
 no neighbor Neighbor capability route-refresh
 neighbor Neighbor soft-reconfiguration inbound
 neighbor Neighbor prefix-list /Common/myPrefixList1 out
 neighbor 10.255.255.4 peer-group Neighbor
 neighbor 10.255.255.5 peer-group Neighbor
 !
 address-family ipv6
 neighbor Neighbor activate
 no neighbor 10.255.255.4 activate
 no neighbor 10.255.255.4 capability graceful-restart
 no neighbor 10.255.255.5 activate
 no neighbor 10.255.255.5 capability graceful-restart
 exit-address-family
!
ip route 0.0.0.0/0 10.255.10.1
!
ip prefix-list /Common/myPrefixList1 seq 10 permit 10.0.0.0/8 ge 16
!
line con 0
 login
line vty 0 39
 login
!
end

Validate BGP on BIG-IP using imish

f5vm01.example.com[0]>show ip bgp summary
BGP router identifier 10.255.20.4, local AS number 65530
BGP table version is 6
2 BGP AS-PATH entries
0 BGP community entries

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.255.255.4    4 65515      20      17        6    0    0 00:02:38        3
10.255.255.5    4 65515      19      20        6    0    0 00:02:38        3

Total number of neighbors 2

##
f5vm01.example.com[0]>show ip bgp
BGP table version is 6, local router ID is 10.255.20.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, l - labeled
              S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric     LocPrf     Weight Path
*  10.1.0.0/16      10.255.255.5             0                     0 65515 i
*>                  10.255.255.4             0                     0 65515 i
*  10.2.0.0/16      10.255.255.5             0                     0 65515 i
*>                  10.255.255.4             0                     0 65515 i
*> 10.100.0.0/16    0.0.0.0                                    32768 ?
*> 10.101.0.0/16    0.0.0.0                                    32768 ?
*> 10.102.0.0/16    0.0.0.0                                    32768 ?
*  10.255.0.0/16    10.255.255.5             0                     0 65515 i
*>                  10.255.255.4             0                     0 65515 i

Total number of prefixes 6

Troubleshooting

If you don't see routes in the spoke VNets, then try deleting the VNet peering and re-run Terraform to have it create the peer again. If you happen to run into this issue, open an issue directly with Azure support to provide feedback.

You can view BIG-IP onboard logs in /var/log/cloud. Review logs for failure message.

You can view BIG-IP onboard config files in /config/cloud. Review the declarative onboarding JSON file as well as the runtime init YAML file for accuracy. Did the variables render correctly?

If BIG-IP imish commands do not provide results to "show ip bgp" or "show run" but you do see "tmsh list net routing", then something happen in the preview tmsh BGP/routing feature. You should simply delete and recreate the device.

# taint BIG-IP resource
terraform taint module.bigip[0].azurerm_linux_virtual_machine.f5vm01
terraform taint module.bigip[0].azurerm_virtual_machine_extension.vmext
# re-run terraform
./setup.sh

You can manually re-run BIG-IP Runtime init and force declarations to re-run. SSH to the BIG-IP, enter 'bash' mode and run this command.

f5-bigip-runtime-init --config-file /config/cloud/runtime-init-conf.yaml

Serial Logs

Review the serial logs for the Azure virtual machine. Login to the Azure portal, open "Virtual Machines", then locate your instance...click it. Hit Serial Console. Then review the serial logs for errors.

Onboard Logs

Depending on where onboard fails, you can attempt SSH login and try to troubleshoot further. Inspect the /config/cloud directory for correct runtime init YAML files. Inspect the /var/log/cloud location for error logs.

F5 Automation Toolchain Components

F5 BIG-IP Runtime Init uses the F5 Automation Toolchain for configuration of BIG-IP instances. Any errors thrown from these components will be surfaced in the bigIpRuntimeInit.log (or a custom log location as specified below).

Help with troubleshooting individual Automation Toolchain components can be found at F5's Public Cloud Docs:

DO: https://clouddocs.f5.com/products/extensions/f5-declarative-onboarding/latest/troubleshooting.html
AS3: https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/troubleshooting.html
FAST: https://clouddocs.f5.com/products/extensions/f5-appsvcs-templates/latest/userguide/troubleshooting.html
TS: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/userguide/troubleshooting.html
CFE: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/troubleshooting.html

Cleanup

Use the following command to destroy all of the resources

./destroy.sh

Requirements

Name	Version
terraform	>= 1.2.0
azurerm	>= 3.48.0

Providers

Name	Version
azurerm	>= 3.48.0
local	n/a
random	n/a

Modules

Name	Source	Version
app	Azure/compute/azurerm	4.0
bigip	F5Networks/bigip-module/azure	1.2.8
client	Azure/compute/azurerm	4.0
network	Azure/vnet/azurerm	n/a
nsg-external	Azure/network-security-group/azurerm	n/a
nsg-internal	Azure/network-security-group/azurerm	n/a
nsg-mgmt	Azure/network-security-group/azurerm	n/a

Resources

Name	Type
azurerm_key_vault_access_policy.main	resource
azurerm_public_ip.routeServerPip	resource
azurerm_resource_group.rg	resource
azurerm_route_table.rt	resource
azurerm_virtual_hub.routeServer	resource
azurerm_virtual_hub_bgp_connection.bigip	resource
azurerm_virtual_hub_ip.routeServerIp	resource
azurerm_virtual_network_peering.hubToSpoke	resource
azurerm_virtual_network_peering.spokeToHub	resource
random_id.buildSuffix	resource
azurerm_key_vault.main	data source
azurerm_subnet.externalSubnetHub	data source
azurerm_subnet.internalSubnetHub	data source
azurerm_subnet.mgmtSubnetHub	data source
azurerm_subnet.routeServerSubnetHub	data source
azurerm_subscription.main	data source
azurerm_user_assigned_identity.main	data source
local_file.appOnboard	data source

Inputs

Name	Description	Type	Default	Required
ssh_key	public key used for authentication in /path/file format (e.g. /.ssh/id_rsa.pub)	`string`	n/a	yes
AS3_URL	URL to download the BIG-IP Application Service Extension 3 (AS3) module	`string`	`"https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.43.0/f5-appsvcs-3.43.0-2.noarch.rpm"`	no
DO_URL	URL to download the BIG-IP Declarative Onboarding module	`string`	`"https://github.com/F5Networks/f5-declarative-onboarding/releases/download/v1.36.1/f5-declarative-onboarding-1.36.1-1.noarch.rpm"`	no
FAST_URL	URL to download the BIG-IP FAST module	`string`	`"https://github.com/F5Networks/f5-appsvcs-templates/releases/download/v1.24.0/f5-appsvcs-templates-1.24.0-1.noarch.rpm"`	no
INIT_URL	URL to download the BIG-IP runtime init	`string`	`"https://cdn.f5.com/product/cloudsolutions/f5-bigip-runtime-init/v1.6.0/dist/f5-bigip-runtime-init-1.6.0-1.gz.run"`	no
TS_URL	URL to download the BIG-IP Telemetry Streaming module	`string`	`"https://github.com/F5Networks/f5-telemetry-streaming/releases/download/v1.32.0/f5-telemetry-1.32.0-2.noarch.rpm"`	no
adminSrcAddr	Allowed Admin source IP prefix	`string`	`"0.0.0.0/0"`	no
availability_zone	Azure Availability Zone for BIG-IP 1	`number`	`1`	no
az_keyvault_authentication	Whether to use key vault to pass authentication	`bool`	`false`	no
bigIqHost	This is the BIG-IQ License Manager host name or IP address	`string`	`""`	no
bigIqHypervisor	BIG-IQ hypervisor	`string`	`"azure"`	no
bigIqLicensePool	BIG-IQ license pool name	`string`	`""`	no
bigIqLicenseType	BIG-IQ license type	`string`	`"licensePool"`	no
bigIqPassword	Admin Password for BIG-IQ	`string`	`"Default12345!"`	no
bigIqSkuKeyword1	BIG-IQ license SKU keyword 1	`string`	`"key1"`	no
bigIqSkuKeyword2	BIG-IQ license SKU keyword 2	`string`	`"key2"`	no
bigIqUnitOfMeasure	BIG-IQ license unit of measure	`string`	`"hourly"`	no
bigIqUsername	Admin name for BIG-IQ	`string`	`"azureuser"`	no
bigip_version	BIG-IP Version	`string`	`"16.1.303000"`	no
dns_server	Leave the default DNS server the BIG-IP uses, or replace the default DNS server with the one you want to use	`string`	`"8.8.8.8"`	no
dns_suffix	DNS suffix for your domain in the GCP project	`string`	`"example.com"`	no
f5_password	BIG-IP Password or Key Vault secret name (value should be Key Vault secret name when az_key_vault_authentication = true, ex. my-bigip-secret)	`string`	`"Default12345!"`	no
f5_username	User name for the BIG-IP	`string`	`"azureuser"`	no
image_name	F5 SKU (image) to deploy. Note: The disk size of the VM will be determined based on the option you select. Important: If intending to provision multiple modules, ensure the appropriate value is selected, such as AllTwoBootLocations or AllOneBootLocation.	`string`	`"f5-big-best-plus-hourly-25mbps"`	no
instanceCountBigIp	Number of BIG-IP instances to deploy	`number`	`1`	no
instance_type	Azure instance type to be used for the BIG-IP VE	`string`	`"Standard_DS4_v2"`	no
keyvault_name	Name of Key Vault	`string`	`null`	no
keyvault_rg	The name of the resource group in which the Azure Key Vault exists	`string`	`""`	no
keyvault_secret	Name of Key Vault secret with BIG-IP password	`string`	`null`	no
libs_dir	Directory on the BIG-IP to download the A&O Toolchain into	`string`	`"/config/cloud/azure/node_modules"`	no
location	Azure Location of the deployment	`string`	`"westus2"`	no
ntp_server	Leave the default NTP server the BIG-IP uses, or replace the default NTP server with the one you want to use	`string`	`"0.us.pool.ntp.org"`	no
product	Azure BIG-IP VE Offer	`string`	`"f5-big-ip-best"`	no
projectPrefix	This value is inserted at the beginning of each Azure object (alpha-numeric, no special character)	`string`	`"demo"`	no
resourceOwner	This is a tag used for object creation. Example is last name.	`string`	`null`	no
timezone	If you would like to change the time zone the BIG-IP uses, enter the time zone you want to use. This is based on the tz database found in /usr/share/zoneinfo (see the full list here). Example values: UTC, US/Pacific, US/Eastern, Europe/London or Asia/Singapore.	`string`	`"UTC"`	no
user_identity	The ID of the managed user identity to assign to the BIG-IP instance	`string`	`null`	no
vm_name	Prefix for BIG-IP instance name. If empty, default is 'bigip' string + prefix + random_id	`string`	`""`	no

Outputs

Name	Description
appPrivateIP	The private ip address allocated for the webapp in Spoke 2
appPublicIP	The public ip address allocated for the app in Spoke 2
bigip-private-ips	The private ip address for BIG-IP
bigipPassword	The password for the BIG-IP (if dynamic_password is choosen it will be random generated password or if azure_keyvault is choosen it will be key vault secret name )
bigipPublicIP	The public ip address allocated for the BIG-IP
bigipUserName	The user name for the BIG-IP
clientPrivateIP	The private ip address allocated for the client/jumphost in Spoke 1
clientPublicIP	The public ip address allocated for the client/jumphost in Spoke 1
vnetIdHub	Hub VNet ID
vnetIdSpoke1	Spoke1 VNet ID
vnetIdSpoke2	Spoke2 VNet ID

How to Contribute

Submit a pull request

Authors

Jeff Giroux

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Description

To Do

Issues

Diagram

Prerequisites

Login to Azure Environment

Usage example

Test your setup

Troubleshooting

Serial Logs

Onboard Logs

F5 Automation Toolchain Components

Cleanup

Requirements

Providers

Modules

Resources

Inputs

Outputs

How to Contribute

Authors

Files

README.md

Latest commit

History

README.md

File metadata and controls

Description

To Do

Issues

Diagram

Prerequisites

Login to Azure Environment

Usage example

Test your setup

Troubleshooting

Serial Logs

Onboard Logs

F5 Automation Toolchain Components

Cleanup

Requirements

Providers

Modules

Resources

Inputs

Outputs

How to Contribute

Authors