Azure Route Server and BIG-IP using BGP and eCMP for traffic distribution
- Community support only. Template is not F5 supported.
- Finish BIG-IP setup with test app
- Remove test network virtuals (10.100, 10.101, 10.102)...testing only
- Find an issue? Fork, clone, create branch, fix and PR. I'll review and merge into the main branch. Or submit a GitHub issue with all necessary details and logs.
- Azure CLI
- Terraform
- Azure Subscription
- Azure User with 'Owner' role
# Login
az login
# Show subscriptions
az account show
# Set default
az account set -s <subscriptionId>- Clone the repo and open the solution's directory
git clone https://github.com/JeffGiroux/f5_terraform.git
cd f5_terraform/Azure/Route_Server/- Create the tfvars file and update it with your settings
cp admin.auto.tfvars.example admin.auto.tfvars
# MODIFY TO YOUR SETTINGS
vi admin.auto.tfvars- Run the setup script to deploy all of the components into your Azure account (remember that you are responsible for the cost of those components)
./setup.sh- View the created objects in Azure Portal. Choose a VM instance or NIC from a spoke VNet and view "Effective Routes". You will see BIG-IP advertised routes via BGP across the VNet peering. This deployment will launch a single BIG-IP instance, but if you modified 'instanceCountBigIp' then you will see identical network prefixes advertised from multiple BIG-IP devices (aka BGP peers) like the screenshot below. The 10.255.10.4 is BIG-IP #1 and 10.255.10.6 is BIG-IP #2.
- View BPG peering on the Azure Route Server object in the portal - https://aka.ms/routeserver
- Validate BGP peering on BIG-IP using tmsh
(tmos)# show net routing bgp
------------------------------------------
Net::BGP Instance (route-domain: 0)
------------------------------------------
Name myBGP
Local AS 65530
----------------------------------------------------------------------------
| Net::BGP Neighbor - 10.255.255.5 via 10.255.10.4
----------------------------------------------------------------------------
| Remote AS 0
| State established 0:06:24
| Notification Cease/Administratively Shutdown.
| Address Family IPv4 Unicast IPv6 Unicast
| Prefix
| Accepted 3
| Announced 6
| Table Version
| Local 6
| Neighbor 6
| Message/Notification/Queue Sent Received
| Message 27 26
| Notification 0 2
| Queued 0 0
| Route Refresh 0 0- View running config on BIG-IP using imish
(tmos)# imish
f5vm01.example.com[0]#show running-config
!
service password-encryption
!
bgp extended-asn-cap
!
router bgp 65530
bgp graceful-restart restart-time 120
aggregate-address 10.100.0.0/16 summary-only
aggregate-address 10.101.0.0/16 summary-only
aggregate-address 10.102.0.0/16 summary-only
redistribute kernel
neighbor Neighbor peer-group
neighbor Neighbor remote-as 65515
neighbor Neighbor ebgp-multihop 2
no neighbor Neighbor capability route-refresh
neighbor Neighbor soft-reconfiguration inbound
neighbor Neighbor prefix-list /Common/myPrefixList1 out
neighbor 10.255.255.4 peer-group Neighbor
neighbor 10.255.255.5 peer-group Neighbor
!
address-family ipv6
neighbor Neighbor activate
no neighbor 10.255.255.4 activate
no neighbor 10.255.255.4 capability graceful-restart
no neighbor 10.255.255.5 activate
no neighbor 10.255.255.5 capability graceful-restart
exit-address-family
!
ip route 0.0.0.0/0 10.255.10.1
!
ip prefix-list /Common/myPrefixList1 seq 10 permit 10.0.0.0/8 ge 16
!
line con 0
login
line vty 0 39
login
!
end- Validate BGP on BIG-IP using imish
f5vm01.example.com[0]>show ip bgp summary
BGP router identifier 10.255.20.4, local AS number 65530
BGP table version is 6
2 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.255.255.4 4 65515 20 17 6 0 0 00:02:38 3
10.255.255.5 4 65515 19 20 6 0 0 00:02:38 3
Total number of neighbors 2
##
f5vm01.example.com[0]>show ip bgp
BGP table version is 6, local router ID is 10.255.20.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, l - labeled
S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 10.1.0.0/16 10.255.255.5 0 0 65515 i
*> 10.255.255.4 0 0 65515 i
* 10.2.0.0/16 10.255.255.5 0 0 65515 i
*> 10.255.255.4 0 0 65515 i
*> 10.100.0.0/16 0.0.0.0 32768 ?
*> 10.101.0.0/16 0.0.0.0 32768 ?
*> 10.102.0.0/16 0.0.0.0 32768 ?
* 10.255.0.0/16 10.255.255.5 0 0 65515 i
*> 10.255.255.4 0 0 65515 i
Total number of prefixes 6If you don't see routes in the spoke VNets, then try deleting the VNet peering and re-run Terraform to have it create the peer again. If you happen to run into this issue, open an issue directly with Azure support to provide feedback.
You can view BIG-IP onboard logs in /var/log/cloud. Review logs for failure message.
You can view BIG-IP onboard config files in /config/cloud. Review the declarative onboarding JSON file as well as the runtime init YAML file for accuracy. Did the variables render correctly?
If BIG-IP imish commands do not provide results to "show ip bgp" or "show run" but you do see "tmsh list net routing", then something happen in the preview tmsh BGP/routing feature. You should simply delete and recreate the device.
# taint BIG-IP resource
terraform taint module.bigip[0].azurerm_linux_virtual_machine.f5vm01
terraform taint module.bigip[0].azurerm_virtual_machine_extension.vmext
# re-run terraform
./setup.shYou can manually re-run BIG-IP Runtime init and force declarations to re-run. SSH to the BIG-IP, enter 'bash' mode and run this command.
f5-bigip-runtime-init --config-file /config/cloud/runtime-init-conf.yamlReview the serial logs for the Azure virtual machine. Login to the Azure portal, open "Virtual Machines", then locate your instance...click it. Hit Serial Console. Then review the serial logs for errors.
Depending on where onboard fails, you can attempt SSH login and try to troubleshoot further. Inspect the /config/cloud directory for correct runtime init YAML files. Inspect the /var/log/cloud location for error logs.
F5 BIG-IP Runtime Init uses the F5 Automation Toolchain for configuration of BIG-IP instances. Any errors thrown from these components will be surfaced in the bigIpRuntimeInit.log (or a custom log location as specified below).
Help with troubleshooting individual Automation Toolchain components can be found at F5's Public Cloud Docs:
- DO: https://clouddocs.f5.com/products/extensions/f5-declarative-onboarding/latest/troubleshooting.html
- AS3: https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/troubleshooting.html
- FAST: https://clouddocs.f5.com/products/extensions/f5-appsvcs-templates/latest/userguide/troubleshooting.html
- TS: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/userguide/troubleshooting.html
- CFE: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/troubleshooting.html
Use the following command to destroy all of the resources
./destroy.sh| Name | Version |
|---|---|
| terraform | >= 1.2.0 |
| azurerm | >= 3.48.0 |
| Name | Version |
|---|---|
| azurerm | >= 3.48.0 |
| local | n/a |
| random | n/a |
| Name | Source | Version |
|---|---|---|
| app | Azure/compute/azurerm | 4.0 |
| bigip | F5Networks/bigip-module/azure | 1.2.8 |
| client | Azure/compute/azurerm | 4.0 |
| network | Azure/vnet/azurerm | n/a |
| nsg-external | Azure/network-security-group/azurerm | n/a |
| nsg-internal | Azure/network-security-group/azurerm | n/a |
| nsg-mgmt | Azure/network-security-group/azurerm | n/a |
| Name | Type |
|---|---|
| azurerm_key_vault_access_policy.main | resource |
| azurerm_public_ip.routeServerPip | resource |
| azurerm_resource_group.rg | resource |
| azurerm_route_table.rt | resource |
| azurerm_virtual_hub.routeServer | resource |
| azurerm_virtual_hub_bgp_connection.bigip | resource |
| azurerm_virtual_hub_ip.routeServerIp | resource |
| azurerm_virtual_network_peering.hubToSpoke | resource |
| azurerm_virtual_network_peering.spokeToHub | resource |
| random_id.buildSuffix | resource |
| azurerm_key_vault.main | data source |
| azurerm_subnet.externalSubnetHub | data source |
| azurerm_subnet.internalSubnetHub | data source |
| azurerm_subnet.mgmtSubnetHub | data source |
| azurerm_subnet.routeServerSubnetHub | data source |
| azurerm_subscription.main | data source |
| azurerm_user_assigned_identity.main | data source |
| local_file.appOnboard | data source |
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| ssh_key | public key used for authentication in /path/file format (e.g. /.ssh/id_rsa.pub) | string |
n/a | yes |
| AS3_URL | URL to download the BIG-IP Application Service Extension 3 (AS3) module | string |
"https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.43.0/f5-appsvcs-3.43.0-2.noarch.rpm" |
no |
| DO_URL | URL to download the BIG-IP Declarative Onboarding module | string |
"https://github.com/F5Networks/f5-declarative-onboarding/releases/download/v1.36.1/f5-declarative-onboarding-1.36.1-1.noarch.rpm" |
no |
| FAST_URL | URL to download the BIG-IP FAST module | string |
"https://github.com/F5Networks/f5-appsvcs-templates/releases/download/v1.24.0/f5-appsvcs-templates-1.24.0-1.noarch.rpm" |
no |
| INIT_URL | URL to download the BIG-IP runtime init | string |
"https://cdn.f5.com/product/cloudsolutions/f5-bigip-runtime-init/v1.6.0/dist/f5-bigip-runtime-init-1.6.0-1.gz.run" |
no |
| TS_URL | URL to download the BIG-IP Telemetry Streaming module | string |
"https://github.com/F5Networks/f5-telemetry-streaming/releases/download/v1.32.0/f5-telemetry-1.32.0-2.noarch.rpm" |
no |
| adminSrcAddr | Allowed Admin source IP prefix | string |
"0.0.0.0/0" |
no |
| availability_zone | Azure Availability Zone for BIG-IP 1 | number |
1 |
no |
| az_keyvault_authentication | Whether to use key vault to pass authentication | bool |
false |
no |
| bigIqHost | This is the BIG-IQ License Manager host name or IP address | string |
"" |
no |
| bigIqHypervisor | BIG-IQ hypervisor | string |
"azure" |
no |
| bigIqLicensePool | BIG-IQ license pool name | string |
"" |
no |
| bigIqLicenseType | BIG-IQ license type | string |
"licensePool" |
no |
| bigIqPassword | Admin Password for BIG-IQ | string |
"Default12345!" |
no |
| bigIqSkuKeyword1 | BIG-IQ license SKU keyword 1 | string |
"key1" |
no |
| bigIqSkuKeyword2 | BIG-IQ license SKU keyword 2 | string |
"key2" |
no |
| bigIqUnitOfMeasure | BIG-IQ license unit of measure | string |
"hourly" |
no |
| bigIqUsername | Admin name for BIG-IQ | string |
"azureuser" |
no |
| bigip_version | BIG-IP Version | string |
"16.1.303000" |
no |
| dns_server | Leave the default DNS server the BIG-IP uses, or replace the default DNS server with the one you want to use | string |
"8.8.8.8" |
no |
| dns_suffix | DNS suffix for your domain in the GCP project | string |
"example.com" |
no |
| f5_password | BIG-IP Password or Key Vault secret name (value should be Key Vault secret name when az_key_vault_authentication = true, ex. my-bigip-secret) | string |
"Default12345!" |
no |
| f5_username | User name for the BIG-IP | string |
"azureuser" |
no |
| image_name | F5 SKU (image) to deploy. Note: The disk size of the VM will be determined based on the option you select. Important: If intending to provision multiple modules, ensure the appropriate value is selected, such as AllTwoBootLocations or AllOneBootLocation. | string |
"f5-big-best-plus-hourly-25mbps" |
no |
| instanceCountBigIp | Number of BIG-IP instances to deploy | number |
1 |
no |
| instance_type | Azure instance type to be used for the BIG-IP VE | string |
"Standard_DS4_v2" |
no |
| keyvault_name | Name of Key Vault | string |
null |
no |
| keyvault_rg | The name of the resource group in which the Azure Key Vault exists | string |
"" |
no |
| keyvault_secret | Name of Key Vault secret with BIG-IP password | string |
null |
no |
| libs_dir | Directory on the BIG-IP to download the A&O Toolchain into | string |
"/config/cloud/azure/node_modules" |
no |
| location | Azure Location of the deployment | string |
"westus2" |
no |
| ntp_server | Leave the default NTP server the BIG-IP uses, or replace the default NTP server with the one you want to use | string |
"0.us.pool.ntp.org" |
no |
| product | Azure BIG-IP VE Offer | string |
"f5-big-ip-best" |
no |
| projectPrefix | This value is inserted at the beginning of each Azure object (alpha-numeric, no special character) | string |
"demo" |
no |
| resourceOwner | This is a tag used for object creation. Example is last name. | string |
null |
no |
| timezone | If you would like to change the time zone the BIG-IP uses, enter the time zone you want to use. This is based on the tz database found in /usr/share/zoneinfo (see the full list here). Example values: UTC, US/Pacific, US/Eastern, Europe/London or Asia/Singapore. | string |
"UTC" |
no |
| user_identity | The ID of the managed user identity to assign to the BIG-IP instance | string |
null |
no |
| vm_name | Prefix for BIG-IP instance name. If empty, default is 'bigip' string + prefix + random_id | string |
"" |
no |
| Name | Description |
|---|---|
| appPrivateIP | The private ip address allocated for the webapp in Spoke 2 |
| appPublicIP | The public ip address allocated for the app in Spoke 2 |
| bigip-private-ips | The private ip address for BIG-IP |
| bigipPassword | The password for the BIG-IP (if dynamic_password is choosen it will be random generated password or if azure_keyvault is choosen it will be key vault secret name ) |
| bigipPublicIP | The public ip address allocated for the BIG-IP |
| bigipUserName | The user name for the BIG-IP |
| clientPrivateIP | The private ip address allocated for the client/jumphost in Spoke 1 |
| clientPublicIP | The public ip address allocated for the client/jumphost in Spoke 1 |
| vnetIdHub | Hub VNet ID |
| vnetIdSpoke1 | Spoke1 VNet ID |
| vnetIdSpoke2 | Spoke2 VNet ID |
Submit a pull request
Jeff Giroux