Skip to content

Latest commit

 

History

History

HA_via_api

README.md

Deploying BIG-IP VEs in AWS Across-Net - High Availability (Active/Standby): 3-NIC

To Do

  • Community support only. Template is not F5 supported.
  • Route table created for CFE demonstration but not associated with subnets

Issues

  • Find an issue? Fork, clone, create branch, fix and PR. I'll review and merge into the main branch. Or submit a GitHub issue with all necessary details and logs.

Contents

Introduction

This solution uses a Terraform template to launch a 3-NIC deployment of a cloud-focused BIG-IP VE cluster (Active/Standby) in AWS across two AWS Availability Zones. Traffic flows to the BIG-IP VE which then processes the traffic to application servers. The BIG-IP VE instance is running with multiple interfaces: management, external, internal. NIC1 is associated with the external network.

The BIG-IP VEs have the Local Traffic Manager (LTM) module enabled to provide advanced traffic management functionality. In addition, the Application Security Module (ASM) can be enabled to provide F5's L4/L7 security features for web application firewall (WAF) and bot protection.

The BIG-IP's configuration, now defined in a single convenient YAML or JSON F5 BIG-IP Runtime Init configuration file, leverages F5 Automation Tool Chain declarations which are easier to author, validate and maintain as code. For instance, if you need to change the configuration on the BIG-IPs in the deployment, you update the instance model by passing a new config file (which references the updated Automation Toolchain declarations) via template's runtimeConfig input parameter. New instances will be deployed with the updated configurations.

Prerequisites

  • Accepted the EULA for the F5 image in the AWS marketplace. If you have not deployed BIG-IP VE in your environment before, search for F5 in the Marketplace and then click Accept Software Terms. This only appears the first time you attempt to launch an F5 image. By default, this solution deploys the F5 BIG-IP BEST with IPI and Threat Campaigns (PAYG, 25Mbps) images. For more information, see K14810: Overview of BIG-IP VE license and throughput limits.

  • Important: When you configure the admin password for the BIG-IP VE in the template, you cannot use the character #. Additionally, there are a number of other special characters that you should avoid using for F5 product user accounts. See K2873 for details.

  • This template requires one or more service accounts for the BIG-IP instance to perform various tasks:

  • The HA BIG-IP VMs use AWS IAM role for the failover

  • These BIG-IP VMs are deployed across different Availability Zones

  • This template requires programmatic API credentials to deploy the Terraform AWS provider and build out all the neccessary AWS objects

  • Passwords and secrets can be located in AWS Secrets Manager.

    • Set aws_secretmanager_auth to 'true'
    • Set aws_secretmanager_secret_id to the AWS secret ID
    • Set aws_iam_instance_profile to an existing IAM profile
      • Note: an IAM profile will be created if not supplied

  • This templates deploys into an EXISTING networking stack. You are required to have an existing VPC network, subnets, and security groups.

    • A NAT gateway or public IP is also required for outbound Internet traffic
    • If you require a new network first, see the Infrastructure Only folder to get started

Important Configuration Notes

  • Variables are configured in variables.tf

  • Sensitive variables like AWS SSH keys are configured in terraform.tfvars or AWS Secrets Manager

    • Note: Other items like BIG-IP password can be stored in AWS Secrets Manager. Refer to the Prerequisites.
    • The BIG-IP instance will query AWS Metadata API to retrieve the service account's token for authentication
    • The BIG-IP instance will then use the secret name and the service account's token to query AWS Metadata API and dynamically retrieve the password for device onboarding

  • This template uses BIG-IP Runtime Init for the initial configuration. As part of the onboarding script, it will download the F5 Toolchain RPMs automatically. See the AS3 documentation and DO documentation for details on how to use AS3 and Declarative Onboarding on your BIG-IP VE(s). The Telemetry Streaming extension is also downloaded and can be configured to point to AWS Cloud Watch. The Cloud Failover Extension documentation is also available.

  • Files

    • bigip.tf - resources for BIG-IP, NICs, public IPs
    • iam.tf - resources to create IAM roles and permissions
    • main.tf - resources for provider, versions
    • f5_onboard.tmpl - onboarding script which is run by user-data. This script is responsible for downloading the neccessary F5 Automation Toolchain RPM files, installing them, and then executing the onboarding REST calls via the BIG-IP Runtime Init tool.

BYOL Licensing

This template uses PayGo BIG-IP image for the deployment (as default). If you would like to use BYOL licenses, then these following steps are needed:

  1. Find available images/versions with "byol" in SKU name using AWS CLI:
        aws ec2 describe-images \
          --region us-west-2 \
          --filters "Name=name,Values=*BIGIP*16.1.3.1*BYOL*" \
          --query 'Images[*].[ImageId,Name]'

        #Output similar to this...
        [
            "ami-089182acbfc02e3bf",
            "F5 BIGIP-16.1.3.1-0.0.11 BYOL-All Modules 2Boot Loc-220721050816-5f5a1994-65df-4235-b79c-a3ea049dc1db"
        ],
  1. In the "variables.tf", modify f5_ami_search_name with a value from previous output
        # BIGIP Image
        variable "f5_ami_search_name" { default = "F5 BIGIP-16.1.3.1* BYOL-All* 2Boot*" }
  1. In the "variables.tf", modify license1 and license2 with valid regkeys
        # BIGIP Setup
        variable license1 { default = "" }
        variable license2 { default = "" }
  1. In the "f5_onboard.tmpl", add the "myLicense" block under the "Common" declaration (example here)
        myLicense:
          class: License
          licenseType: regKey
          regKey: '${regKey}'

BIG-IQ License Manager

This template uses PayGo BIG-IP image for the deployment (as default). If you would like to use BYOL/ELA/Subscription licenses from BIG-IQ License Manager (LM), then these following steps are needed:

  1. Find BYOL image. Reference BYOL Licensing step #1.
  2. Replace BIG-IP f5_ami_search_name in "variables.tf". Reference BYOL Licensing step #2.
  3. In the "variables.tf", modify the BIG-IQ license section to match your environment
  4. In the "f5_onboard.tmpl", add the "myLicense" block under the "Common" declaration (example here)
        myLicense:
          class: License
          licenseType: ${bigIqLicenseType}
          bigIqHost: ${bigIqHost}
          bigIqUsername: ${bigIqUsername}
          bigIqPassword: ${bigIqPassword}
          licensePool: ${bigIqLicensePool}
          skuKeyword1: ${bigIqSkuKeyword1}
          skuKeyword2: ${bigIqSkuKeyword2}
          unitOfMeasure: ${bigIqUnitOfMeasure}
          reachable: false
          hypervisor: ${bigIqHypervisor}
          overwrite: true

Requirements

Name Version
terraform >= 1.2.0
aws >= 4.59.0

Providers

Name Version
aws 4.59.0
random 3.4.3

Modules

Name Source Version
bigip F5Networks/bigip-module/aws 1.1.11
bigip2 F5Networks/bigip-module/aws 1.1.11

Resources

Name Type
aws_ec2_tag.bigip2_ext_label resource
aws_ec2_tag.bigip2_ext_nicmap resource
aws_ec2_tag.bigip2_int_label resource
aws_ec2_tag.bigip2_int_nicmap resource
aws_ec2_tag.bigip2_vip_ips resource
aws_ec2_tag.bigip2_vip_label resource
aws_ec2_tag.bigip_ext_label resource
aws_ec2_tag.bigip_ext_nicmap resource
aws_ec2_tag.bigip_int_label resource
aws_ec2_tag.bigip_int_nicmap resource
aws_iam_instance_profile.bigip_profile resource
aws_iam_role.bigip_role resource
aws_route_table.main resource
aws_s3_bucket.main resource
aws_s3_bucket_server_side_encryption_configuration.main resource
random_id.buildSuffix resource
aws_ami.f5_ami data source
aws_caller_identity.main data source
aws_eip.bigip2_vip data source
aws_iam_policy_document.bigip_policy data source
aws_iam_policy_document.bigip_role data source
aws_secretsmanager_secret.password data source
aws_secretsmanager_secret_version.current data source
aws_vpc.main data source

Inputs

Name Description Type Default Required
AS3_URL URL to download the BIG-IP Application Service Extension 3 (AS3) module string "https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.43.0/f5-appsvcs-3.43.0-2.noarch.rpm" no
CFE_URL URL to download the BIG-IP Cloud Failover Extension module string "https://github.com/F5Networks/f5-cloud-failover-extension/releases/download/v1.14.0/f5-cloud-failover-1.14.0-0.noarch.rpm" no
DO_URL URL to download the BIG-IP Declarative Onboarding module string "https://github.com/F5Networks/f5-declarative-onboarding/releases/download/v1.36.1/f5-declarative-onboarding-1.36.1-1.noarch.rpm" no
FAST_URL URL to download the BIG-IP FAST module string "https://github.com/F5Networks/f5-appsvcs-templates/releases/download/v1.24.0/f5-appsvcs-templates-1.24.0-1.noarch.rpm" no
INIT_URL URL to download the BIG-IP runtime init string "https://cdn.f5.com/product/cloudsolutions/f5-bigip-runtime-init/v1.6.0/dist/f5-bigip-runtime-init-1.6.0-1.gz.run" no
TS_URL URL to download the BIG-IP Telemetry Streaming module string "https://github.com/F5Networks/f5-telemetry-streaming/releases/download/v1.32.0/f5-telemetry-1.32.0-2.noarch.rpm" no
adminSrcAddr Allowed Admin source IP prefix string "0.0.0.0/0" no
awsAz1 Availability zone, will dynamically choose one if left empty string "us-west-2a" no
awsAz2 Availability zone, will dynamically choose one if left empty string "us-west-2b" no
awsRegion aws region string "us-west-2" no
aws_iam_instance_profile Name of IAM role to assign to the BIG-IP instance string null no
aws_secretmanager_auth Whether to use secret manager to pass authentication bool false no
aws_secretmanager_secret_id The ARN of Secrets Manager secret with BIG-IP password string null no
bigIqHost This is the BIG-IQ License Manager host name or IP address string "" no
bigIqHypervisor BIG-IQ hypervisor string "aws" no
bigIqLicensePool BIG-IQ license pool name string "" no
bigIqLicenseType BIG-IQ license type string "licensePool" no
bigIqPassword Admin Password for BIG-IQ string "Default12345!" no
bigIqSkuKeyword1 BIG-IQ license SKU keyword 1 string "key1" no
bigIqSkuKeyword2 BIG-IQ license SKU keyword 2 string "key2" no
bigIqUnitOfMeasure BIG-IQ license unit of measure string "hourly" no
bigIqUsername Admin name for BIG-IQ string "azureuser" no
cfe_managed_route A route can used for testing managed-route failover. Enter address prefix like x.x.x.x/x string "0.0.0.0/0" no
dns_server Leave the default DNS server the BIG-IP uses, or replace the default DNS server with the one you want to use string "8.8.8.8" no
ec2_instance_type AWS instance type for the BIG-IP string "m5n.xlarge" no
ec2_key_name AWS EC2 Key name for SSH access string null no
extNsg ID of external security group string null no
extSubnetAz1 ID of External subnet AZ1 string null no
extSubnetAz2 ID of External subnet AZ2 string null no
f5_ami_search_name AWS AMI search filter to find correct BIG-IP VE for region string "F5 BIGIP-16.1.3.2* PAYG-Best Plus 200Mbps*" no
f5_cloud_failover_label This is a tag used for F5 Cloud Failover extension. Must match value of 'f5_cloud_failover_label' in externalnic_failover_tags and internalnic_failover_tags. string "myFailover" no
f5_password BIG-IP Password or Secret ARN (value should be ARN of secret when aws_secretmanager_auth = true, ex. arn:aws:secretsmanager:us-west-2:1234:secret:bigip-secret-abcd) string "Default12345!" no
f5_username User name for the BIG-IP (Note: currenlty not used. Defaults to 'admin' based on AMI string "admin" no
intNsg ID of internal security group string null no
intSubnetAz1 ID of Internal subnet AZ1 string null no
intSubnetAz2 ID of Internal subnet AZ2 string null no
libs_dir Directory on the BIG-IP to download the A&O Toolchain into string "/config/cloud/aws/node_modules" no
license1 The license token for the 1st F5 BIG-IP VE (BYOL) string "" no
license2 The license token for the 2nd F5 BIG-IP VE (BYOL) string "" no
mgmtNsg ID of management security group string null no
mgmtSubnetAz1 ID of Management subnet AZ1 string null no
mgmtSubnetAz2 ID of Management subnet AZ2 string null no
ntp_server Leave the default NTP server the BIG-IP uses, or replace the default NTP server with the one you want to use string "0.us.pool.ntp.org" no
projectPrefix This value is inserted at the beginning of each AWS object (alpha-numeric, no special character) string "demo" no
resourceOwner This is a tag used for object creation. Example is last name. string null no
timezone If you would like to change the time zone the BIG-IP uses, enter the time zone you want to use. This is based on the tz database found in /usr/share/zoneinfo (see the full list here). Example values: UTC, US/Pacific, US/Eastern, Europe/London or Asia/Singapore. string "UTC" no
vpcId The AWS network VPC ID string null no

Outputs

Name Description
f5vm01_ext_private_ip f5vm01 external primary IP address (self IP)
f5vm01_ext_public_ip f5vm01 external public IP address (self IP)
f5vm01_ext_secondary_ip f5vm01 external secondary IP address (VIP)
f5vm01_instance_ids f5vm01 management device name
f5vm01_int_private_ip f5vm01 internal primary IP address
f5vm01_mgmt_pip_url f5vm01 management public URL
f5vm01_mgmt_private_ip f5vm01 management private IP address
f5vm01_mgmt_public_ip f5vm01 management public IP address
f5vm02_ext_private_ip f5vm02 external primary IP address (self IP)
f5vm02_ext_public_ip f5vm02 external public IP address (self IP)
f5vm02_ext_secondary_ip f5vm02 external secondary IP address (VIP)
f5vm02_instance_ids f5vm02 management device name
f5vm02_int_private_ip f5vm01 internal primary IP address
f5vm02_mgmt_pip_url f5vm02 management public URL
f5vm02_mgmt_private_ip f5vm02 management private IP address
f5vm02_mgmt_public_ip f5vm02 management public IP address
public_vip Public IP for the BIG-IP listener (VIP)
public_vip_url public URL for application
route_table Route table ID
storage_bucket AWS storage bucket ARN

Installation Example

To run this Terraform template, perform the following steps:

  1. Clone the repo to your favorite location
  2. Modify terraform.tfvars with the required information
    # BIG-IP Environment
    adminSrcAddr  = "0.0.0.0/0"
    vpcId         = "vpc-1234"
    mgmtSubnetAz1 = "subnet-1111"
    mgmtSubnetAz2 = "subnet-2222"
    extSubnetAz1  = "subnet-3333"
    extSubnetAz2  = "subnet-4444"
    intSubnetAz1  = "subnet-5555"
    intSubnetAz2  = "subnet-6666"
    mgmtNsg       = "sg-1111"
    extNsg        = "sg-3333"
    intNsg        = "sg-5555"
    ec2_key_name  = "mySshKey123"
    f5_username   = "admin"
    f5_password   = "Default12345!"

    # AWS Environment
    awsRegion     = "us-west-2"
    projectPrefix = "mydemo"
    resourceOwner = "myname"

    # Secrets Manager - Uncomment to use Secret Manager integration
    #aws_secretmanager_auth      = true
    #aws_secretmanager_secret_id = "arn:aws:secretsmanager:us-west-2:xxxx:secret:mySecret123"
    #aws_iam_instance_profile    = "myRole123"
  1. Initialize the directory
    terraform init
  1. Test the plan and validate errors
    terraform plan
  1. Finally, apply and deploy
    terraform apply
  1. When done with everything, don't forget to clean up!
    terraform destroy

Configuration Example

The following is an example configuration diagram for this solution deployment. In this scenario, all access to the BIG-IP VE cluster (Active/Standby) is direct to each BIG-IP via the management interface. The IP addresses in this example may be different in your implementation.

Configuration Example

Documentation

For more information on F5 solutions for AWS, including manual configuration procedures for some deployment scenarios, see the AWS section of F5 CloudDocs. Also check out the Using Cloud Templates for BIG-IP in AWS on DevCentral. This particular HA example is based on the BIG-IP Failover F5 AWS Cloud Template on GitHub.

Creating Virtual Servers on the BIG-IP VE

In order to pass traffic from your clients to the servers through the BIG-IP system, you must create a virtual server on the BIG-IP VE. In this template, the AS3 declaration creates 2 VIPs: one for each BIG-IP. It is preconfigured as an example.

Note: These next steps illustrate the manual way in the GUI to create a virtual server

  1. Open the BIG-IP VE Configuration utility
  2. Click Local Traffic > Virtual Servers
  3. Click the Create button
  4. Type a name in the Name field
  5. Type an address (ex. 0.0.0.0/0) in the Destination/Mask field
  6. Type a port (ex. 80) in the Service Port
  7. Configure the rest of the virtual server as appropriate
  8. Select a pool name from the Default Pool list
  9. Click the Finished button
  10. Repeat as necessary for other applications

Redeploy BIG-IP for Replacement or Upgrade

This example illustrates how to replace or upgrade the BIG-IP VE.

  1. Change the f5_ami_search_name variable to the desired release
  2. Revoke the problematic BIG-IP VE's license (if BYOL)
  3. Run command
terraform taint module.bigip.aws_instance.f5_bigip
terraform taint module.bigip2.aws_instance.f5_bigip
  1. Run command
terraform apply

Troubleshooting

Serial Logs

Review the serial logs for the AWS virtual machine. Login to the AWS portal, open "EC2", then locate your instance...click it. Hit Actions > Monitor and Troubleshoot > Get system log. Then review the serial logs for errors.

Onboard Logs

Depending on where onboard fails, you can attempt SSH login and try to troubleshoot further. Inspect the /config/cloud directory for correct runtime init YAML files. Inspec the /var/log/cloud location for error logs.

F5 Automation Toolchain Components

F5 BIG-IP Runtime Init uses the F5 Automation Toolchain for configuration of BIG-IP instances. Any errors thrown from these components will be surfaced in the bigIpRuntimeInit.log (or a custom log location as specified below).

Help with troubleshooting individual Automation Toolchain components can be found at F5's Public Cloud Docs: