Cloudexport azure automation across resourcegroups (#45)

* Add kentik-side cloudexport configuration for azure
* Add service principal creation to further automate onboarding
* Azure cloudexport across resource groups

.github/workflows/azure-terraform.yml

@@ -21,10 +21,10 @@ jobs:
 
       # Initialize the Terraform code
       - name: (HELPER) Init Code 
-        uses: docker://hashicorp/terraform:0.12.29
+        uses: docker://hashicorp/terraform:1.0.0
         with:
           entrypoint: terraform
-          args: init cloud_Azure/terraform/module/tests/
+          args: -chdir=cloud_Azure/terraform/module/tests/ init
 
       # Lint the Terraform code 
       # Using: https://github.com/terraform-linters/tflint
@@ -36,7 +36,7 @@ jobs:
       # Validate the Terraform code using inbuilt
       # validate command
       - name: Validate Module
-        uses: docker://hashicorp/terraform:0.12.29
+        uses: docker://hashicorp/terraform:1.0.0
         with:
           entrypoint: terraform
-          args: validate cloud_Azure/terraform/module/tests/ -no-color
+          args: -chdir=cloud_Azure/terraform/module/tests/ validate -no-color

cloud_AWS/terraform/module/tests/main.tf

@@ -32,7 +32,7 @@ provider "aws" {
 }
 
 provider "kentik-cloudexport" {
-  email = "dummy@tesl.mail"
+  email = "dummy@test.mail"
   token = "dummy_token"
 }
 

cloud_Azure/README.md

@@ -1,5 +1,5 @@
-Code for preparing Azure cloud to work with portal.kentik.com [portal.kentik.com](https://portal.kentik.com)
+# Automation support for configuring flow log export from Azure to [Kentik](https://portal.kentik.com)
 
-Currently supported:
-* [terraform](terraform/module)
-* [ansible](ansible/roles/kentik_az)
+Currently supported frameworks:
+* [Terraform] (terrafrom/module)
+* [Ansible] (ansible/roles/kentik_az) 

cloud_Azure/terraform/module/README.md

@@ -1,12 +1,21 @@
 # Azure Kentik integration Terraform module
 
-Terraform module which creates Azure resources required for Kentik to enable integration
+Module supporting management of Azure and Kentik resources required for flow log export from Azure to Kentik.  
 
 Module enables:
-* Flow logs in existing Network Security Groups
+* Flow logs in all Network Security Groups (NSG) found in requested Resource Groups
 
 Module creates:
+* Service Principal for Kentik NSG Flow Exporter application
+* Reader and Contributor Roles for above mentioned Service Principal
+* One Storage Account for flow logs per requested Resource Group
+* One Flow log per NSG across all requested Resource Groups
+* Registers flow in Kentik platform per requested Resource Group
 
+All created resources are tagged with:  
+`app = "kentik_flow_log_exporter"`  
+
+Module assumes that NetworkWatcher resource exists in NetworkWatcherRG resource group in specified Azure location (see variable "location"). It is automatically created by Azure when VirtualNetwork is created or updated, [as per documentation.](https://docs.microsoft.com/en-us/azure/network-watcher/network-watcher-create)
 
 ## Usage
 
@@ -16,28 +25,29 @@ Module creates:
 module kentik_azure_integration {
   source  = "../../"
   location = var.location
-  resource_group_name = var.resource_group_name
-  principal_id = var.principal_id
   subscription_id = var.subscription_id
+  resource_group_names = var.resource_group_names
+  prefix = var.prefix
+  plan_id = var.plan_id
+  name = var.name
 }
 ```
 
 ## Examples
 
-* [All Network Security Groups in Resource Group](examples/all_nsg)
+* [All Network Security Groups in requested Resource Groups](examples/all_nsg)
 
 ## Demo
 * [Demo showing how to add list of subnets to Kentik portal using this module](demo) (TBD)
 
-## Note
-* this module creates Azure resources only. This won't register resources in Kentik platform automatically.
-
 ## Requirements
 
 | Name | Version |
 |------|---------|
-| terraform | >=0.12.0 |
-| azurem provider | >= =2.20.0 |
+| terraform | >= 1.0.0 |
+| azurerm provider | >= 2.85.0 |
+| azuread provider | >= 2.14.0 |
+| kentik-cloudexport provider | >= 0.4.1 |
 | null provider | >= 2.1.2 |
 | external provider | >= 2.0.0 |
 | python | >= 3.7.5 |
@@ -47,30 +57,25 @@ module kentik_azure_integration {
 
 ### Python and dependencies
 
-This module uses python to gather all NSG from Resource Group and expose it to terraform as external data source.
+This module uses python to gather all NSG from Resource Groups and expose it to terraform as external data source.
 To install python and its requirements:
 * [Install Python 3](https://docs.python.org/3/using/index.html)
 * [Install pip3](https://pip.pypa.io/en/stable/installing/)
 * Install packages: run `pip3 install -r ../../requirements.txt` in example directory
 
-## Providers
-
-| Name | Version |
-|------|---------|
-| azurem | >= =2.20.0 |
-| null | >= 2.1.2 |
-| external | >= 2.0.0 |
-
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | location | Azure location of the resources to gather logs | `string` | `` | yes |
 | subscription_id | Id of the subscription in which resource are located | `string` | `` | yes |
-| resource_group_name | Name of the resource group to gather logs from | `string` | `` | yes |
-| principal_id | Id of the Service Principal Id for kentik app connection | `string` | `` | yes |
-| prefix| Prefix for the naming resources created by this module | `string` | `kentik` | no |
-
+| resource_group_names | List of Resource Group names to gather logs from | `list of strings` | `` | yes |
+| prefix| Prefix for the naming resources created by this module | `string` | `` | yes |
+| plan_id | Billing plan ID | `string` | `` | yes |
+| name | Cloudexport entry name in Kentik | `string` | `` | yes |
+| flow_exporter_application_id | Kentik NSG Flow Exporter application ID | `string` | `a20ce222-63c0-46db-86d5-58551eeee89f` | no |
+| enabled | Defines if cloud export to Kentik is enabled | `bool` | true | no |
+| description | Cloudexport entry description in Kentik | `string` | `` | no |
 
 
 ## Outputs
@@ -79,5 +84,6 @@ To install python and its requirements:
 |------|-------------|
 | network_security_groups | Id's of the Network Security groups that logs will be gathered from |
 | subscription_id | Subscription Id |
-| resource_group | Resource group name |
-| storage_account | Storage account name where logs will be gathered |
+| resource_group_names | Resource group names |
+| storage_accounts | Storage account names where logs will be gathered |
+| principal_id | Principal ID created for Kentik NSG Flow Exporter application |

cloud_Azure/terraform/module/cloudexport.tf

@@ -0,0 +1,27 @@
+terraform {
+  required_providers {
+    kentik-cloudexport = {
+      source  = "kentik/kentik-cloudexport"
+      version = ">= 0.4.1"
+    }
+  }
+}
+
+# Creates one Kentik CloudExport for each requested Resource Group
+resource "kentik-cloudexport_item" "azure_export" {
+  count = length(var.resource_group_names)
+
+  name           = "${var.name}_${var.resource_group_names[count.index]}" # name must be unique
+  type           = "CLOUD_EXPORT_TYPE_KENTIK_MANAGED"
+  enabled        = var.enabled
+  description    = var.description
+  plan_id        = var.plan_id
+  cloud_provider = "azure"
+  azure {
+    location                   = var.location
+    resource_group             = var.resource_group_names[count.index]
+    storage_account            = azurerm_storage_account.logs_storage_account[count.index].name # storage accounts are mapped to resource groups 1:1
+    subscription_id            = var.subscription_id
+    security_principal_enabled = true
+  }
+}

cloud_Azure/terraform/module/examples/all_nsg/README.md

@@ -1,35 +1,51 @@
-# All Network Security Groups in Resource Group
+# All Network Security Groups in requested Resource Groups
 
-Configuration in this directory creates configuration for all Network Security Group in Resource Group.
+This example creates cloud export configuration for all Network Security Groups in requested Resource Groups.
+
+## Requirements
+
+* Information about Azure deployment: location, resource group names, subscription ID
+* Information about Kentik subscription: plan ID
+* Azure CLI - [Installation](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
+* Authenticating Azure CLI to your account - [Logging-in](https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli)
+* Kentik API credentials present in execution environment:
+  ```bash
+  export KTAPI_AUTH_EMAIL="[email protected]"
+  export KTAPI_AUTH_TOKEN="token123"
+  ```
 
 ## Usage
 
 To run this example you need to execute:
 ```
 $ terraform init
-$ terraform plan
-$ terraform apply
+$ terraform apply \  
+  --var subscription_id=<azure_subscription_id> \  
+  --var location=<azure_location> \
+  --var resource_group_names=<resource_group_names> \
+  --var prefix=<unique_prefix_for_azure_resources> \
+  --var plan_id=<kentik_plan_id> \
+  --var name=<kentik_cloudexport_name>
 ```
 
-## Requirements
-
-* Example requires location, resource group name, principal id, subscription
-* Installed and logged az-cli [Installation](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) [Logging](https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli)
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | location | Azure location of the resources to gather logs | `string` | `` | yes |
 | subscription_id | Id of the subscription in which resource are located | `string` | `` | yes |
-| resource_group_name | Name of the resource group to gather logs from | `string` | `` | yes |
-| principal_id | Id of the Service Principal Id for kentik app connection | `string` | `` | yes |
+| resource_group_names | List of Resource Group names to gather logs from | `list of strings` | `` | yes |
+| prefix| Prefix for the naming resources | `string` | `` | yes |
+| plan_id | Billing plan ID | `string` | `` | yes |
+| name | Cloudexport entry name in Kentik | `string` | `` | yes |
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
 | network_security_groups | Id's of the Network Security groups that logs will be gathered from |
 | subscription_id | Subscription Id |
-| resource_group | Resource group name |
-| storage_account | Storage account name where logs will be gathered |
+| resource_group_names | Resource group names |
+| storage_accounts | Storage account names where logs will be gathered |
+| principal_id | Principal ID created for Kentik NSG Flow Exporter application |

cloud_Azure/terraform/module/examples/all_nsg/main.tf

@@ -1,16 +1,32 @@
 terraform {
-  required_version = ">= 0.12.0"
+  required_version = ">= 1.0.0"
+  required_providers {
+    azurerm = {
+      source  = "hashicorp/azurerm"
+      version = ">= 2.85.0"
+    }
+    azuread = {
+      source = "hashicorp/azuread"
+      version = ">= 2.14.0"
+    }
+    kentik-cloudexport = {
+      source  = "kentik/kentik-cloudexport"
+      version = ">= 0.4.1"
+    }
+  }
 }
 
 provider "azurerm" {
-  version = "=2.20.0"
   features {}
 }
 
+
 module kentik_azure_integration {
   source  = "../../"
   location = var.location
-  resource_group_name = var.resource_group_name
-  principal_id = var.principal_id
   subscription_id = var.subscription_id
+  resource_group_names = var.resource_group_names
+  prefix = var.prefix
+  plan_id = var.plan_id
+  name = var.name
 }

cloud_Azure/terraform/module/examples/all_nsg/output.tf

@@ -7,10 +7,14 @@ output "subscription_id" {
   value = module.kentik_azure_integration.subscription_id
 }
 
-output "resource_group" {
-  value = module.kentik_azure_integration.resource_group
+output "resource_group_names" {
+  value = module.kentik_azure_integration.resource_group_names
 }
 
-output "storage_account" {
-  value = module.kentik_azure_integration.storage_account
+output "storage_accounts" {
+  value = module.kentik_azure_integration.storage_accounts
 }
+
+output "principal_id" {
+  value = module.kentik_azure_integration.principal_id
+}

cloud_Azure/terraform/module/examples/all_nsg/variables.tf

@@ -8,12 +8,21 @@ variable "subscription_id" {
     description = "Subscription Id"
 }
 
-variable "resource_group_name" {
-    type = string
-    description = "Resource group name"
+variable "resource_group_names" {
+    type = list
+    description = "List of resource group names"
 }
 
-variable "principal_id" {
+variable "prefix" {
     type = string
-    description = "Service Principal Id"
+    description = "Unique prefix to be used for resource creation; can only consist of lowercase letters and numbers, max length is 17"
+}
+variable "plan_id" {
+  description = "Billing plan ID"
+  type        = string
 }
+
+variable "name" {
+  description = "Cloudexport entry name in Kentik"
+  type        = string
+}

cloud_Azure/terraform/module/get_nsg.py

@@ -1,22 +1,32 @@
+import sys
+from typing import Dict, List
+
 from az.cli import az
 from terraform_external_data import terraform_external_data
 
-import sys
-
 
 @terraform_external_data
-def get_nsg_from_rg(query):
+def get_nsg_from_rg(query: Dict[str, str]) -> Dict[str, str]:
     """
-    Functions that gest all Network Security Groups from Resource Group
+    Gather all Network Security Groups for each requested Resource Group
     """
 
-    exit_code, result_dict, logs = az(
-        "network nsg list --resource-group " + str(sys.argv[1]) + " --query '[].id' -o json")
-    if exit_code == 0:
-        return {query['network_security_groups']: str(result_dict).strip("[]").replace("'", "").replace(" ", "")}
-    else:
-        print(logs)
+    if query["resource_group_names"] == "":
+        return {}
+
+    resource_group_names: List[str] = query["resource_group_names"].split(",")
+    result: Dict[str, str] = {}
+
+    for rg in resource_group_names:
+        exit_code, result_dict, logs = az(f"network nsg list --resource-group {rg} --query '[].id' -o json")
+        if exit_code == 0:
+            network_security_groups = str(result_dict).strip("[]").replace("'", "").replace(" ", "")
+            result[rg] = network_security_groups
+        else:
+            print(logs, file=sys.stderr)
+            exit(1)
+    return result  # result is consumed by function decorator
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     get_nsg_from_rg()