Skip to content

This repository aims to apply minimal permissions required for ARO clusters.

Notifications You must be signed in to change notification settings

rh-mobb/terraform-aro-permissions

Repository files navigation

Summary

This project aims to provide a Terraform module to make it easy to setup Azure permissions needed to install and manage ARO. The problem we see in the field, is that there are overlapping identities that need specific permission sets and this often times results in incorrect permissions and makes for a confusing experience.

WARN: It should be noted that certain product changes will force this to change. This is a community-supported project and you should consult your appropriate product documentation prior to using this in your environment to ensure it is appropriate for your needs.

Identities

This section identifies the individual identities that need ARO permissions. These identities will be used in verbiage when describing what permissions are needed.

NOTE: the Install Flags column helps to associate these service principals to actual ARO installation flags using the CLI.

Identity Type Install Flags Description
Cluster Service Principal Service Principal --client-id and --client-secret Runs all operations in the cluster and that interacts with the Azure API as part of operations within the cluster. Cluster Autoscaler and the Service Controller for Azure are two key components which leverage these credentials.
Installer User or Service Principal N/A Whomever the installation process is run as. This may be a user account or a service principal. This is who is logged in using the az login command.
Resource Provider Service Principal Service Principal N/A Azure Resource Provider that represents ARO. This is automatically created in the account when first running the setup step az provider register -n Microsoft.RedHatOpenShift --wait. This service principal can be found by running az ad sp list --filter "displayname eq 'Azure Red Hat OpenShift RP'".

Objects

This section defines the objects which need individual permissions.

NOTE: the Install Flags column helps to associate these objects to actual ARO installation flags using the CLI.

Object Type Install Flags Description
Subscription --subscription The highest level a permission will be applied. Inherits down to all objects within that subscription. This is not a mandatory flag and the subscription may be set based on how a user has logged in with az login.
ARO Resource Group --resource-group Resource group in the above subscription where the actual ARO object is created.
Cluster Resource Group --cluster-resource-group Resource group in the above subscription where the underlying ARO objects (e.g. VMs, load balancers) are created. This is created automatically as part of provisioning.
Network Resource Group --vnet-resource-group Resource group in the above subscription where network resources (e.g. VNET, NSG) exist. Some organizations will use the Cluster Resource Group for this purpose as well and do not need a dedicated Network Resource Group.
VNET --vnet VNET where the ARO cluster will be provisioned.
Network Security Group N/A Only required for BYO-NSG scenarios. Network security group, applied to the subnets. This is is pre-applied by the user to the subnets prior to installation.
Disk Encryption Set --disk-encryption-set The disk encryption set used to encrypt master and worker node disks.

Permissions

This section identifies what permissions are needed by each individual identity.

NOTE: row numbers are used to indicate in the code where permissions are aligned.

Permission Number Identity Object Permission Comment
1 Cluster Service Principal VNET Network Contributor or Minimal Network Permissions
2 Cluster Service Principal Network Security Group Network Contributor or Minimal Network Permissions Only needed if BYO-NSG is pre-attached to the subnet.
3 Cluster Service Principal ARO Resource Group Contributor
4 Cluster Service Principal Disk Encryption Set Other
5 Installer ARO Resource Group Contributor or Minimal ARO Permissions
6 Installer Network Resource Group Reader Only required if az aro create is used to install.
7 Installer Subscription User Access Administrator Only required if az aro create is used to install.
8 Installer Azure AD Directory Readers Only required if az aro create is used to install.
9 Installer VNET Network Contributor or Minimal Network Permissions Only required if az aro create is used to install.
10 Resource Provider Service Principal VNET Network Contributor or Minimal Network Permissions
11 Resource Provider Service Principal Network Security Group Network Contributor or Minimal Network Permissions
12 Resource Provider Service Principal Disk Encryption Set Other
13 Resource Provider Service Principal Cluster Resource Group Owner This permission does not need to pre-exist. It is applied when the Resource Provider Service Principal creates the resource group as part of installation. This is for documentation purposes only.

Minimal Network Permissions

In many cases, such as separation of duties and where network teams must provide infrastructure to consume, a reduced permission set lower than Network Contributor is required. This is possible, however it should be noted that product documentation indicates higher permissions and the product will be developed against that assumption unless otherwise noted.

The following permission, in place of Network Contributor, have been successful (including links to the code which validates the permissions). The VNET this applies to equates to the value of the --vnet flag in the az aro create command:

Needed always:

Needed when provided VNET subnets have route table(s) attached:

Needed when provided VNET subnets have NAT gateway(s) attached:

Minimal ARO Permissions

In addition to minimizing network permissions, the installer role may need minimal permissions as well. These permissions are as follows:

  • Microsoft.RedHatOpenShift/openShiftClusters/read
  • Microsoft.RedHatOpenShift/openShiftClusters/write
  • Microsoft.RedHatOpenShift/openShiftClusters/delete
  • Microsoft.RedHatOpenShift/openShiftClusters/listCredentials/action
  • Microsoft.RedHatOpenShift/openShiftClusters/listAdminCredentials/action

Other Permissions

In addition to the above, the following other permissions may be needed by specific identities:

Prereqs

Prior to running this module, the following must be satisfied:

  1. Must be logged in as an administrator user using the az login command. Because assigning permissions is an administrative task, it is assumed whomever is running this module is an administrator. Alternative to full tenant administrator permissions, a user that has the User Access Administrator role should be able to complete this task.

  2. Must have the az CLI installed and configured locally. There are some external commands ran in this module which makes this necessary. It is not ideal but it works for now.

  3. Must have the jq CLI installed locally. There are some external commands ran in this module which makes this necessary. It is not ideal but it works for now.

  4. Must have a VNET architecture pre-deployed and used as an input.

Examples

Examples of how to use this module are located in the examples directory. The examples thus far are:

  1. When ARO will be installed with the Azure az CLI - examples/cli
  2. When ARO will be installed with an automation tool like Terraform (using the Microsoft API) - examples/api

About

This repository aims to apply minimal permissions required for ARO clusters.

Resources

Stars

Watchers

Forks

Packages

No packages published