create_environment_v2

Module Cloud Composer Environment (V2)

This module is used to create a Cloud Composer V2 environment.

Compatibility

This module is meant for use with Terraform 1.3+ and tested using Terraform 1.3+. If you find incompatibilities using Terraform >=1.3, please open an issue.

Version

Current version is 4.0. Upgrade guides:

• 3.X -> 4.0.
• 4.X -> 5.0.

module "simple-composer-environment" {
  source                               = "terraform-google-modules/composer/google//modules/create_environment_v2"
  version                              = "~> 5.0"
  project_id                           = var.project_id
  composer_env_name                    = "test-composer-env"
  region                               = "us-central1"
  composer_service_account             = var.composer_service_account
  network                              = "test-vpc"
  subnetwork                           = "test-subnet"
  pod_ip_allocation_range_name         = "test-subnet-pod-ip-name"
  service_ip_allocation_range_name     = "test-subnet-service-ip-name"
  grant_sa_agent_permission            = false
  environment_size                     = "ENVIRONMENT_SIZE_SMALL"
  enable_private_endpoint              = true
  use_private_environment              = true
  cloud_composer_connection_subnetwork = var.subnetwork_self_link
  enable_privately_used_public_ips     = var.enable_privately_used_public_ips

  scheduler = {
    cpu        = 0.5
    memory_gb  = 1.875
    storage_gb = 1
    count      = 2
  }

  web_server = {
    cpu        = 0.5
    memory_gb  = 1.875
    storage_gb = 1
  }

  worker = {
    cpu        = 0.5
    memory_gb  = 1.875
    storage_gb = 1
    min_count  = 2
    max_count  = 3
  }

  triggerer = {
    cpu       = 1
    memory_gb = 1
    count     = 2
  }

}

Inputs

Name	Description	Type	Default	Required
airflow_config_overrides	Airflow configuration properties to override. Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation".	map(string)	{}	no
cloud_composer_connection_subnetwork	Subnetwork self-link. When specified, the environment will use Private Service Connect instead of VPC peerings to connect to CloudSQL in the Tenant Project. IP address of psc endpoint is allocated from this subnet	string	null	no
cloud_composer_network_ipv4_cidr_block	The CIDR block from which IP range in tenant project will be reserved. Required if VPC peering is used to connect to CloudSql instead of PSC	string	null	no
cloud_data_lineage_integration	Whether or not Dataplex data lineage integration is enabled. Cloud Composer environments in versions composer-2.1.2-airflow-..* and newer)	bool	false	no
cloud_sql_ipv4_cidr	The CIDR block from which IP range in tenant project will be reserved for Cloud SQL private service access. Required if VPC peering is used to connect to CloudSql instead of PSC	string	null	no
composer_env_name	Name of Cloud Composer Environment	string	n/a	yes
composer_service_account	Service Account for running Cloud Composer.	string	null	no
enable_ip_masq_agent	Deploys 'ip-masq-agent' daemon set in the GKE cluster and defines nonMasqueradeCIDRs equals to pod IP range so IP masquerading is used for all destination addresses, except between pods traffic.	bool	false	no
enable_private_endpoint	Configure private access to the cluster endpoint. If true, access to the public endpoint of the GKE cluster is denied	bool	false	no
enable_privately_used_public_ips	When enabled, IPs from public (non-RFC1918) ranges can be used for pod_ip_allocation_range_name and service_ip_allocation_range_name.	bool	false	no
env_variables	Variables of the airflow environment.	map(string)	{}	no
environment_size	The environment size controls the performance parameters of the managed Cloud Composer infrastructure that includes the Airflow database. Values for environment size are: ENVIRONMENT_SIZE_SMALL, ENVIRONMENT_SIZE_MEDIUM, and ENVIRONMENT_SIZE_LARGE.	string	"ENVIRONMENT_SIZE_MEDIUM"	no
grant_sa_agent_permission	Cloud Composer relies on Workload Identity as Google API authentication mechanism for Airflow.	bool	true	no
image_version	The version of the aiflow running in the cloud composer environment.	string	"composer-2.5.0-airflow-2.6.3"	no
kms_key_name	Customer-managed Encryption Key fully qualified resource name, i.e. projects/project-id/locations/location/keyRings/keyring/cryptoKeys/key.	string	null	no
labels	The resource labels (a map of key/value pairs) to be applied to the Cloud Composer.	map(string)	{}	no
maintenance_end_time	Time window specified for recurring maintenance operations in RFC3339 format	string	null	no
maintenance_recurrence	Frequency of the recurring maintenance window in RFC5545 format.	string	null	no
maintenance_start_time	Time window specified for daily or recurring maintenance operations in RFC3339 format	string	"05:00"	no
master_authorized_networks	List of master authorized networks. If none are provided, disallow external access (except the cluster node IPs, which GKE automatically whitelists).	list(object({ cidr_block = string display_name = string }))	[]	no
master_ipv4_cidr	The CIDR block from which IP range in tenant project will be reserved for the GKE master. Required when use_private_environment and enable_private_endpoint is true	string	null	no
network	The VPC network to host the composer cluster.	string	n/a	yes
network_project_id	The project ID of the shared VPC's host (for shared vpc support)	string	""	no
pod_ip_allocation_range_name	The name of the subnet secondary range, used to allocate IP addresses for the pods.	string	null	no
project_id	Project ID where Cloud Composer Environment is created.	string	n/a	yes
pypi_packages	Custom Python Package Index (PyPI) packages to be installed in the environment. Keys refer to the lowercase package name (e.g. "numpy").	map(string)	{}	no
region	Region where the Cloud Composer Environment is created.	string	"us-central1"	no
resilience_mode	Cloud Composer 2.1.15 or newer only. The resilience mode states whether high resilience is enabled for the environment or not. Values for resilience mode are HIGH_RESILIENCE for high resilience and STANDARD_RESILIENCE for standard resilience	string	null	no
scheduled_snapshots_config	The recovery configuration settings for the Cloud Composer environment	object({ enabled = optional(bool, false) snapshot_location = optional(string) snapshot_creation_schedule = optional(string) time_zone = optional(string) })	null	no
scheduler	Configuration for resources used by Airflow schedulers.	object({ cpu = string memory_gb = number storage_gb = number count = number })	{ "count": 2, "cpu": 2, "memory_gb": 7.5, "storage_gb": 5 }	no
service_ip_allocation_range_name	The name of the subnet secondary range, used to allocate IP addresses for the Services.	string	null	no
storage_bucket	Name of an existing Cloud Storage bucket to be used by the environment	string	null	no
subnetwork	The name of the subnetwork to host the composer cluster.	string	n/a	yes
subnetwork_region	The subnetwork region of the shared VPC's host (for shared vpc support)	string	""	no
tags	Tags applied to all nodes. Tags are used to identify valid sources or targets for network firewalls.	set(string)	[]	no
task_logs_retention_storage_mode	The mode of storage for Airflow workers task logs. Values for storage mode are CLOUD_LOGGING_ONLY to only store logs in cloud logging and CLOUD_LOGGING_AND_CLOUD_STORAGE to store logs in cloud logging and cloud storage. Cloud Composer 2.0.23 or newer only	string	null	no
triggerer	Configuration for resources used by Airflow triggerer	object({ cpu = string memory_gb = number count = number })	null	no
use_private_environment	Create a private environment.	bool	false	no
web_server	Configuration for resources used by Airflow web server.	object({ cpu = string memory_gb = number storage_gb = number })	{ "cpu": 2, "memory_gb": 7.5, "storage_gb": 5 }	no
web_server_network_access_control	The network-level access control policy for the Airflow web server. If unspecified, no network-level access restrictions are applied	list(object({ allowed_ip_range = string description = string }))	null	no
worker	Configuration for resources used by Airflow workers.	object({ cpu = string memory_gb = number storage_gb = number min_count = number max_count = number })	{ "cpu": 2, "max_count": 6, "memory_gb": 7.5, "min_count": 2, "storage_gb": 5 }	no

Outputs

Name	Description
airflow_uri	URI of the Apache Airflow Web UI hosted within the Cloud Composer Environment.
composer_env	Cloud Composer Environment
composer_env_id	ID of Cloud Composer Environment.
composer_env_name	Name of the Cloud Composer Environment.
gcs_bucket	Google Cloud Storage bucket which hosts DAGs for the Cloud Composer Environment.
gke_cluster	Google Kubernetes Engine cluster used to run the Cloud Composer Environment.