Add Dedicated Cluster type with networking options

[rjudin]

Please add deployment.sku="DEDICATED" with networking"Internet", "PrivateLink", "VPCPeering"

resource "confluentcloud_kafka_cluster" "test" {
  name           = "provider-test"
  environment_id = confluentcloud_environment.env0.id
  #bootstrap_servers = string
  service_provider = "aws" # AWS/GCP
  region           = "ap-southeast-1"
  availability     = "LOW" # LOW(single-zone) or HIGH(multi-zone)
  storage          = 5000  # Storage limit(GB)
  network_ingress  = 100   #Network ingress limit(MBps)
  network_egress   = 100   #Network egress limit(MBps)
  deployment = {           #"Deployment settings.  Currently only `sku` is supported."
    sku = "DEDICATED"      #"BASIC"/"STANDARD"; "DEDICATED" - not supported yet

    # For sku="DEDICATED" only:
    networking       = "VPCPeering" # "Internet", "PrivateLink", "VPCPeering"
    cku = 1
    ## For sku="DEDICATED" && networking="VPCPeering"only:
    cidr_for_confluentcloud = "10.12.0.0/16"
  }
}

UPDATE 2020.10.08: seems go-client has it already - we need just port this arguments into tf-provider :)

[rjudin]

There is a related issue about missed cku: cgroschupp/go-client-confluent-cloud#6

[SimonBirds]

+1 for both the ability to configure Dedicated Cluster networking and adding 'cku'.

[grrywlsn]

+1 for this. Trying to set up a dedicated cluster with VPC peering but can't do it with the provider yet.

[Mongey]

I think this is actually supported now 🤔

resource "confluentcloud_kafka_cluster" "test" {
  name             = "provider-test"
  service_provider = "aws"
  region           = "eu-west-1"
  availability     = "LOW"
  environment_id   = confluentcloud_environment.environment.id
  network_egress  = 100
  network_ingress = 100
  storage         = 5000

  deployment = {
    sku = "DEDICATED"
  }
  cku = 1
}

[grrywlsn]

I can provide sku = "DEDICATED" and that works, but the cluster is set to be public and that is an immutable property once the cluster is created. I want to provision with VPC peering only, which (at least from what I have tried) doesn't seem to be supported yet.

[grrywlsn]

I guess the TF provider isn't yet implementing the ClusterDeploymentNetworkAccess struct? https://github.com/cgroschupp/go-client-confluent-cloud/blob/master/confluentcloud/cluster.go#L36-L41

[Marcus-James-Adams]

Whilst I can't help on the code side of adding the networking options, I can assist on the testing debugging or Azure-based Private Link and peering if it helps

[onobc]

In response to the initial comment of this ticket... the Go client actually does not have support for this during create. It returns the networking info in the response from the create request but does not allow it to be specified on the create request.

A step further, the Confluent CLI does not support creating clusters w/ PrivateLink. Does the API even support this?

[Marcus-James-Adams]

@bono007 please excuse my ignorance as i am an infra guy rather than a dev - but does not this https://github.com/cgroschupp/go-client-confluent-cloud/blob/master/confluentcloud/cluster.go#L39 suggest that it is supported?

[onobc]

@Marcus-James-Adams 👋🏻

So at first glance I thought the same. However, that struct is not on the ClusterCreateConfig (aka the struct that is used for creation of the cluster) but rather in the return payload.

[Marcus-James-Adams]

Not sure if it will help get things moving but I've put a request in against the go client - cgroschupp/go-client-confluent-cloud#14 If people could add their votes to it.

[Marcus-James-Adams]

Once again not being a dev but an infra guy i notice that confluent provide their own client https://github.com/confluentinc/confluent-kafka-go - Would it not make sense to wrap the terraform provider around the actual client, or is there a technical reason for using somebody elses?

[amitkgupta]

@Marcus-James-Adams https://github.com/confluentinc/confluent-kafka-go is a client for Kafka APIs, it enables doing things like producing messages to and consuming messages from a topic in a Kafka cluster. It works against any Kafka cluster, it is not specific in any way to Confluent Cloud.

https://github.com/cgroschupp/go-client-confluent-cloud/ is a client for Confluent Cloud APIs, it enables doing things like creating and deleting Kafka clusters on demand in Confluent Cloud. It is not for reading/writing data to a particular Kafka cluster.

The difference is analogous to using the aws CLI or SDK to create RDS PostgreSQL clusters, but using the psql CLI or SDK to SELECT or INSERT rows of data into a PostgreSQL cluster.

[emmaLP]

Need private link support. Although you can set dedicated, the cluster can only be public dedicated

[emmaroberts-nbs]

Any update on this please?

[Marcus-James-Adams]

Just as its been some time, is there any movement on this?