Deploy TiDB on AWS EKS

This document describes how to deploy a TiDB cluster on AWS EKS with your laptop (Linux or macOS) for development or testing.

Prerequisites

Before deploying a TiDB cluster on AWS EKS, make sure the following requirements are satisfied:

awscli >= 1.16.73, to control AWS resources

You must configure awscli before it can interact with AWS. The fastest way is using the aws configure command:

{{< copyable "shell-regular" >}}
```
aws configure
```
Replace AWS Access Key ID and AWS Secret Access Key with your own keys:
```
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json
```
Note:

The access key must have at least permissions to: create VPC, create EBS, create EC2 and create role.
terraform >= 0.12
kubectl >= 1.12
helm >= 2.11.0 && < 3.0.0 && != 2.16.4
jq

aws-iam-authenticator installed in PATH, to authenticate with AWS

The easiest way to install aws-iam-authenticator is to download the prebuilt binary as shown below:

Download the binary for Linux:

curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.12.7/2019-03-27/bin/linux/amd64/aws-iam-authenticator

Or, download binary for macOS:

curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.12.7/2019-03-27/bin/darwin/amd64/aws-iam-authenticator

Then execute the following commands:

chmod +x ./aws-iam-authenticator && \
sudo mv ./aws-iam-authenticator /usr/local/bin/aws-iam-authenticator

Deploy

This section describes how to deploy EKS, TiDB operator, TiDB cluster and monitor.

Deploy EKS, TiDB Operator, and TiDB cluster node pool

Use the following commands to deploy EKS, TiDB Operator, and the TiDB cluster node pool.

Get the code from Github:

git clone --depth=1 https://github.com/pingcap/tidb-operator && \
cd tidb-operator/deploy/aws

The default setup creates a new VPC and a t2.micro instance as the bastion machine, and an EKS cluster with following Amazon EC2 instances as worker nodes:

3 m5.xlarge instances for PD
3 c5d.4xlarge instances for TiKV
2 c5.4xlarge instances for TiDB
1 c5.2xlarge instance for monitor

You can create or modify terraform.tfvars to set the value of variables and configure the cluster as needed. See the variables that can be set and their descriptions in variables.tf.

The following is an example of how to configure the EKS cluster name, the TiDB cluster name, the TiDB Operator version, and the number of PD, TiKV and TiDB nodes:

default_cluster_pd_count   = 3
default_cluster_tikv_count = 3
default_cluster_tidb_count = 2
default_cluster_name = "tidb"
eks_name = "my-cluster"
operator_version = "v1.1.0-rc.1"

To deploy TiFlash in the cluster, set create_tiflash_node_pool = true in terraform.tfvars. You can also configure the node count and instance type of the TiFlash node pool by modifying cluster_tiflash_count and cluster_tiflash_instance_type. By default, the value of cluster_tiflash_count is 2, and the value of cluster_tiflash_instance_type is i3.4xlarge.
To deploy TiCDC in the cluster, set create_cdc_node_pool = true in terraform.tfvars. You can also configure the node count and instance type of the TiCDC node pool by modifying cluster_cdc_count and cluster_cdc_instance_type. By default, the value of cluster_cdc_count is 3, and the value of cluster_cdc_instance_type is c5.2xlarge.

Note:

Check the operator_version in the variables.tf file for the default TiDB Operator version of the current scripts. If the default version is not your desired one, configure operator_version in terraform.tfvars.

After configuration, execute the terraform command to initialize and deploy the cluster:

terraform init

terraform apply

It might take 10 minutes or more to finish the process. After terraform apply is executed successfully, some useful information is printed to the console.

A successful deployment will give the output like:

Apply complete! Resources: 67 added, 0 changed, 0 destroyed.

Outputs:

bastion_ip = [
  "34.219.204.217",
]
default-cluster_monitor-dns = not_created
default-cluster_tidb-dns = not_created
eks_endpoint = https://9A9A5ABB8303DDD35C0C2835A1801723.yl4.us-west-2.eks.amazonaws.com
eks_version = 1.12
kubeconfig_filename = credentials/kubeconfig_my-cluster
region = us-west-2

You can use the terraform output command to get the output again.

Note:

EKS versions earlier than 1.14 do not support auto enabling cross-zone load balancing via Network Load Balancer (NLB). Therefore, unbalanced pressure distributed among TiDB instances can be expected in default settings. It is strongly recommended that you refer to AWS Documentation to manually enable cross-zone load balancing for a production environment.

Deploy TiDB cluster and monitor

Prepare the TidbCluster and TidbMonitor CR files:

cp manifests/db.yaml.example db.yaml && cp manifests/db-monitor.yaml.example db-monitor.yaml

To complete the CR file configuration, refer to API documentation and Configure a TiDB Cluster Using TidbCluster.

To deploy TiFlash, configure spec.tiflash in db.yaml as follows:

spec:
  ...
  tiflash:
    baseImage: pingcap/tiflash
    maxFailoverCount: 3
    nodeSelector:
      dedicated: CLUSTER_NAME-tiflash
    replicas: 1
    storageClaims:
    - resources:
        requests:
          storage: 100Gi
      storageClassName: local-storage
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: CLUSTER_NAME-tiflash

Modify replicas, storageClaims[].resources.requests.storage, and storageClassName according to your needs.

To deploy TiCDC, configure spec.ticdc in db.yaml as follows:

spec:
  ...
  tiflash:
    baseImage: pingcap/ticdc
    nodeSelector:
      dedicated: CLUSTER_NAME-cdc
    replicas: 3
    tolerations:
    - effect: NoSchedule
      key: dedicated
      operator: Equal
      value: CLUSTER_NAME-cdc

Modify replicas according to your needs.

Note:

Replace all CLUSTER_NAME in db.yaml and db-monitor.yaml files with default_cluster_name configured during EKS deployment.

Make sure that during EKS deployment, the number of PD, TiKV, TiFlash, TiCDC, or TiDB nodes is >= the value of the replicas field of the corresponding component in db.yaml.

Make sure that spec.initializer.version in db-monitor.yaml and spec.version in db.yaml are the same to ensure normal monitor display.

Create Namespace:

{{< copyable "shell-regular" >}}
```
kubectl --kubeconfig credentials/kubeconfig_${eks_name} create namespace ${namespace}
```
Note:

A namespace is a virtual cluster backed by the same physical cluster. You can give it a name that is easy to memorize, such as the same name as default_cluster_name.

Deploy the TiDB cluster:

kubectl --kubeconfig credentials/kubeconfig_${eks_name} create -f db.yaml -n ${namespace} &&
kubectl --kubeconfig credentials/kubeconfig_${eks_name} create -f db-monitor.yaml -n ${namespace}

Enable cross-zone load balancing for the LoadBalancer of the TiDB service

Due to an issue of AWS Network Load Balancer (NLB), the NLB created for the TiDB service cannot automatically enable cross-zone load balancing. You can manually enable it by taking the following steps:

Get the name of the NLB created for the TiDB service:

kubectl --kubeconfig credentials/kubeconfig_${eks_name} get svc ${default_cluster_name}-tidb -n ${namespace}