en: update doc for tiflash (pingcap#272)

en/TOC.md

@@ -16,6 +16,7 @@
   - [Prerequisites](prerequisites.md)
   - [TiDB Operator](deploy-tidb-operator.md)
   - [TiDB in General Kubernetes](deploy-on-general-kubernetes.md)
+  - [Deploy TiFlash](deploy-tiflash.md)
   - [TiDB in AWS EKS](deploy-on-aws-eks.md)
   - [TiDB in GCP GKE](deploy-on-gcp-gke.md)
   - [TiDB in Alibaba Cloud ACK](deploy-on-alibaba-cloud.md)

en/configure-cluster-using-tidbcluster.md

@@ -6,13 +6,13 @@ category: how-to
 
 # Configure a TiDB Cluster Using TidbCluster
 
-This document introduces how to configure the parameters of TiDB/TiKV/PD using TidbCluster.
+This document introduces how to configure the parameters of TiDB/TiKV/PD/TiFlash using TidbCluster.
 
 The current TiDB Operator v1.1 supports all parameters of TiDB v3.1. For parameters of different components, refer to [TiDB documentation](https://pingcap.com/docs/).
 
 ## Configure TiDB parameters
 
-TiDB parameters can be configured by `TidbCluster.Spec.Tidb.Config`.
+TiDB parameters can be configured by `spec.tidb.config` in TidbCluster Custom Resource.
 
 For example:
 
@@ -44,7 +44,7 @@ For all the configurable parameters of TiDB, refer to [TiDB Configuration File](
 
 ## Configure TiKV parameters
 
-TiKV parameters can be configured by `TidbCluster.Spec.Tikv.Config`.
+TiKV parameters can be configured by `spec.tikv.config` in TidbCluster Custom Resource.
 
 For example:
 
@@ -73,7 +73,7 @@ For all the configurable parameters of TiKV, refer to [TiKV Configuration File](
 
 ## Configure PD parameters
 
-PD parameters can be configured by `TidbCluster.Spec.Pd.Config`.
+PD parameters can be configured by `spec.pd.config` in TidbCluster Custom Resource.
 
 For example:
 
@@ -96,3 +96,24 @@ For all the configurable parameters of PD, refer to [PD Configuration File](http
 > **Note:**
 >
 > If you deploy your TiDB cluster using CR, make sure that `Config: {}` is set, no matter you want to modify `config` or not. Otherwise, PD components might not be started successfully. This step is meant to be compatible with `Helm` deployment.
+
+## Configure TiFlash parameters
+
+TiFlash parameters can be configured by `spec.tiflash.config` in TidbCluster Custom Resource.
+
+For example:
+
+```yaml
+apiVersion: pingcap.com/v1alpha1
+kind: TidbCluster
+metadata:
+  name: basic
+spec:
+  ...
+  tiflash:
+    config:
+      config:
+        logger:
+          count: 5
+          level: information
+```

en/deploy-on-general-kubernetes.md

@@ -36,6 +36,44 @@ The modified configuration is not automatically applied to the TiDB cluster by d
 
 It is recommended that you set `spec.configUpdateStrategy` to `RollingUpdate` to enable automatic update of configurations. This way, every time the configuration is updated, all components are rolling updated automatically, and the modified configuration is applied to the cluster.
 
+If you want to enable TiFlash in the cluster, configure `spec.pd.config.replication.enable-placement-rules` to `true` and configure `spec.tiflash` in the `${cluster_name}/tidb-cluster.yaml` file as follows:
+
+```yaml
+  pd:
+    config:
+      ...
+      replication:
+        enable-placement-rules: "true"
+        ...
+  tiflash:
+    baseImage: pingcap/tiflash
+    maxFailoverCount: 3
+    replicas: 1
+    storageClaims:
+    - resources:
+        requests:
+          storage: 100Gi
+      storageClassName: local-storage
+```
+
+TiFlash supports mounting multiple Persistent Volumes (PVs). If you want to configure multiple PVs for TiFlash, configure multiple `resources` in `tiflash.storageClaims`, each `resources` with a separate `storage request` and `storageClassName`. For example:
+
+```yaml
+  tiflash:
+    baseImage: pingcap/tiflash
+    maxFailoverCount: 3
+    replicas: 1
+    storageClaims:
+    - resources:
+        requests:
+          storage: 100Gi
+      storageClassName: local-storage
+    - resources:
+        requests:
+          storage: 100Gi
+      storageClassName: local-storage
+```
+
 To deploy TiDB cluster monitor, refer to the [TidbMonitor example](https://github.com/pingcap/tidb-operator/blob/master/manifests/monitor/tidb-monitor.yaml) and [API documentation](api-references.md) to complete TidbMonitor CR, and save it to the `${cluster_name}/tidb-monitor.yaml` file. Please switch the TidbMonitor example and API documentation to the currently used version of TiDB Operator.
 
 ### Storage class

en/deploy-tiflash.md

@@ -0,0 +1,62 @@
+---
+title: Deploy TiFlash on Kubernetes
+summary: Learn how to deploy TiFlash on Kubernetes.
+category: how-to
+---
+
+# Deploy TiFlash on Kubernetes
+
+This document describes how to deploy TiFlash on Kubernetes.
+
+## Prerequisites
+
+* [Deploy TiDB Operator](deploy-tidb-operator.md).
+
+## Fresh TiFlash deployment
+
+To deploy TiFlash when deploying the TiDB cluster, refer to [Deploy TiDB on General Kubernetes](deploy-on-general-kubernetes.md).
+
+## Add TiFlash component to an existing TiDB cluster
+
+Edit TidbCluster Custom Resource:
+
+{{< copyable "shell-regular" >}}
+
+``` shell
+kubectl eidt tc ${cluster_name} -n ${namespace}
+```
+
+Add the TiFlash configuration as follows:
+
+```yaml
+spec:
+  tiflash:
+    baseImage: pingcap/tiflash
+    maxFailoverCount: 3
+    replicas: 1
+    storageClaims:
+    - resources:
+        requests:
+          storage: 100Gi
+      storageClassName: local-storage
+```
+
+TiFlash supports mounting multiple Persistent Volumes (PVs). If you want to configure multiple PVs for TiFlash, configure multiple `resources` in `tiflash.storageClaims`, each `resources` with a separate `storage request` and `storageClassName`. For example:
+
+```yaml
+  tiflash:
+    baseImage: pingcap/tiflash
+    maxFailoverCount: 3
+    replicas: 1
+    storageClaims:
+    - resources:
+        requests:
+          storage: 100Gi
+      storageClassName: local-storage
+    - resources:
+        requests:
+          storage: 100Gi
+      storageClassName: local-storage
+```
+
+To [add TiFlash component to an existing TiDB cluster](https://pingcap.com/docs/stable/tiflash/deploy-tiflash/#add-tiflash-component-to-an-existing-tidb-cluster), `replication.enable-placement-rules` should be set to `true` in PD. After you add the TiFlash configuration in TidbCluster by taking the above steps, TiDB Operator will automatically configure `replication.enable-placement-rules: "true"` in PD.

en/scale-a-tidb-cluster.md

@@ -14,6 +14,12 @@ Horizontally scaling TiDB means that you scale TiDB out or in by adding or remov
 
 Currently, the TiDB cluster supports management by Helm or by TidbCluster Custom Resource (CR). You can choose the scaling method based on the management method of your TiDB cluster.
 
+### Horizontal scaling operations (CR)
+
+Modify `spec.pd.replicas`, `spec.tidb.replicas`, and `spec.tikv.replicas` in the `TidbCluster` object of the cluster to a desired value using kubectl.
+
+If TiFlash is deployed in the cluster, you can scale in and out TiFlash by modifying `spec.tiflash.replicas`.
+
 ### Horizontal scaling operations (Helm)
 
 To perform a horizontal scaling operation, take the following steps:
@@ -28,10 +34,6 @@ To perform a horizontal scaling operation, take the following steps:
     helm upgrade ${release_name} pingcap/tidb-cluster -f values.yaml --version=${version}
     ```
 
-### Horizontal scaling operations (CR)
-
-Modify `spec.pd.replicas`, `spec.tidb.replicas`, and `spec.tikv.replicas` in the `TidbCluster` object that corresponds to the cluster to the desired values using kubectl.
-
 ### View the scaling status
 
 To view the scaling status of the cluster, run the following command:
@@ -46,17 +48,24 @@ When the number of Pods for all components reaches the preset value and all comp
 
 > **Note:**
 >
-> - The PD and TiKV components do not trigger scaling in and out operations during the rolling update.
-> - When the TiKV component scales in, it calls the PD interface to mark the corresponding TiKV instance as offline, and then migrates the data on it to other TiKV nodes. During the data migration, the TiKV Pod is still in the `Running` state, and the corresponding Pod is deleted only after the data migration is completed. The time consumed by scaling in depends on the amount of data on the TiKV instance to be scaled in. You can check whether TiKV is in the `Offline` state by running `kubectl get tidbcluster -n ${namespace} ${release_name} -o json | jq '.status.tikv.stores'`.
-> - When the PD and TiKV components scale in, the PVC of the deleted node is retained during the scaling in process. Because the PV's reclaim policy is changed to `Retain`, the data can still be retrieved even if the PVC is deleted.
+> - The PD, TiKV and TiFlash components do not trigger scaling in and out operations during the rolling update.
+> - When the TiKV component scales in, TiDB Operator calls the PD interface to mark the corresponding TiKV instance as offline, and then migrates the data on it to other TiKV nodes. During the data migration, the TiKV Pod is still in the `Running` state, and the corresponding Pod is deleted only after the data migration is completed. The time consumed by scaling in depends on the amount of data on the TiKV instance to be scaled in. You can check whether TiKV is in the `Offline` state by running `kubectl get tidbcluster -n ${namespace} ${release_name} -o json | jq '.status.tikv.stores'`.
 > - The TiKV component does not support scale out while a scale-in operation is in progress. Forcing a scale-out operation might cause anomalies in the cluster. If an anomaly already happens, refer to [TiKV Store is in Tombstone status abnormally](troubleshoot.md#tikv-store-is-in-tombstone-status-abnormally) to fix it.
+> - The TiFlash component has the same scale-in logic as TiKV.
+> - When the PD, TiKV, and TiFlash components scale in, the PVC of the deleted node is retained during the scaling in process. Because the PV's reclaim policy is changed to `Retain`, the data can still be retrieved even if the PVC is deleted.
 
 ## Vertical scaling
 
 Vertically scaling TiDB means that you scale TiDB up or down by increasing or decreasing the limit of resources on the node. Vertically scaling is essentially the rolling update of the nodes.
 
 Currently, the TiDB cluster supports management by Helm or by TidbCluster Custom Resource (CR). You can choose the scaling method based on the management method of your TiDB cluster.
 
+### Vertical scaling operations (CR)
+
+Modify `spec.pd.resources`, `spec.tikv.resources`, and `spec.tidb.resources` in the `TidbCluster` object that corresponds to the cluster to the desired values using kubectl.
+
+If TiFlash is deployed in the cluster, you can scale up and down TiFlash by modifying `spec.tiflash.resources`.
+
 ### Vertical scaling operations (Helm)
 
 To perform a vertical scaling operation:
@@ -71,10 +80,6 @@ To perform a vertical scaling operation:
     helm upgrade ${release_name} pingcap/tidb-cluster -f values.yaml --version=${version}
     ```
 
-### Vertical scaling operations (CR)
-
-Modify `spec.pd.resources`, `spec.tikv.resources`, and `spec.tikv.resources` in the `TidbCluster` object that corresponds to the cluster to the desired values using kubectl.
-
 ### View the upgrade progress
 
 To view the upgrade progress of the cluster, run the following command:
@@ -89,5 +94,5 @@ When all Pods are rebuilt and in the `Running` state, the vertical scaling is co
 
 > **Note:**
 >
-> - If the resource's `requests` field is modified during the vertical scaling process, because PD and TiKV use `Local PV`, they need to be scheduled back to the original node after the upgrade. At this time, if the original node does not have enough resources, the Pod ends up staying in the `Pending` status and thus impacts the service.
+> - If the resource's `requests` field is modified during the vertical scaling process, and if PD, TiKV, and TiFlash use `Local PV`, they will be scheduled back to the original node after the upgrade. At this time, if the original node does not have enough resources, the Pod ends up staying in the `Pending` status and thus impacts the service.
 > - TiDB is a horizontally scalable database, so it is recommended to take advantage of it simply by adding more nodes rather than upgrading hardware resources like you do with a traditional database.

en/use-auto-failover.md

@@ -32,9 +32,11 @@ controllerManager:
  tikvFailoverPeriod: 5m
  # tidb failover period default(5m)
  tidbFailoverPeriod: 5m
+ # tiflash failover period default(5m)
+ tiflashFailoverPeriod: 5m
 ```
 
-By default, `pdFailoverPeriod`, `tikvFailoverPeriod` and `tidbFailoverPeriod` are set to be 5 minutes, which is the waiting timeout after an instance failure is identified. After this time, TiDB Operator begins the automatic failover process.
+By default, `pdFailoverPeriod`, `tikvFailoverPeriod`, `tiflashFailoverPeriod` and `tidbFailoverPeriod` are set to be 5 minutes, which is the waiting timeout after an instance failure is identified. After this time, TiDB Operator begins the automatic failover process.
 
 ## Automatic failover policies
 
@@ -84,3 +86,38 @@ status
 ### Failover with TiDB
 
 The TiDB automatic failover policy works the same way as `Deployment` does in Kubernetes. Assume that there are 3 nodes in a TiDB cluster. If a TiDB node is down for over 5 minutes (configurable by modifying `tidbFailoverPeriod`), TiDB Operator creates a new TiDB node. At this time, there are 4 nodes in the cluster. When the failed TiDB node gets back online, TiDB Operator deletes the newly created node and the number of nodes gets back to 3.
+
+### Failover with TiFlash
+
+When a TiFlash Pod fails, its store status turns to `Disconnected`. After 30 minutes (configurable by modifying `max-store-down-time = "30m"` in the `[schedule]` section of the `pd.config` file), the store status turns to `Down`. After waiting for 5 minutes (configurable by modifying `tiflashFailoverPeriod`), TiDB Operator creates a new TiFlash Pod if this TiFlash Pod is still down. If the failed TiFlash Pod gets back online, TiDB Operator does not automatically delete the newly created Pod, and you need to manually drop it and restore the original number of Pods. To do this, you can delete the TiFlash Pod from the `status.tiflash.failureStores` field of the `TidbCluster` object.
+
+For example, assume two TiFlash Pods are in abnormal state:
+
+{{< copyable "shell-regular" >}}
+
+```shell
+kubectl edit tc -n ${namespace} ${cluster_name}
+```
+
+```
+status
+  tiflash:
+    failureStores:
+      "1":
+        podName: cluster1-tiflash-0
+        storeID: "1"
+      "2":
+        podName: cluster1-tiflash-1
+        storeID: "2"
+```
+
+After the `cluster1-tiflash-0` Pod recovers, delete it manually:
+
+```
+status
+  tiflash:
+    failureStores:
+      "2":
+        podName: cluster1-tiflash-1
+        storeID: "2"
+```