lightning: add docs for local backend (pingcap#3660)

* lightning: add docs for local backend

* Update tidb-lightning-faq.md

* refine headings

* Apply suggestions from code review

Co-authored-by: Keke Yi <[email protected]>

* Update tidb-lightning/deploy-tidb-lightning.md

* add back mistakenly deleted content

* Apply suggestions from code review

Co-authored-by: Keke Yi <[email protected]>

* change "import modes" to "backend"

* Update tidb-lightning-backends.md

* move "what backend can do"

* Apply suggestions from code review

Co-authored-by: Keke Yi <[email protected]>

Co-authored-by: Keke Yi <[email protected]>
Co-authored-by: Lilian Lee <[email protected]>

Author: 3 people

TOC.md

@@ -187,7 +187,7 @@
       + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md)
       + [Table Filter](/table-filter.md)
       + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md)
-      + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md)
+      + [Backends](/tidb-lightning/tidb-lightning-backends.md)
       + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md)
     + [Monitor](/tidb-lightning/monitor-tidb-lightning.md)
     + [Troubleshoot](/troubleshoot-tidb-lightning.md)

backup-and-restore-using-dumpling-lightning.md

@@ -83,4 +83,4 @@ The steps to manually modify the GC time are as follows:
 
 ## Restore data into TiDB
 
-To restore data into TiDB, use TiDB Lightning to import the exported data. See [TiDB Lightning Tutorial](/tidb-lightning/tidb-lightning-tidb-backend.md).
+To restore data into TiDB, use TiDB Lightning to import the exported data. See [TiDB Lightning Tutorial](/tidb-lightning/tidb-lightning-backends.md).

backup-and-restore-using-mydumper-lightning.md

@@ -93,4 +93,4 @@ Then execute two more commands:
 
 ## Restore data into TiDB
 
-To restore data into TiDB, use TiDB Lightning to import the exported data. See [TiDB Lightning Tutorial](/tidb-lightning/tidb-lightning-tidb-backend.md).
+To restore data into TiDB, use TiDB Lightning to import the exported data. See [TiDB Lightning Tutorial](/tidb-lightning/tidb-lightning-backends.md).

dumpling-overview.md

@@ -123,7 +123,7 @@ The above command exports all the tables in the `employees` database and the `Wo
 Dumpling can also export specific databases with the `-B` option or specific tables with the `-T` option.
 
 > **Note:**
-> 
+>
 > - The `--filter` option and the `-T` option cannot be used at the same time.
 > - The `-T` option can only accept a complete form of inputs like `database-name.table-name`, and inputs with only the table name are not accepted. Example: Dumpling cannot recognize `-T WorkOrder`.
 
@@ -210,7 +210,7 @@ After your operation is completed, set the GC time back (the default value is `1
 update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time';
 ```
 
-Finally, all the exported data can be imported back to TiDB using [Lightning](/tidb-lightning/tidb-lightning-tidb-backend.md).
+Finally, all the exported data can be imported back to TiDB using [Lightning](/tidb-lightning/tidb-lightning-backends.md).
 
 ## Option list of Dumpling
 

ecosystem-tool-user-guide.md

@@ -7,7 +7,7 @@ aliases: ['/docs/dev/ecosystem-tool-user-guide/','/docs/dev/reference/tools/user
 
 This document introduces the functionalities of TiDB ecosystem tools and their relationship.
 
-## Full data export 
+## Full data export
 
 [Dumpling](/dumpling-overview.md) is a tool for the logical full data export from MySQL or TiDB.
 
@@ -22,10 +22,11 @@ The following are the basics of Dumpling:
 
 [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md) (Lightning) is a tool used for the full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Dumpling or CSV data source.
 
-TiDB Lightning supports two modes:
+TiDB Lightning supports three modes:
 
-- `importer`: This mode uses tikv-importer as the backend, which is usually for importing a large amount of data (at the TB level). During the import, the cluster cannot provide services.
-- `tidb`: This mode uses TiDB/MySQL as the backend, which is slower than the `importer` mode but can be performed online. It also supports importing data to MySQL.
+- `local`: TiDB Lightning parses data into ordered key-value pairs and directly imports them into TiKV. This mode is usually for importing a large amount of data (at the TB level) to a new cluster. During the import, the cluster cannot provide services.
+- `importer`: This mode is similar to the `local` mode. To use this mode, you need to deploy an additional component `tikv-importer` to help import key-value pairs. If the target cluster is in v4.0 or later versions, it is recommended to use the `local` mode.
+- `tidb`: This mode uses TiDB/MySQL as the backend, which is slower than the `local` mode and `importer` mode but can be performed online. It also supports importing data to MySQL.
 
 The following are the basics of TiDB Lightning:
 
@@ -37,7 +38,7 @@ The following are the basics of TiDB Lightning:
 
 > **Note:**
 >
-> The Loader tool is no longer maintained. For scenarios related to Loader, it is recommended that you use the `tidb` mode of TiDB Lighting instead.
+> The Loader tool is no longer maintained. For scenarios related to Loader, it is recommended that you use the `tidb` mode of TiDB Lighting instead. For details, see [TiDB Lightning TiDB backends](/tidb-lightning/tidb-lightning-backends.md#migrating-from-loader-to-tidb-lightning-tidb-backend).
 
 ## Backup and restore
 

get-started-with-tidb-lightning.md

@@ -11,17 +11,11 @@ aliases: ['/docs/dev/get-started-with-tidb-lightning/','/docs/dev/how-to/get-sta
 + Import **large amounts** of **new** data **quickly**
 + Back up and restore all the data
 
-The TiDB Lightning tool set consists of two components:
-
-- **`tidb-lightning`** (the "front end") reads the data source and imports the database structure into the TiDB cluster, and also transforms the data into Key-Value (KV) pairs and sends them to `tikv-importer`.
-
-- **`tikv-importer`** (the "back end") combines and sorts the KV pairs and then imports these sorted pairs as a whole into the TiKV cluster.
-
 ![Architecture of TiDB Lightning tool set](/media/tidb-lightning-architecture.png)
 
 ## Prerequisites
 
-This tutorial assumes you use several new and clean CentOS 7 instances. You can use VMware, VirtualBox or other tools to deploy a virtual machine locally or a small cloud virtual machine on a vendor-supplied platform. Because TiDB Lightning consumes a large amount of computer resources, it is recommended that you allocate at least 4 GB memory for running it.
+This tutorial assumes you use several new and clean CentOS 7 instances. You can use VMware, VirtualBox or other tools to deploy a virtual machine locally or a small cloud virtual machine on a vendor-supplied platform. Because TiDB Lightning consumes a large amount of computer resources, it is recommended that you allocate at least 16 GB memory and CPU of 32 cores for running it with the best performance.
 
 > **Warning:**
 >
@@ -51,69 +45,63 @@ After executing this command, the full backup data is exported to the `/data/my_
 
 ### Step 1: Deploy TiDB cluster
 
-Before the data import, you need to deploy a TiDB cluster (later than v2.0.9). In this tutorial, TiDB v3.0.4 is used. For the deployment method, refer to [TiDB Introduction](/overview.md).
+Before the data import, you need to deploy a TiDB cluster (later than v2.0.9). In this tutorial, TiDB v4.0.3 is used. For the deployment method, refer to [TiDB Introduction](/overview.md).
 
 ### Step 2: Download TiDB Lightning installation package
 
 Download the TiDB Lightning installation package from the following link:
 
-- **v3.0.4**: [tidb-toolkit-v3.0.4-linux-amd64.tar.gz](https://download.pingcap.org/tidb-toolkit-v3.0.0-linux-amd64.tar.gz)
+- **v4.0.3**: [tidb-toolkit-v4.0.3-linux-amd64.tar.gz](https://download.pingcap.org/tidb-toolkit-v4.0.3-linux-amd64.tar.gz)
 
 > **Note:**
 >
 > Choose the same version of TiDB Lightning as that of the TiDB cluster.
 
-### Step 3: Start `tikv-importer`
-
-1. Upload `bin/tikv-importer` in the package to the server where TiDB Lightning is deployed.
+### Step 3: Start `tidb-lightning`
 
-2. Configure `tikv-importer.toml`.
+1. Upload `bin/tidb-lightning` and `bin/tidb-lightning-ctl` in the package to the server where TiDB Lightning is deployed.
+2. Upload the [prepared data source](#prepare-full-backup-data) to the server.
+3. Configure `tidb-lightning.toml` as follows:
 
     ```toml
-    # The template of the tikv-importer configuration file
-
-    # Log file
-    log-file = "tikv-importer.log"
-    # Log level: "trace", "debug", "info", "warn", "error" or "off"
-    log-level = "info"
-
-    [server]
-    # The listening address of tikv-importer. tidb-lightning connects to this address for data write.
-    addr = "192.168.20.10:8287"
-
-    [import]
-    # The directory of the engine file.
-    import-dir = "/mnt/ssd/data.import/"
+    [lightning]
+    # logging
+    level = "info"
+    file = "tidb-lightning.log"
+
+    [tikv-importer]
+    # Uses the Local-backend
+    backend = "local"
+    # Sets the directory for temporarily storing the sorted key-value pairs.
+    # The target directory must be empty.
+    "sorted-kv-dir" = "/mnt/ssd/sorted-kv-dir"
+
+    [mydumper]
+    # Mydumper local source data directory
+    data-source-dir = "/data/my_datasource/"
+
+    [tidb]
+    # Information of the target cluster
+    host = "172.16.31.2"
+    port = 4000
+    user = "root"
+    password = "rootroot"
+    # Table schema information is fetched from TiDB via this status-port.
+    status-port = 10080
+    # The PD address of the cluster
+    pd-addr = "172.16.31.3:2379"
     ```
 
-3. Run `tikv-importer`:
-
-    {{< copyable "shell-regular" >}}
-
-    ```sh
-    nohup ./tikv-importer -C tikv-importer.toml > nohup.out &
-    ```
-
-### Step 4: Start `tidb-lightning`
-
-1. Upload `bin/tidb-lightning` and `bin/tidb-lightning-ctl` in the installation package to the server where TiDB Lightning is deployed.
-2. Upload the [prepared data source](#prepare-full-backup-data) to the server.
-3. After configuring the parameters properly, use a `nohup` command to start the `tidb-lightning` process. If you directly run the command in the command-line, the process might exit because of the SIGHUP signal received. Instead, it's preferable to run a bash script that contains the `nohup` command:
+4. After configuring the parameters properly, use a `nohup` command to start the `tidb-lightning` process. If you directly run the command in the command-line, the process might exit because of the SIGHUP signal received. Instead, it's preferable to run a bash script that contains the `nohup` command:
 
     {{< copyable "shell-regular" >}}
 
     ```sh
     #!/bin/bash
-    nohup ./tidb-lightning \
-                --importer 172.16.31.10:8287 \
-                -d /data/my_database/ \
-                --tidb-host 172.16.31.2 \
-                --tidb-user root \
-                --log-file tidb-lightning.log \
-            > nohup.out &
+    nohup ./tidb-lightning -config tidb-lightning.toml > nohup.out &
     ```
 
-### Step 5: Check data integrity
+### Step 4: Check data integrity
 
 After the import is completed, TiDB Lightning exits automatically. If the import is successful, you can find `tidb lightning exit` in the last line of the log file.
 

loader-overview.md

@@ -8,7 +8,7 @@ aliases: ['/docs/dev/loader-overview/','/docs/dev/reference/tools/loader/','/doc
 
 > **Warning:**
 >
-> Loader is no longer maintained. Its features are completely superseded by [TiDB Lightning TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md). It is strongly recommend to use TiDB Lightning instead.
+> Loader is no longer maintained. Its features are completely superseded by [TiDB Lightning TiDB-backend](/tidb-lightning/tidb-lightning-backends.md#tidb-lightning-tidb-backend). It is strongly recommend to use TiDB Lightning instead.
 
 ## What is Loader?
 

media/tidb-lightning-architecture.png

@@ -16,9 +16,9 @@ Before you start the migration, [deploy TiDB Lightning](/tidb-lightning/deploy-t
 
 > **Note:**
 >
-> - If you choose the Importer-backend to import data, you need to deploy `tikv-importer` along with TiDB Lightning. During the import process, the TiDB cluster cannot provide services. This mode imports data quickly, which is suitable for importing a large amount of data (above the TB level).
-> - If you choose the TiDB-backend, deploy TiDB Lightning only. The cluster can provide services normally during the import.
-> - For detailed differences between the two backend mode, see [TiDB Lightning Backend](/tidb-lightning/tidb-lightning-tidb-backend.md).
+> - If you choose the Local-backend to import data, the TiDB cluster cannot provide services during the import process. This mode imports data quickly, which is suitable for importing a large amount of data (above the TB level).
+> - If you choose the TiDB-backend, the cluster can provide services normally during the import, at a slower import speed.
+> - For detailed differences between the two backend modes, see [TiDB Lightning Backends](/tidb-lightning/tidb-lightning-backends.md).
 
 ## Step 2: Configure data source of TiDB Lightning