Skip to content

Commit

Permalink
cherry pick pingcap#4117 to release-3.1
Browse files Browse the repository at this point in the history
Signed-off-by: ti-srebot <[email protected]>
  • Loading branch information
spencerkee authored and ti-srebot committed Nov 2, 2020
1 parent 85602e9 commit b4bc0f1
Show file tree
Hide file tree
Showing 6 changed files with 284 additions and 17 deletions.
46 changes: 40 additions & 6 deletions auto-random.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,34 +21,52 @@ Take the following created table as an example:
{{< copyable "sql" >}}

```sql
<<<<<<< HEAD
create table t (a int primary key auto_increment, b varchar(255))
=======
CREATE TABLE t (a bigint PRIMARY KEY AUTO_INCREMENT, b varchar(255))
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
```

On this `t` table, you execute a large number of `INSERT` statements that do not specify the values of the primary key as below:

{{< copyable "sql" >}}

```sql
insert into t(b) values ('a'), ('b'), ('c')
INSERT INTO t(b) VALUES ('a'), ('b'), ('c')
```

In the above statement, values of the primary key (column `a`) are not specified, so TiDB uses the continuous auto-increment row values as the row IDs, which might cause write hotspot in a single TiKV node and affect the performance. To avoid such performance decrease, you can specify the `AUTO_RANDOM` attribute rather than the `AUTO_INCREMENT` attribute for the column `a` when you create the table. See the follow examples:

{{< copyable "sql" >}}

```sql
<<<<<<< HEAD
create table t (a int primary key auto_random, b varchar(255))
=======
CREATE TABLE t (a bigint PRIMARY KEY AUTO_RANDOM, b varchar(255))
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
```

or

{{< copyable "sql" >}}

```sql
<<<<<<< HEAD
create table t (a int auto_random, b varchar(255), primary key (a))
```

Then execute the `INSERT` statement such as `INSERT INTO t(b) values...`. Now the results will be as follows:
=======
CREATE TABLE t (a bigint AUTO_RANDOM, b varchar(255), PRIMARY KEY (a))
```
Then execute the `INSERT` statement such as `INSERT INTO t(b) VALUES...`. Now the results will be as follows:
+ Implicitly allocating values: If the `INSERT` statement does not specify the values of the integer primary key column (column `a`) or specify the value as `NULL`, TiDB automatically allocates values to this column. These values are not necessarily auto-increment or continuous but are unique, which avoids the hotspot problem caused by continuous row IDs.
+ Explicitly inserting values: If the `INSERT` statement explicitly specifies the values of the integer primary key column, TiDB saves these values, which works similarly to the `AUTO_INCREMENT` attribute. Note that if you do not set `NO_AUTO_VALUE_ON_ZERO` in the `@@sql_mode` system variable, TiDB will automatically allocate values to this column even if you explicitly specify the value of the integer primary key column as `0`.
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
+ If the `INSERT` statement does not specify the values of the integer primary key column (column `a`) or specify the value as `NULL`, TiDB automatically assigns values to this column. These values are not necessarily auto-increment or continuous but are unique, which avoids the hotspot problem caused by continuous row IDs.
+ If the `INSERT` statement explicitly specifies the values of the integer primary key column, TiDB saves these values, which works similarly to the `AUTO_INCREMENT` attribute. Note that if you do not set `NO_AUTO_VALUE_ON_ZERO` in the `@@sql_mode` system variable, TiDB will automatically assign values to this column even if you explicitly specify the value of the integer primary key column as `0`.
Expand All @@ -62,7 +80,11 @@ To use different number of shard bits, append a pair of parentheses to `AUTO_RAN
{{< copyable "sql" >}}
```sql
<<<<<<< HEAD
create table t (a int primary key auto_random(3), b varchar(255))
=======
CREATE TABLE t (a bigint PRIMARY KEY AUTO_RANDOM(3), b varchar(255))
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
```

In the above `CREATE TABLE` statement, `3` shard bits are specified. The range of the number of shard bits is `[1, field_max_bits)`. `field_max_bits` is the length of bits occupied by the primary key column.
Expand All @@ -72,7 +94,7 @@ After creating the table, use the `SHOW WARNINGS` statement to see the maximum n
{{< copyable "sql" >}}

```sql
show warnings
SHOW WARNINGS
```

```
Expand All @@ -85,14 +107,18 @@ show warnings

In addition, for tables with the `AUTO_RANDOM` attribute, the value of the corresponding `TIDB_ROW_ID_SHARDING_INFO` column in the `information_schema.tables` system table is `PK_AUTO_RANDOM_BITS=x`. `x` is the number of shard bits.

<<<<<<< HEAD
You can use `select last_insert_id ()` to see the last implicit ID assigned by TiDB. For example:
=======
Values allocated to the `AUTO_RANDOM` column affect `last_insert_id()`. You can use `SELECT last_insert_id ()` to get the ID that TiDB last implicitly allocates. For example:
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
{{< copyable "sql" >}}

```sql
insert into t (b) values ("b")
select * from t;
select last_insert_id()
INSERT INTO t (b) VALUES ("b")
SELECT * FROM t;
SELECT last_insert_id()
```

You might see the following result:
Expand All @@ -117,18 +143,26 @@ TiDB supports parsing the version comment syntax. See the following example:
{{< copyable "sql" >}}

```sql
<<<<<<< HEAD
create table t (a int primary key /*T![auto_rand] auto_random */)
=======
CREATE TABLE t (a bigint PRIMARY KEY /*T![auto_rand] auto_random */)
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
```

{{< copyable "sql" >}}

```sql
<<<<<<< HEAD
create table t (a int primary key auto_random)
=======
CREATE TABLE t (a bigint PRIMARY KEY AUTO_RANDOM)
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
```

The above two statements have the same meaning.

In the result of `show create table`, the `AUTO_RANDOM` attribute is commented out. This comment includes an attribute identifier (for example, `/*T![auto_rand] auto_random */`). Here `auto_rand` represents the `AUTO_RANDOM` attribute. Only the version of TiDB that implements the feature corresponding to this identifier can parse the SQL statement fragment properly.
In the result of `SHOW CREATE TABLE`, the `AUTO_RANDOM` attribute is commented out. This comment includes an attribute identifier (for example, `/*T![auto_rand] auto_random */`). Here `auto_rand` represents the `AUTO_RANDOM` attribute. Only the version of TiDB that implements the feature corresponding to this identifier can parse the SQL statement fragment properly.

This attribute supports forward compatibility, namely, downgrade compatibility. TiDB of earlier versions that do not implement this feature ignore the `AUTO_RANDOM` attribute of a table (with the above comment) and can also use the table with the attribute.

Expand Down
86 changes: 86 additions & 0 deletions backup-and-restore-using-dumpling-lightning.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
title: Use Dumpling and TiDB Lightning for Data Backup and Restoration
summary: Introduce how to use Dumpling and TiDB Lightning to backup and restore full data of TiDB.
aliases: ['/docs/dev/export-or-backup-using-dumpling/','/tidb/dev/export-or-backup-using-dumpling']
---

# Use Dumpling and TiDB Lightning for Data Backup and Restoration

This document introduces in detail how to use Dumpling and TiDB Lightning to backup and restore full data of TiDB. For incremental backup and replication to downstream, refer to [TiDB Binlog](/tidb-binlog/tidb-binlog-overview.md).

Suppose that the TiDB server information is as follows:

|Server Name|Server Address|Port|User|Password|
|----|-------|----|----|--------|
|TiDB|127.0.0.1|4000|root|*|

Use the following tools for data backup and restoration:

- [Dumpling](/dumpling-overview.md): to export data from TiDB
- [TiDB Lightning](/tidb-lightning/tidb-lightning-overview.md): to import data into TiDB

## Best practices for full backup and restoration using Dumpling/TiDB Lightning

To quickly backup and restore data (especially large amounts of data), refer to the following recommendations:

* Keep the exported data file as small as possible. It is recommended to use the `-F` option of Dumpling to set the file size. If you use TiDB Lightning to restore data, it is recommended that you set the value of `-F` to `256m`.
* If some of the exported tables have many rows, you can enable concurrency in the table by setting the `-r` option.

## Backup data from TiDB

Use the following `dumpling` command to backup data from TiDB.

{{< copyable "shell-regular" >}}

```bash
./bin/dumpling -h 127.0.0.1 -P 4000 -u root -t 32 -F 256m -T test.t1 -T test.t2 -o ./var/test
```

In this command:

- `-T test.t1 -T test.t2` means that only the two tables `test`.`t1` and `test`.`t2` are exported. For more methods to filter exported data, refer to [Filter exported data](/dumpling-overview.md#filter-the-exported-data).
- `-t 32` means that 32 threads are used to export the data.
- `-F 256m` means that a table is partitioned into chunks, and one chunk is 256MB.

Starting from v4.0.0, Dumpling can automatically extends the GC time if it can access the PD address of the TiDB cluster. But for TiDB earlier than v4.0.0, you need to manually modify the GC time. Otherwise, you might bump into the following error:

```log
Could not read data from testSchema.testTable: GC life time is shorter than transaction duration, transaction starts at 2019-08-05 21:10:01.451 +0800 CST, GC safe point is 2019-08-05 21:14:53.801 +0800 CST
```

The steps to manually modify the GC time are as follows:

1. Before executing the `dumpling` command, query the [GC](/garbage-collection-overview.md) value of the TiDB cluster and execute the following statement in the MySQL client to adjust it to a suitable value:

{{< copyable "sql" >}}

```sql
SELECT * FROM mysql.tidb WHERE VARIABLE_NAME = 'tikv_gc_life_time';
```

```sql
+-----------------------+------------------------------------------------------------------------------------------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+-----------------------+------------------------------------------------------------------------------------------------+
| tikv_gc_life_time | 10m0s |
+-----------------------+------------------------------------------------------------------------------------------------+
1 rows in set (0.02 sec)
```

{{< copyable "sql" >}}

```sql
UPDATE mysql.tidb SET VARIABLE_VALUE = '720h' WHERE VARIABLE_NAME = 'tikv_gc_life_time';
```

2. After executing the `dumpling` command, restore the GC value of the TiDB cluster to the initial value in step 1:

{{< copyable "sql" >}}

```sql
UPDATE mysql.tidb SET VARIABLE_VALUE = '10m' WHERE VARIABLE_NAME = 'tikv_gc_life_time';
```

## Restore data into TiDB

To restore data into TiDB, use TiDB Lightning to import the exported data. See [TiDB Lightning Tutorial](/tidb-lightning/tidb-lightning-backends.md).
4 changes: 2 additions & 2 deletions backup-and-restore-using-mydumper-lightning.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,15 +80,15 @@ Then execute two more commands:
{{< copyable "sql" >}}

```sql
update mysql.tidb set VARIABLE_VALUE = '720h' where VARIABLE_NAME = 'tikv_gc_life_time';
UPDATE mysql.tidb SET VARIABLE_VALUE = '720h' WHERE VARIABLE_NAME = 'tikv_gc_life_time';
```

2. After running the `mydumper` command, adjust GC value of the TiDB cluster to its original value in step 1.

{{< copyable "sql" >}}

```sql
update mysql.tidb set VARIABLE_VALUE = '10m' where VARIABLE_NAME = 'tikv_gc_life_time';
UPDATE mysql.tidb SET VARIABLE_VALUE = '10m' WHERE VARIABLE_NAME = 'tikv_gc_life_time';
```

## Restore data into TiDB
Expand Down
6 changes: 5 additions & 1 deletion basic-sql-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,11 @@ To delete a database, use the `DROP DATABASE` statement:
{{< copyable "sql" >}}

```sql
<<<<<<< HEAD
DROP DATABASE samp_db;
=======
USE mysql;
>>>>>>> 8595ca53... Capitalize sql keywords in several files (#4117)
```

## Create, show, and drop a table
Expand Down Expand Up @@ -175,7 +179,7 @@ To show all the indexes in a table, use the `SHOW INDEX` statement:
{{< copyable "sql" >}}

```sql
SHOW INDEX from person;
SHOW INDEX FROM person;
```

### Delete an index
Expand Down
16 changes: 8 additions & 8 deletions blocklist-control-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ You can use the blocklist of optimization rules to disable some of them if some
{{< copyable "sql" >}}

```sql
insert into mysql.opt_rule_blacklist values("join_reorder"), ("topn_push_down");
INSERT INTO mysql.opt_rule_blacklist VALUES("join_reorder"), ("topn_push_down");
```

Executing the following SQL statement can make the above operation take effect immediately. The effective range includes all old connections of the corresponding TiDB server:
Expand All @@ -62,7 +62,7 @@ You can use the blocklist of optimization rules to disable some of them if some
{{< copyable "sql" >}}

```sql
delete from mysql.opt_rule_blacklist where name in ("join_reorder", "topn_push_down");
DELETE FROM mysql.opt_rule_blacklist WHERE name IN ("join_reorder", "topn_push_down");
```

{{< copyable "sql" >}}
Expand Down Expand Up @@ -95,7 +95,7 @@ The schema of `mysql.expr_pushdown_blacklist` is shown as follows:
{{< copyable "sql" >}}

```sql
desc mysql.expr_pushdown_blacklist;
DESC mysql.expr_pushdown_blacklist;
```

```sql
Expand Down Expand Up @@ -153,7 +153,7 @@ To judge whether the blocklist takes effect, observe the results of `EXPLAIN` (S
{{< copyable "sql" >}}

```sql
explain select * from t where a < 2 and a > 2;
EXPLAIN SELECT * FROM t WHERE a < 2 AND a > 2;
```

```sql
Expand All @@ -172,7 +172,7 @@ To judge whether the blocklist takes effect, observe the results of `EXPLAIN` (S
{{< copyable "sql" >}}

```sql
insert into mysql.expr_pushdown_blacklist values('<','tikv',''), ('>','tikv','');
INSERT INTO mysql.expr_pushdown_blacklist VALUES('<','tikv',''), ('>','tikv','');
```

```sql
Expand All @@ -195,7 +195,7 @@ To judge whether the blocklist takes effect, observe the results of `EXPLAIN` (S
{{< copyable "sql" >}}

```sql
explain select * from t where a < 2 and a > 2;
EXPLAIN SELECT * FROM t WHERE a < 2 and a > 2;
```

```sql
Expand All @@ -214,7 +214,7 @@ To judge whether the blocklist takes effect, observe the results of `EXPLAIN` (S
{{< copyable "sql" >}}

```sql
delete from mysql.expr_pushdown_blacklist where name = '>';
DELETE FROM mysql.expr_pushdown_blacklist WHERE name = '>';
```

```sql
Expand All @@ -236,7 +236,7 @@ To judge whether the blocklist takes effect, observe the results of `EXPLAIN` (S
{{< copyable "sql" >}}

```sql
explain select * from t where a < 2 and a > 2;
EXPLAIN SELECT * FROM t WHERE a < 2 AND a > 2;
```

```sql
Expand Down
Loading

0 comments on commit b4bc0f1

Please sign in to comment.