-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
127 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,121 @@ | ||
--- | ||
date: 2019-12-15 | ||
spot: 大冲新城花园 | ||
sort: Computer Science | ||
tags: | ||
- Network | ||
- iptables | ||
- NAT | ||
- TCP | ||
- MySQL | ||
- Unix domain socket | ||
- man page | ||
--- | ||
|
||
# 慎用 iptables:误用规则引发的疑问 | ||
|
||
![BCY0349 Masquerade](./bcy0349.jpg "Permitted under the [Terms and conditions](https://www.dfo-mpo.gc.ca/terms-conditions-avis-eng.htm). ©️ [**Contributors**](https://www.dfo-mpo.gc.ca/species-especes/mammals-mammiferes/humpback-rorqual-a-bosse/photos/index-eng.html) on [*dfo-mpo.gc.ca*](https://www.dfo-mpo.gc.ca/species-especes/mammals-mammiferes/humpback-rorqual-a-bosse/photos/bcy-eng.html).") | ||
|
||
昨天去了一趟广州。在深圳安检排队时微信突然来了一串消息: | ||
|
||
> 有个 Web 服务突然被数据库拒绝访问。 | ||
事态比较紧急,我们组长先做了临时处理,之后通知了我们几个相关的人。由于我前阵子接手了这个项目,所以也就承担调查事故原因的任务。 | ||
|
||
## 背景 | ||
|
||
这里涉及一些业务层面的东西,需要脱敏,所以只提取出涉事技术因素: | ||
|
||
- Web 服务(下文以 `IDLE` 代称):一个重要而不繁忙的内部网站,只有工作时间会有人使用。 | ||
- 数据库(下文以 `DB` 代称):一个重要且繁忙的数据库,`IDLE` 会对 `DB` 进行只读操作。 | ||
- 另一个 Web 服务(下文以 `BUSY` 代称):持续对 `DB` 进行高频读写操作。 | ||
- `IDLE` 和 `BUSY` 位于同一个 Linux 服务器。 | ||
|
||
### 具体故障 | ||
|
||
`IDLE` 被 `DB` 拒绝访问。 | ||
|
||
在这之前,`IDLE` 对 `DB` 的访问是完全正常的。作为当前唯一的维护人员,我上周根本就没有对它做任何变更,出了这个故障实属让我摸不着头脑。 | ||
|
||
### 初步调查及临时措施 | ||
|
||
组长做了初步调查及临时措施,并提出疑问: | ||
|
||
- Web 服务报错:`sqlMessage: "Host 'IP.IP.IP.IP' is not allowed to connect to this MySQL server"` | ||
- 临时措施: | ||
- `GRANT ALL PRIVILEGES ON *.* TO 'USER'@'IP.IP.IP.IP';` | ||
- 对 `DB` 做了此变更之后 `IDLE` 恢复正常 | ||
- 疑问: | ||
> 1. MySQL 拒绝 `IDLE` 访问,为什么昨天没有出现,而今天早晨出现了?周五晚上有人修改了 MySQL 配置吗? | ||
> 2. `IDLE` 服务连接数据库的配置中,`host = "localhost"`,为什么 MySQL 的报错会是 `"IP.IP.IP.IP"`? | ||
### 可能相关的运维变更 | ||
|
||
对于疑问 `1`,我并没有对服务器或服务做任何变更,所以应该是其他人做了相关操作。有位同事补充了周五晚上他对服务器做的变更,出于运维需要,给 iptables 加了如下规则: | ||
|
||
```sh | ||
sudo iptables -t nat -A POSTROUTING -j MASQUERADE | ||
``` | ||
|
||
之后,又使用 `iptables-restore` 将变更前备份的 iptables 规则表重新导入。 | ||
|
||
### 我的疑问 | ||
|
||
得到以上信息之后,我也产生了新的疑问:为什么在同一台服务器上面,且使用同样配置连接 `DB` 的 `BUSY` 服务不受影响? | ||
|
||
## 调查 | ||
|
||
同事对服务器 iptables 做了相关的配置,而刚好出了故障,从直觉上来说这两件事有关联的概率很大。但由于后续用 `iptables-restore` 导入了原来的配置,所以这个因素暂时就先搁置了。 | ||
|
||
对于疑问 `2`,按照正常情况,如果客户端使用 `localhost` 连接 `DB`,那么客户端自身的地址也会是 `localhost` [[1]],现在却变成 `IP.IP.IP.IP`,导致被 `DB` 拒绝连接。 | ||
|
||
会是 `localhost` 指向有问题吗? | ||
|
||
```sh | ||
$ ping localhost | ||
PING localhost (127.0.0.1) 56(84) bytes of data. | ||
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.026 ms | ||
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.024 ms | ||
... | ||
``` | ||
|
||
显然,`localhost` 正常。 | ||
|
||
### MySQL client 的连接方式 | ||
|
||
如果用 `mysql` 的命令行客户端登录呢?也正常: | ||
|
||
```sh | ||
$ mysql -uUSER -p | ||
Enter password: | ||
Welcome to the MySQL monitor. Commands end with ; or \g | ||
... | ||
``` | ||
|
||
之后跟朋友 `@Shady` 吐槽了这个现象,他告诉我在 Linux 上 `mysql` 命令行客户端默认使用 Unix domain socket [[2]] 而不是 TCP 协议。这个信息很重要: | ||
|
||
```sh | ||
$ mysql -uUSER -h 127.0.0.1 -p --protocol=TCP | ||
Enter password: | ||
ERROR 1130 (HY000): Host 'IP.IP.IP.IP' is not allowed to connect to this MySQL server | ||
``` | ||
|
||
可以看到,只要是通过 TCP 协议通过 `localhost` (`127.0.0.1`) 连接 `DB`,就会有问题。 | ||
|
||
>>>>> progress | ||
--- | ||
|
||
:::details 封面图 | ||
|
||
::: | ||
|
||
## References | ||
|
||
1. [What is the source IP address when we ping 127.0.0.1?][1]. *forum.networklessons.com*. | ||
2. [4.3.3 Connecting using Unix Sockets and Windows Named Pipes][2]. *dev.mysql.com*. | ||
3. [Linux man pages][20]. *man7.org*. | ||
|
||
[1]: <https://forum.networklessons.com/t/what-is-the-source-ip-address-when-we-ping-127-0-0-1/3643> | ||
[2]: <https://dev.mysql.com/doc/mysql-shell/8.2/en/mysql-shell-connection-socket.html> | ||
[20]: <https://man7.org/linux/man-pages/index.html> |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.