07.srt

1
00:00:00,066 --> 00:00:04,199
字幕生成：qiaokai 字幕校对：mkwei

2
00:00:05,466 --> 00:00:07,199
嗨大家好我是ZOMI

3
00:00:07,200 --> 00:00:09,700
今天呢来到AI编译器里面的

4
00:00:09,700 --> 00:00:10,933
前端优化的

5
00:00:10,933 --> 00:00:11,899
常量折叠

6
00:00:11,900 --> 00:00:15,000
那常量折叠叫做Constant Fold

7
00:00:15,600 --> 00:00:18,166
在常量折叠里面呢主要是有两个概念

8
00:00:18,166 --> 00:00:19,333
今天的内容不会很多

9
00:00:19,333 --> 00:00:20,466
第一个呢就是

10
00:00:20,566 --> 00:00:22,699
传统编译器里面的常量折叠这个概念

11
00:00:22,866 --> 00:00:24,399
接着呢去看看

12
00:00:24,400 --> 00:00:25,800
AI编译器里面的常量折叠

13
00:00:25,800 --> 00:00:27,333
具体是怎么实现的

14
00:00:28,933 --> 00:00:30,599
现在呢来了解一下

15
00:00:30,600 --> 00:00:33,466
传统编译器里面一个比较通用的概念

16
00:00:33,466 --> 00:00:34,566
就是常量折叠

17
00:00:34,566 --> 00:00:35,733
Constant folding

18
00:00:35,966 --> 00:00:38,099
常量折叠的这个概念是比较简单

19
00:00:38,100 --> 00:00:40,733
通过对编译时常量或者常量表达式呢

20
00:00:40,766 --> 00:00:42,066
进行计算

21
00:00:42,133 --> 00:00:43,933
来简化整体的代码

22
00:00:43,933 --> 00:00:46,466
那简单的去看一下第三行的代码

23
00:00:46,466 --> 00:00:50,399
i=320*200*32

24
00:00:50,533 --> 00:00:52,133
在传统的编译器里面呢

25
00:00:52,133 --> 00:00:53,766
把它变成一个IR

26
00:00:54,100 --> 00:00:55,933
首先呢我会产生一个变量的a

27
00:00:55,933 --> 00:00:57,999
那这个a呢去加载我的320

28
00:00:58,000 --> 00:01:00,266
另外一个变量b去加载200

29
00:01:00,266 --> 00:01:04,199
接着呢这个x呢就是我的计算a乘以b

30
00:01:04,500 --> 00:01:07,900
然后我再有一个变量去加载32

31
00:01:07,900 --> 00:01:11,366
接着呢有一个变量y等于x乘以c

32
00:01:11,366 --> 00:01:13,199
也就是x乘以c

33
00:01:13,366 --> 00:01:14,933
完成了整一个操作

34
00:01:14,933 --> 00:01:17,999
那这里面呢有三次的内存读取

35
00:01:18,000 --> 00:01:21,166
首先a b c三次内存读取的load

36
00:01:21,166 --> 00:01:23,066
然后呢有两个指令的计算

37
00:01:23,066 --> 00:01:25,499
就是x和y两个都是相乘

38
00:01:25,500 --> 00:01:26,700
所以呢总结一下

39
00:01:26,700 --> 00:01:28,133
有三次的内存读取

40
00:01:28,133 --> 00:01:29,466
两次的指令相乘

41
00:01:29,866 --> 00:01:30,766
实际上呢

42
00:01:30,900 --> 00:01:33,966
第三行这个代码320*200*32呢

43
00:01:34,000 --> 00:01:36,600
实际上呢可以直接从内存里面呢

44
00:01:36,700 --> 00:01:38,400
直接把这个数加载进来

45
00:01:38,400 --> 00:01:41,733
204800就是我在编译的时候呢

46
00:01:41,800 --> 00:01:45,466
直接把320*200*32这个数呢

47
00:01:45,566 --> 00:01:47,299
提前的算出来

48
00:01:47,533 --> 00:01:48,899
这个算出来的工作呢

49
00:01:48,900 --> 00:01:50,133
就交给编译器

50
00:01:50,133 --> 00:01:52,799
而不用到真正执行的时候再去算

51
00:01:54,533 --> 00:01:56,166
现在来看看下一个概念

52
00:01:56,166 --> 00:01:58,199
就是常量的传播

53
00:01:58,200 --> 00:02:00,466
constant propagation常量传播

54
00:02:00,466 --> 00:02:01,966
这也是传统编译器

55
00:02:01,966 --> 00:02:03,899
优化里面的其中一个手段

56
00:02:03,900 --> 00:02:05,366
可以在一段代码里面呢

57
00:02:05,366 --> 00:02:06,799
将表达式的常量

58
00:02:06,900 --> 00:02:08,966
替换成为相关表达式

59
00:02:08,966 --> 00:02:10,199
然后再使用常量折叠

60
00:02:10,200 --> 00:02:12,000
来进行一个代码的简化

61
00:02:12,000 --> 00:02:13,866
那看看两个比较

62
00:02:14,300 --> 00:02:15,666
典型的一个概念

63
00:02:15,666 --> 00:02:17,899
首先呢有一个函数foo

64
00:02:17,966 --> 00:02:19,533
那现在呢定义一个

65
00:02:19,800 --> 00:02:23,600
变量x=14 y=7-x/2

66
00:02:23,600 --> 00:02:26,200
其实呢x呢可以直接带进来

67
00:02:26,200 --> 00:02:30,133
然后return y*(28/x + 2)

68
00:02:30,133 --> 00:02:32,333
那这个时候呢我x等于14

69
00:02:32,333 --> 00:02:33,866
可以直接带进来

70
00:02:33,866 --> 00:02:36,533
那带进来之后呢y已经知道了

71
00:02:36,566 --> 00:02:39,499
我把长量x呢进行往下传播

72
00:02:39,533 --> 00:02:40,333
那最后呢

73
00:02:40,333 --> 00:02:43,066
会使用一个常量折叠这个技术

74
00:02:43,066 --> 00:02:44,466
来简化代码

75
00:02:44,466 --> 00:02:46,166
那常量折叠体现在哪了

76
00:02:46,166 --> 00:02:48,066
就是我最后x已经知道了

77
00:02:48,100 --> 00:02:48,600
我y

78
00:02:48,600 --> 00:02:51,000
其实可以通过编译器来提前算出来

79
00:02:51,000 --> 00:02:52,400
算出来这个y之后呢

80
00:02:52,400 --> 00:02:53,666
我这个y已经有了

81
00:02:53,700 --> 00:02:55,266
那这个y已经求得

82
00:02:55,266 --> 00:02:55,966
到我return的时候

83
00:02:55,966 --> 00:02:59,499
再通过编译器把这个数提前预算出来

84
00:02:59,500 --> 00:03:01,133
那函数foo呢

85
00:03:01,266 --> 00:03:02,666
就已经得到它的值了

86
00:03:02,666 --> 00:03:05,299
就不需要通过硬件再去计算

87
00:03:05,300 --> 00:03:07,166
直接在编译的过程当中

88
00:03:07,166 --> 00:03:09,399
输出foo的一个返回值

89
00:03:10,866 --> 00:03:13,066
这种方式呢就是先通过

90
00:03:13,066 --> 00:03:13,766
常量传播

91
00:03:13,766 --> 00:03:16,799
再使用常量折叠来去简化代码

92
00:03:17,166 --> 00:03:18,999
是不是觉得编译器很有意思

93
00:03:19,000 --> 00:03:20,800
不用去真正的去执行

94
00:03:20,800 --> 00:03:23,100
在编译阶段已经把这些数呢

95
00:03:23,100 --> 00:03:24,666
提前的算好了

96
00:03:28,000 --> 00:03:29,366
现在看一个新的概念

97
00:03:29,366 --> 00:03:31,766
就是AI编译器和常量折叠

98
00:03:31,766 --> 00:03:34,766
常量折叠怎么泛化到AI编译器

99
00:03:34,766 --> 00:03:37,599
怎么使用图层IR进行表达呢

100
00:03:38,400 --> 00:03:39,066
看一下

101
00:03:39,066 --> 00:03:40,966
这里面有个非常有意思概念

102
00:03:40,966 --> 00:03:42,166
就是常量折叠呢

103
00:03:42,166 --> 00:03:44,466
虽然是用在传统的编译器的

104
00:03:44,466 --> 00:03:46,099
但是呢这里面会将

105
00:03:46,100 --> 00:03:47,900
计算图可以预先确定的

106
00:03:47,900 --> 00:03:49,300
输出值的节点呢

107
00:03:49,300 --> 00:03:50,866
替换成为常量

108
00:03:50,866 --> 00:03:52,566
就是把计算图某些

109
00:03:52,566 --> 00:03:53,999
点变成一个常量

110
00:03:54,066 --> 00:03:57,299
然后呢对计算图进行一些结构的优化

111
00:03:57,600 --> 00:03:59,666
这个呢就是实际的过程

112
00:03:59,666 --> 00:04:01,766
那往下看一些具体的例子

113
00:04:01,800 --> 00:04:03,366
那假设现在以AddN

114
00:04:03,400 --> 00:04:04,900
作为一个具体的

115
00:04:05,000 --> 00:04:07,800
例子对于两个形状大小为NCHW

116
00:04:08,066 --> 00:04:09,899
四维的张量Tensor

117
00:04:09,966 --> 00:04:12,766
那这个张量呢它是一个常量就是NCHW

118
00:04:12,800 --> 00:04:14,733
我所有数我都是知道的

119
00:04:14,866 --> 00:04:16,199
Add的结果也是一定的

120
00:04:16,200 --> 00:04:17,133
那这个结果呢

121
00:04:17,133 --> 00:04:18,666
就可以通过AI编译器

122
00:04:18,666 --> 00:04:20,166
提前的算出来

123
00:04:21,500 --> 00:04:22,966
那提前算出来有什么好处呢

124
00:04:22,966 --> 00:04:24,699
就是不需要给Add节点呢

125
00:04:24,700 --> 00:04:26,733
额外的分配存储资源呢

126
00:04:26,766 --> 00:04:28,166
在AI编译器里面呢

127
00:04:28,166 --> 00:04:30,733
直接基于计算图直接算出来

128
00:04:30,733 --> 00:04:33,533
不需要反复的去计算Add这个操作

129
00:04:33,533 --> 00:04:35,666
可以进行直接的内存访问

130
00:04:35,700 --> 00:04:37,133
这个所谓的好处呢

131
00:04:37,133 --> 00:04:38,699
其实跟传统编译器是一样的

132
00:04:38,700 --> 00:04:40,200
看看一个具体的例子

133
00:04:40,333 --> 00:04:42,366
那左边的这个图呢是计算图

134
00:04:42,366 --> 00:04:43,933
其中的一部分

135
00:04:43,933 --> 00:04:46,066
AddN这个算子呢

136
00:04:46,066 --> 00:04:48,699
它的输入呢是一个常量的张量

137
00:04:48,933 --> 00:04:50,733
b呢也是一个常量的张量

138
00:04:50,766 --> 00:04:53,733
那这个时候呢我可以先把Tensor a

139
00:04:53,800 --> 00:04:57,000
跟Tensor b在编译器里面进行相加的

140
00:04:57,066 --> 00:04:58,999
得到我的Tensor x的时候

141
00:04:59,066 --> 00:05:02,533
这个Tensor x呢再给卷积进行计算

142
00:05:02,566 --> 00:05:03,366
那这种方式呢

143
00:05:03,366 --> 00:05:05,733
就可以简化了计算图

144
00:05:05,733 --> 00:05:07,566
计算图里面的内容少了

145
00:05:07,566 --> 00:05:10,099
我就可以提前编译出来提前算好

146
00:05:10,200 --> 00:05:11,900
然后真正在硬件执行的时候呢

147
00:05:11,900 --> 00:05:14,333
就不需要再次调度AddN

148
00:05:14,466 --> 00:05:15,866
也不需要从内存里面呢

149
00:05:15,866 --> 00:05:18,499
再去加载a和b这个张量

150
00:05:18,500 --> 00:05:20,000
直接加载x这个张量

151
00:05:20,000 --> 00:05:22,166
然后进行去计算就完了

152
00:05:23,133 --> 00:05:25,733
下面呢再看一个比较经典的例子

153
00:05:25,733 --> 00:05:27,599
就是BN折叠

154
00:05:28,533 --> 00:05:30,499
这个BN折叠呢也是ZOMI

155
00:05:30,533 --> 00:05:31,566
最开始的时候

156
00:05:31,566 --> 00:05:33,733
写AI编译器里面实现的一个功能

157
00:05:34,300 --> 00:05:35,900
看一下Batch Normalization

158
00:05:35,900 --> 00:05:38,166
的这个算子的作用

159
00:05:38,366 --> 00:05:39,399
它最主要的作用呢

160
00:05:39,400 --> 00:05:41,333
是对各层的输入呢

161
00:05:41,333 --> 00:05:42,366
进行一个归一化

162
00:05:42,366 --> 00:05:44,733
然后再给下一层训练学习的

163
00:05:44,733 --> 00:05:46,799
使得训练学习的目的呢

164
00:05:46,800 --> 00:05:48,600
是希望能够在训练过程当中呢

165
00:05:48,600 --> 00:05:50,466
数据更加稳定更加收敛

166
00:05:50,600 --> 00:05:51,900
那在实践的过程当中呢

167
00:05:51,900 --> 00:05:53,600
它是一个额外的一层

168
00:05:53,600 --> 00:05:56,666
通常呢会放在卷积、MatMul、Transformer

169
00:05:56,866 --> 00:05:57,899
等计算之后

170
00:05:57,900 --> 00:06:00,200
在一些非线性激活之前

171
00:06:00,200 --> 00:06:01,966
就是卷积 BN ReLU

172
00:06:01,966 --> 00:06:02,799
那可以看到

173
00:06:02,800 --> 00:06:05,000
卷积跟非线性层之间呢

174
00:06:05,000 --> 00:06:06,266
就会插一个BN

175
00:06:06,266 --> 00:06:09,133
Batch Normalization主要分为两个步骤

176
00:06:09,700 --> 00:06:12,100
第一步呢是先减去平均值

177
00:06:12,100 --> 00:06:14,700
然后再除以标准差

178
00:06:15,000 --> 00:06:15,600
接着呢

179
00:06:15,600 --> 00:06:18,933
去计算Gama缩放和Bata的偏移

180
00:06:19,666 --> 00:06:21,799
具体的公式呢就是下面这两个

181
00:06:21,800 --> 00:06:24,800
首先呢我计算我的均值计算我的方差

182
00:06:24,966 --> 00:06:27,399
对输入的数据呢减去均值

183
00:06:27,400 --> 00:06:29,333
然后除以标准差

184
00:06:29,533 --> 00:06:31,466
最后通过Gama和Bata

185
00:06:31,533 --> 00:06:33,566
进行一个缩放和偏移

186
00:06:33,866 --> 00:06:36,366
得到BN的计算结果

187
00:06:37,966 --> 00:06:38,966
现在看看啊

188
00:06:38,966 --> 00:06:40,599
一旦训练结束之后呢

189
00:06:40,600 --> 00:06:42,333
每一个BN层都有Beta

190
00:06:42,333 --> 00:06:44,099
Gama还有均值和方差

191
00:06:44,166 --> 00:06:46,066
那其实在训练的过程当中呢

192
00:06:46,066 --> 00:06:48,333
这两个呢是可学习的参数

193
00:06:48,333 --> 00:06:50,566
这两个也是可学习的参数

194
00:06:50,600 --> 00:06:52,300
但是呢在推理场景

195
00:06:52,400 --> 00:06:54,266
这些参数呢都已经固定了

196
00:06:54,266 --> 00:06:55,666
就他不需要学习了

197
00:06:55,733 --> 00:06:57,266
这个时候呢就可以

198
00:06:57,266 --> 00:06:57,799
这个时候呢

199
00:06:57,800 --> 00:06:59,900
就可以对上面的公式进行一个简

200
00:06:59,900 --> 00:07:00,933
单的线性转换

201
00:07:01,000 --> 00:07:02,466
那现在看一下

202
00:07:02,466 --> 00:07:04,933
由于BN经常放在卷积后面

203
00:07:04,933 --> 00:07:07,266
而卷积呢是一个线性的变换

204
00:07:07,266 --> 00:07:09,266
这个时候呢意味着两个操作呢

205
00:07:09,266 --> 00:07:11,466
可以合成一个简单的操作

206
00:07:12,000 --> 00:07:13,800
这个呢叫做常量折叠

207
00:07:13,800 --> 00:07:15,933
需要跟算子融合呢区分一下

208
00:07:15,933 --> 00:07:17,566
因为这里面最主要的作用呢

209
00:07:17,566 --> 00:07:19,199
就是把一些常量呢

210
00:07:19,200 --> 00:07:21,666
在AI编译器里面提前算出来

211
00:07:22,866 --> 00:07:23,899
在推理场景呢

212
00:07:23,900 --> 00:07:26,333
这个就是简单的卷积的操作

213
00:07:26,400 --> 00:07:27,866
由于这里面的超参呢

214
00:07:27,866 --> 00:07:29,533
其实在训练的过程当中

215
00:07:29,533 --> 00:07:30,399
已经得到了

216
00:07:30,400 --> 00:07:31,100
所以呢

217
00:07:31,100 --> 00:07:32,933
会对它进行一个常量的折叠

218
00:07:32,933 --> 00:07:35,766
那这个常量折叠又叫做BN折叠

219
00:07:35,766 --> 00:07:38,666
w_flold可以通过提前预算出来

220
00:07:38,666 --> 00:07:41,399
b_fold也可以通过提前预算出来

221
00:07:41,666 --> 00:07:43,799
AI编译器呢在BN的阶段呢

222
00:07:43,800 --> 00:07:46,366
就把这些超参跟权重参数

223
00:07:46,366 --> 00:07:47,899
算成w_fold

224
00:07:48,000 --> 00:07:49,366
把b_fold的呢

225
00:07:49,366 --> 00:07:51,933
也是在AI编译器里面提前算好

226
00:07:52,000 --> 00:07:54,466
然后呢在真正推理的时候呢

227
00:07:54,466 --> 00:07:56,466
就没有了BN这一层了

228
00:07:56,766 --> 00:07:59,099
直接把BN层算出来的超参呢

229
00:07:59,100 --> 00:08:00,800
放在权重里面

230
00:08:00,900 --> 00:08:03,500
最后呢只执行这一条卷积的操作

231
00:08:03,500 --> 00:08:07,266
就完成了整个BN加卷积的计算了

232
00:08:07,733 --> 00:08:09,999
所以理解一下就是w_fold

233
00:08:10,066 --> 00:08:10,699
b_fold呢

234
00:08:10,700 --> 00:08:13,366
这个都是在AI编译器里面去实现的

235
00:08:15,733 --> 00:08:16,766
下面这个表呢

236
00:08:16,766 --> 00:08:19,566
就是BN折叠之后的一个性能的对比

237
00:08:19,566 --> 00:08:21,499
可以看到以GPU为例子呢

238
00:08:21,533 --> 00:08:24,566
在Resnet50就是BN折叠之前呢

239
00:08:24,666 --> 00:08:28,199
它运算速率呢执行一个Resnet50是11.03

240
00:08:28,266 --> 00:08:29,533
但是BN折叠之后呢

241
00:08:29,533 --> 00:08:32,599
它整体的运算速率呢是7.3毫秒

242
00:08:32,600 --> 00:08:35,066
可以看到性能提升了51%

243
00:08:35,066 --> 00:08:37,266
还是有一个非常高的性能的提升的

244
00:08:37,266 --> 00:08:39,366
而编译器最重要的作用呢

245
00:08:39,366 --> 00:08:41,766
就是帮助提前算好一些工作

246
00:08:41,800 --> 00:08:44,466
免得在真正执行的时候再去算

247
00:08:44,466 --> 00:08:46,799
为了提升运行效率

248
00:08:49,333 --> 00:08:51,766
最后看一下在AI编译器里面呢

249
00:08:51,766 --> 00:08:53,966
怎么去实现常量折叠的

250
00:08:54,066 --> 00:08:56,699
刚才呢其实只是举了两个例子

251
00:08:56,700 --> 00:08:57,400
一个是Add

252
00:08:57,400 --> 00:08:58,866
一个是BN折叠

253
00:08:58,900 --> 00:09:00,800
实际上AI编译器呢

254
00:09:00,800 --> 00:09:03,700
会通过一些人工提前预设的规则呢

255
00:09:03,700 --> 00:09:05,166
去做一些折叠的

256
00:09:05,166 --> 00:09:07,533
那看一下常量折叠有几个分类

257
00:09:07,533 --> 00:09:07,999
第一个呢

258
00:09:08,000 --> 00:09:10,333
就是传统编译器里面的常量折叠

259
00:09:10,400 --> 00:09:13,600
找到输入节点均为常量的一些节点

260
00:09:13,600 --> 00:09:16,700
然后呢去提前算出来例如add

261
00:09:16,900 --> 00:09:19,666
第二种呢就是常量折叠与数据的类型

262
00:09:19,666 --> 00:09:20,666
shape有关系

263
00:09:20,766 --> 00:09:22,799
通过计算图已有的信息呢

264
00:09:22,800 --> 00:09:24,700
去推断出这个shape之后呢

265
00:09:24,700 --> 00:09:27,566
然后再来替换掉原来的节点

266
00:09:27,900 --> 00:09:29,733
这种最典型的例子呢

267
00:09:29,733 --> 00:09:33,166
有shape、transformer、size这种是一些节点

268
00:09:33,166 --> 00:09:35,099
可以提前把它算好

269
00:09:35,133 --> 00:09:36,899
那第三个呢就是常量折叠

270
00:09:36,900 --> 00:09:39,900
跟已有的常量的代数关系相关的

271
00:09:39,900 --> 00:09:40,300
这种

272
00:09:40,300 --> 00:09:43,566
最经典的就是刚才讲到的卷积BN

273
00:09:43,566 --> 00:09:46,299
这些已经人工提前算好

274
00:09:47,933 --> 00:09:50,766
下面这个算法虽然我是1234列出来

275
00:09:50,766 --> 00:09:52,099
但它其实是一个算法

276
00:09:52,100 --> 00:09:53,800
就是TensorFlow常量折叠的Pass

277
00:09:53,800 --> 00:09:54,700
怎么处理的

278
00:09:54,700 --> 00:09:55,866
首先TensorFlow呢

279
00:09:55,866 --> 00:09:59,466
先去处理Shape、Size、Rank这三类的节点

280
00:09:59,466 --> 00:10:00,766
这三类节点的输出呢

281
00:10:00,766 --> 00:10:02,566
都取决于输入Tensor的形状

282
00:10:02,966 --> 00:10:05,199
跟具体的输入的值呢是没有关系的

283
00:10:05,200 --> 00:10:07,400
所以可以提前把它算出来

284
00:10:07,966 --> 00:10:10,766
算出来之后呢就转换成为Const节点

285
00:10:10,766 --> 00:10:12,933
就是转换成为一些常量的节点

286
00:10:13,733 --> 00:10:14,266
接着呢

287
00:10:14,266 --> 00:10:17,333
去折叠计算图里面的一些常量的操作

288
00:10:17,333 --> 00:10:19,299
如果在遍历整个计算图的时候呢

289
00:10:19,300 --> 00:10:21,533
判断到这个节点的输入呢都是常量

290
00:10:21,600 --> 00:10:23,300
那么这个节点的输出呢

291
00:10:23,300 --> 00:10:25,200
也都可以提前算出来的

292
00:10:26,200 --> 00:10:27,500
基于这个原理呢

293
00:10:27,500 --> 00:10:29,400
去迭代整个计算图

294
00:10:29,666 --> 00:10:30,566
遍历计算图

295
00:10:30,566 --> 00:10:31,466
直到没有任何

296
00:10:31,466 --> 00:10:33,299
常量的节点可以替换为止

297
00:10:33,333 --> 00:10:34,166
那第四步呢

298
00:10:34,166 --> 00:10:35,166
就是去处理

299
00:10:35,166 --> 00:10:37,666
Sum、Min、Max这种point-wise的节点

300
00:10:38,300 --> 00:10:39,133
将这些节点呢

301
00:10:39,133 --> 00:10:41,066
替换成占位符的节点

302
00:10:41,066 --> 00:10:41,766
那这种呢

303
00:10:41,766 --> 00:10:44,266
就是TensorFlow常量折叠的一个pass

304
00:10:44,266 --> 00:10:46,799
而其实在真正实现的时候呢

305
00:10:47,066 --> 00:10:47,733
更重要的是

306
00:10:47,733 --> 00:10:50,166
发现类似于BN折叠这类型的规则

307
00:10:50,166 --> 00:10:52,933
好了今天的内容到这里为止

308
00:10:52,933 --> 00:10:54,866
谢谢各位拜了个拜

309
00:10:57,000 --> 00:10:57,766
卷的不行了

310
00:10:57,766 --> 00:10:58,699
卷的不行了

311
00:10:58,700 --> 00:11:00,366
记得一键三连加关注哦

312
00:11:00,466 --> 00:11:01,799
所有的内容都会开源

313
00:11:01,800 --> 00:11:03,666
在下面这条链接里面

314
00:11:04,000 --> 00:11:05,066
拜了个拜