03.srt

1
00:00:00,000 --> 00:00:04,500
字幕生成：qiaokai 字幕校对：mkwei

2
00:00:05,533 --> 00:00:08,200
哈喽大家好我是ZOMI

3
00:00:08,400 --> 00:00:08,933
今天呢

4
00:00:08,933 --> 00:00:11,500
还是回到AI编译器系列里面

5
00:00:11,500 --> 00:00:12,733
的前端优化

6
00:00:12,733 --> 00:00:13,900
而前端优化上一节呢

7
00:00:13,900 --> 00:00:15,333
讲了一个图层的IR

8
00:00:15,333 --> 00:00:17,066
有了图层IR之后呢

9
00:00:17,066 --> 00:00:19,200
第一个pass就是算子融合

10
00:00:19,300 --> 00:00:22,166
算子融合又叫做OP Fusion

11
00:00:22,166 --> 00:00:23,766
那在今天算子融合里面呢

12
00:00:23,766 --> 00:00:24,766
主要分开

13
00:00:24,766 --> 00:00:26,766
三个内容给大家介绍的

14
00:00:26,800 --> 00:00:29,766
第一个呢就是算子融合的方式

15
00:00:29,766 --> 00:00:31,366
算子融合有很多种方式的

16
00:00:31,366 --> 00:00:33,700
大家不要觉得算子融合是很简单

17
00:00:33,900 --> 00:00:35,700
那平时觉得简单呢

18
00:00:35,700 --> 00:00:38,133
是因为看的更多的是卷积 BN

19
00:00:38,166 --> 00:00:40,133
ReLU这种算子的融合的策略

20
00:00:40,200 --> 00:00:41,800
那有了这些策略之后呢

21
00:00:41,800 --> 00:00:43,366
就会对他进行总结

22
00:00:43,366 --> 00:00:45,366
然后变成一个具体的一个

23
00:00:45,500 --> 00:00:47,666
优化的pass和优化的算法

24
00:00:47,666 --> 00:00:48,300
那接着呢

25
00:00:48,300 --> 00:00:50,900
来看看现在处于哪个位置

26
00:00:51,500 --> 00:00:53,366
第一个就是AI框架呀

27
00:00:53,366 --> 00:00:54,533
好多的AI框架

28
00:00:54,533 --> 00:00:56,566
会对Python的代码进行解析

29
00:00:56,566 --> 00:00:59,333
变成一个图层的IR就是计算图

30
00:00:59,400 --> 00:01:00,500
有了计算图之后呢

31
00:01:00,500 --> 00:01:02,666
就丢给真正的AI编译器

32
00:01:02,666 --> 00:01:04,700
AI编译器就会对代码

33
00:01:04,700 --> 00:01:06,100
对图层的AI

34
00:01:06,400 --> 00:01:08,300
进行前端的优化

35
00:01:08,300 --> 00:01:10,766
而现在呢还是在这个位置

36
00:01:11,700 --> 00:01:12,700
算子融合呢

37
00:01:12,700 --> 00:01:14,133
作为前端的一个pass

38
00:01:14,133 --> 00:01:16,133
对graph IR进行改变

39
00:01:16,133 --> 00:01:18,566
把一些小的算子变成一个大的算子

40
00:01:19,366 --> 00:01:21,366
现在来看看第一个最重要的内容

41
00:01:21,366 --> 00:01:23,600
就是算子的融合方式

42
00:01:23,966 --> 00:01:25,966
其实呢在真正做调研的时候啊

43
00:01:25,966 --> 00:01:28,800
我并不觉得算子的融合方式有多难

44
00:01:28,800 --> 00:01:30,900
但是真正看了之后我才发现哎

45
00:01:30,900 --> 00:01:32,000
原来算子的融合方式

46
00:01:32,000 --> 00:01:33,600
还能这么多吗

47
00:01:34,366 --> 00:01:35,533
好像有点大声啊

48
00:01:35,766 --> 00:01:38,166
小声点不然吵到隔壁录课的时候

49
00:01:38,533 --> 00:01:40,533
现在来看看网络模型的结构啊

50
00:01:40,533 --> 00:01:42,800
网络模型的结构呢其实也非常复杂

51
00:01:42,800 --> 00:01:44,800
可以看到一串一串下来

52
00:01:44,866 --> 00:01:47,333
大部分网络模型都是以这种方式来去

53
00:01:47,333 --> 00:01:48,166
组织的

54
00:01:48,266 --> 00:01:50,700
而其实有非常多的小算子

55
00:01:50,700 --> 00:01:51,966
能不能把这些小算子

56
00:01:51,966 --> 00:01:53,366
合成一个大的算子

57
00:01:53,366 --> 00:01:55,333
减少对Kernel的调度

58
00:01:55,333 --> 00:01:57,133
然后减少对内存的访问呢

59
00:01:57,133 --> 00:01:58,100
这个就是

60
00:01:58,133 --> 00:01:58,566
这个呢

61
00:01:58,566 --> 00:02:01,500
就是做算子融合最大的一个目的

62
00:02:01,866 --> 00:02:04,000
现在来看看融合有哪种方式

63
00:02:04,000 --> 00:02:05,600
假设现在有这么一个

64
00:02:05,600 --> 00:02:07,366
左边的这个计算图

65
00:02:07,400 --> 00:02:08,900
那我做一个纵向的融合

66
00:02:08,900 --> 00:02:11,500
把c算子和d算子进行融合

67
00:02:11,600 --> 00:02:12,266
这个时候呢

68
00:02:12,266 --> 00:02:15,500
已经减少了一次的Kernel的开销

69
00:02:15,500 --> 00:02:18,766
而且也减少了一次中间的数据的缓存

70
00:02:20,200 --> 00:02:21,333
那第二种方式呢

71
00:02:21,333 --> 00:02:23,533
就是我算子a计算完之后呢

72
00:02:23,533 --> 00:02:26,400
分别给b算子和c算子进行执行

73
00:02:26,466 --> 00:02:28,400
而b算子呢和c算子

74
00:02:28,400 --> 00:02:30,100
它是一个并行的执行呢

75
00:02:30,100 --> 00:02:32,366
就是b和c是同时执行的

76
00:02:32,366 --> 00:02:35,000
但这里面呢就会有两次的访存

77
00:02:35,300 --> 00:02:37,000
那我向右边的这个所示

78
00:02:37,000 --> 00:02:39,933
我把a算子和b算子一起去执行

79
00:02:39,933 --> 00:02:42,466
再把a算子和c算子进行执行

80
00:02:42,500 --> 00:02:45,000
AB跟AC是同时并行的

81
00:02:45,100 --> 00:02:48,300
这个时候呢我只执行两次Kernel的开销

82
00:02:48,300 --> 00:02:50,366
而且并发呢只有一个来回

83
00:02:50,366 --> 00:02:52,566
就是我只需要一次的访存

84
00:02:52,566 --> 00:02:54,300
这种方式也是可以的

85
00:02:54,766 --> 00:02:57,533
另外图还是这个图他的方式有很多啊

86
00:02:57,533 --> 00:03:00,700
像现在左边的b和c呢需要

87
00:03:00,700 --> 00:03:03,333
并发然后他需要调用两次的Kernel开销

88
00:03:03,766 --> 00:03:06,333
右边呢我把b和c进行融合

89
00:03:06,333 --> 00:03:08,266
那这种横向的融合

90
00:03:08,500 --> 00:03:10,366
确实叫做横向融合

91
00:03:10,366 --> 00:03:13,066
这里面呢减少了一次Kernel的调度

92
00:03:13,066 --> 00:03:14,900
而且两次的计算结果呢

93
00:03:14,900 --> 00:03:17,300
都放在同一个内存块里面

94
00:03:17,400 --> 00:03:20,066
加快了内存访问的效率

95
00:03:21,300 --> 00:03:22,933
哎没想到这里面这种图

96
00:03:22,933 --> 00:03:24,466
还有这么多融合方式啊

97
00:03:24,466 --> 00:03:25,900
现在呢来看看了

98
00:03:25,900 --> 00:03:28,200
a b c三个算子进行计算的时候呢

99
00:03:28,200 --> 00:03:30,066
会有三次的Kernel开销

100
00:03:30,066 --> 00:03:32,133
如果我把a和b进行融合

101
00:03:32,133 --> 00:03:34,500
就是a和b进行融合呢

102
00:03:34,666 --> 00:03:35,600
然后合并完之后

103
00:03:35,600 --> 00:03:38,066
把所有的结果丢在内存里面

104
00:03:38,133 --> 00:03:40,300
这个时候呢再给c进行计算

105
00:03:40,300 --> 00:03:42,600
另外一个结果呢给下一个进行计算

106
00:03:42,600 --> 00:03:43,166
这个时候呢

107
00:03:43,166 --> 00:03:45,500
可以提高内存的使用效率

108
00:03:45,766 --> 00:03:47,666
从上面举个例子可以看出

109
00:03:47,666 --> 00:03:49,266
不同的算子融合策略

110
00:03:49,266 --> 00:03:51,266
会产生不同的算子的开销

111
00:03:51,300 --> 00:03:52,066
也可以带来

112
00:03:52,066 --> 00:03:54,500
不同的内存访问的性能的提升

113
00:03:55,400 --> 00:03:57,566
现在呢来看看一个比较

114
00:03:57,700 --> 00:03:59,466
综合性的一个

115
00:03:59,566 --> 00:04:00,400
融合的方式

116
00:04:00,400 --> 00:04:03,566
现在有一个卷积算子1*1*256

117
00:04:03,566 --> 00:04:07,200
然后有一个卷积算子3*3*256

118
00:04:07,300 --> 00:04:10,266
我会把鼠标所在的这个卷积的算子呢

119
00:04:10,266 --> 00:04:11,700
把它进行一个扩充

120
00:04:11,866 --> 00:04:15,366
Enlarge卷积然后变成一个3*3*256

121
00:04:15,566 --> 00:04:15,933
接着呢

122
00:04:15,933 --> 00:04:18,900
对这两个3*3*256的进行一个合并

123
00:04:18,900 --> 00:04:20,266
合并成一个算子

124
00:04:20,266 --> 00:04:22,866
变成卷积3*3*522

125
00:04:23,066 --> 00:04:25,266
那有了这个横向的融合之后呢

126
00:04:25,266 --> 00:04:27,133
想做一个纵向的融合

127
00:04:27,200 --> 00:04:29,300
纵向的融合呢现在有个Split

128
00:04:29,300 --> 00:04:30,533
有个卷积有个Add

129
00:04:30,533 --> 00:04:33,100
现在呢把Split卷积Add

130
00:04:33,333 --> 00:04:36,766
把它合成一个算子变成Conv2d 3*3*256

131
00:04:36,766 --> 00:04:38,700
但是我觉得还不够

132
00:04:38,700 --> 00:04:40,700
像激活一般来说都可以合并在

133
00:04:40,700 --> 00:04:41,866
前一个计算里面

134
00:04:41,866 --> 00:04:44,700
所以呢要把卷积ReLU合并起来

135
00:04:45,300 --> 00:04:46,533
这个操作很有意思

136
00:04:46,533 --> 00:04:48,166
就是一开始的计算图

137
00:04:48,166 --> 00:04:49,000
是比较复杂的

138
00:04:49,000 --> 00:04:50,000
比较多算子的

139
00:04:50,066 --> 00:04:51,000
最后融合起来了

140
00:04:51,000 --> 00:04:53,100
就算子就减少了很多

141
00:04:53,100 --> 00:04:55,466
虽然有部分的计算呢可能会变大

142
00:04:55,466 --> 00:04:58,566
但是总体来说减少了Kernel的开销

143
00:04:58,566 --> 00:05:01,100
减少了对内存的不断的访问

144
00:05:03,766 --> 00:05:05,166
刚才上面那个例子呢

145
00:05:05,166 --> 00:05:07,766
其实已经很清楚的看到了

146
00:05:08,500 --> 00:05:10,800
算子融合确实对整个计算图

147
00:05:10,800 --> 00:05:12,066
或者整个运算的时候呢

148
00:05:12,066 --> 00:05:13,333
产生很大的收益

149
00:05:13,333 --> 00:05:14,933
那主要是解决两个问题

150
00:05:14,933 --> 00:05:16,966
一个是内存墙的问题

151
00:05:16,966 --> 00:05:19,266
就是减少访问内存

152
00:05:19,600 --> 00:05:21,966
那第二个呢就是并行墙的问题

153
00:05:21,966 --> 00:05:24,333
将计算图的算子的节点呢

154
00:05:24,333 --> 00:05:25,400
并行的编排

155
00:05:25,400 --> 00:05:28,133
从而提升整体的并行的效率

156
00:05:28,133 --> 00:05:30,600
这个呢就是做算子融合的目的

157
00:05:32,000 --> 00:05:34,533
现在呢一个非常经典的

158
00:05:34,533 --> 00:05:37,000
卷积 BN ReLU三个算子

159
00:05:37,000 --> 00:05:39,866
三个非常经典的算子进行融合

160
00:05:39,866 --> 00:05:41,600
看看它到底是怎么操作的

161
00:05:41,666 --> 00:05:43,366
那首先呢看一下BN

162
00:05:43,366 --> 00:05:44,400
在计算的时候呢

163
00:05:44,400 --> 00:05:46,933
需要去求那个输入数据

164
00:05:46,966 --> 00:05:49,766
特别是X的一个均值和方差

165
00:05:50,100 --> 00:05:52,533
在这里面呢先对X求均值

166
00:05:52,533 --> 00:05:54,500
然后再求方差

167
00:05:54,500 --> 00:05:56,733
然后呢使用均值和方差

168
00:05:56,766 --> 00:05:58,200
对每一个数据

169
00:05:58,200 --> 00:06:01,400
输入的数据做一个归一化和缩放的

170
00:06:02,166 --> 00:06:03,800
而这个均值和方差呢

171
00:06:03,800 --> 00:06:05,900
是学习的一个参数

172
00:06:05,900 --> 00:06:06,533
而X呢

173
00:06:06,533 --> 00:06:09,600
就是前面卷积算出来的一个输入

174
00:06:09,600 --> 00:06:11,800
接着呢给ReLU进行计算

175
00:06:11,800 --> 00:06:14,866
这个就是BN的一个正常的流程

176
00:06:15,966 --> 00:06:16,766
那看看

177
00:06:16,766 --> 00:06:18,933
正向的时候其实还是很简单

178
00:06:18,933 --> 00:06:20,400
但是反向的时候

179
00:06:20,400 --> 00:06:22,700
问题就变得非常复杂了

180
00:06:22,733 --> 00:06:23,800
反向的时候呢

181
00:06:23,800 --> 00:06:25,966
数据是从后面往前流的

182
00:06:25,966 --> 00:06:29,733
就我先算完ReLU再算BN再算卷积

183
00:06:29,900 --> 00:06:31,300
我在反向的时候呢

184
00:06:31,300 --> 00:06:33,166
首先我要求参数的误差

185
00:06:33,166 --> 00:06:35,700
例如∆γ ∆ß

186
00:06:35,700 --> 00:06:37,600
而这两个参数误差呢

187
00:06:37,600 --> 00:06:40,333
会依赖于输入∆y

188
00:06:41,300 --> 00:06:43,600
和之前的输入X

189
00:06:43,600 --> 00:06:46,100
这个X呢也会对它产生影响

190
00:06:46,133 --> 00:06:48,100
所以呢反向的过程呢

191
00:06:48,100 --> 00:06:51,666
包括求参数误差还有求输入误差

192
00:06:51,733 --> 00:06:52,733
两个部分

193
00:06:52,733 --> 00:06:55,500
BN这个算子呢在反向计算的时候呢

194
00:06:55,500 --> 00:06:56,966
关键的访存特征就是

195
00:06:56,966 --> 00:06:59,266
我需要对内存里面的这些参数

196
00:06:59,400 --> 00:07:00,566
进行访问

197
00:07:00,566 --> 00:07:01,133
第二个呢

198
00:07:01,133 --> 00:07:03,866
我需要大量的去访问∆y

199
00:07:03,866 --> 00:07:06,400
把之前反向计算出来的一个

200
00:07:06,400 --> 00:07:07,200
输出误差

201
00:07:07,200 --> 00:07:09,133
给到∆γ计算

202
00:07:09,133 --> 00:07:10,766
给到∆ß计算

203
00:07:10,800 --> 00:07:13,766
还给到∆X进行计算

204
00:07:13,766 --> 00:07:16,166
这种方式呢就引起计算

205
00:07:16,166 --> 00:07:18,600
对内存进行大量的访问

206
00:07:18,900 --> 00:07:21,166
在实际的AI框架计算里面呢

207
00:07:21,166 --> 00:07:23,966
它这个∆y呀不是单单一个数

208
00:07:23,966 --> 00:07:26,466
而是一个向量一个矩阵一个张量

209
00:07:26,500 --> 00:07:28,533
就可能甚至是高维的张量

210
00:07:28,533 --> 00:07:29,800
数据量是非常大的

211
00:07:29,800 --> 00:07:32,133
所以它对访存的要求是非常高的

212
00:07:32,133 --> 00:07:34,500
现在来看看计算访存的分析

213
00:07:34,500 --> 00:07:37,500
就是对这个访存进行一个简单的分析

214
00:07:37,966 --> 00:07:39,866
在网络模型训练的过程当中

215
00:07:39,866 --> 00:07:42,300
需要保存每一层的前项的

216
00:07:42,300 --> 00:07:43,366
计算的结果

217
00:07:43,400 --> 00:07:45,566
就是刚才所谓的BN的X

218
00:07:45,566 --> 00:07:48,100
在反向计算误差的时候用到的

219
00:07:48,100 --> 00:07:48,600
但是

220
00:07:48,600 --> 00:07:51,333
随着深度模型的网络模型越大

221
00:07:51,333 --> 00:07:52,600
模型层数越深

222
00:07:52,600 --> 00:07:55,200
需要保存的中间结果和参数量

223
00:07:55,200 --> 00:07:56,800
就会急剧的增加

224
00:07:57,133 --> 00:07:59,333
这个时候呢就需要消耗大量的内存

225
00:07:59,333 --> 00:08:00,200
所以在训练

226
00:08:00,200 --> 00:08:01,733
一个大一点的网络模型的时候

227
00:08:01,733 --> 00:08:03,000
经常说内存不够了

228
00:08:03,000 --> 00:08:05,533
希望内存或者显存更大

229
00:08:05,533 --> 00:08:06,500
而另一方面呢

230
00:08:06,500 --> 00:08:08,400
因为加速器或者NPU

231
00:08:08,400 --> 00:08:09,500
或者芯片上面的

232
00:08:09,500 --> 00:08:11,600
访存容量是非常有限的

233
00:08:11,600 --> 00:08:14,333
没办法无限制的去保存数据

234
00:08:14,500 --> 00:08:15,133
所以这个时候呢

235
00:08:15,133 --> 00:08:16,700
就需要把中间结果

236
00:08:16,700 --> 00:08:18,666
可能会offload到

237
00:08:18,866 --> 00:08:20,066
CPU内存里面

238
00:08:20,066 --> 00:08:22,533
并且在反向的时候再读取出来

239
00:08:22,566 --> 00:08:24,766
那这种方式是非常低效的

240
00:08:25,100 --> 00:08:25,533
那下面呢

241
00:08:25,533 --> 00:08:27,933
来看看一个更加明确的例子

242
00:08:27,933 --> 00:08:29,666
还是刚才的卷积 BN ReLU

243
00:08:29,666 --> 00:08:30,500
三个算子

244
00:08:30,533 --> 00:08:33,333
中间的这个横框呢就是内存

245
00:08:33,333 --> 00:08:35,899
那假设现在有两块内存

246
00:08:36,000 --> 00:08:36,866
正向的时候呢

247
00:08:36,866 --> 00:08:39,666
需要 Z1 W1进行完卷积计算

248
00:08:39,666 --> 00:08:40,799
输出X

249
00:08:41,066 --> 00:08:43,600
在BN计算的时候呢我需要把X输进去

250
00:08:43,600 --> 00:08:45,166
把γ ß输进去

251
00:08:45,166 --> 00:08:47,533
求得平均值和方差

252
00:08:47,566 --> 00:08:50,366
还有Y这个时候呢我Y输给我的ReLU

253
00:08:50,366 --> 00:08:51,566
输出我的Z

254
00:08:51,733 --> 00:08:53,000
在反向的时候呢

255
00:08:53,000 --> 00:08:55,333
我同样需要把刚才计算的Y

256
00:08:55,400 --> 00:08:58,200
把刚才的X还有∆X

257
00:08:58,366 --> 00:08:59,066
∆γ

258
00:08:59,066 --> 00:08:59,866
Ƨ

259
00:08:59,900 --> 00:09:03,333
∆X也是需要计算和存储的

260
00:09:03,333 --> 00:09:05,900
可以看到这个图呢有大量的箭头

261
00:09:05,900 --> 00:09:07,500
而每种箭头都不一样

262
00:09:07,500 --> 00:09:09,866
实线红色呢就是正向

263
00:09:09,866 --> 00:09:11,600
对内存的访问

264
00:09:11,600 --> 00:09:13,866
虚线红色的就是求参数误差

265
00:09:13,866 --> 00:09:15,100
对内存的访问

266
00:09:15,366 --> 00:09:16,733
而绿色的虚线呢

267
00:09:16,733 --> 00:09:18,666
就是求输入的误差

268
00:09:18,666 --> 00:09:20,133
对内存的访问

269
00:09:20,133 --> 00:09:23,400
可以看到对内存的访问次数非常的多

270
00:09:23,400 --> 00:09:25,000
每次都要大量的交互

271
00:09:25,000 --> 00:09:26,066
所以希望把

272
00:09:26,100 --> 00:09:28,533
所以希望把这些零散的算子

273
00:09:28,533 --> 00:09:29,400
合成一个大的Kernel

274
00:09:29,400 --> 00:09:32,333
可能把这些零散的一些数据

275
00:09:32,333 --> 00:09:33,900
直接一次性的读取

276
00:09:33,900 --> 00:09:35,200
一次性的读出

277
00:09:36,133 --> 00:09:37,666
针对卷积 BN ReLU呢

278
00:09:37,666 --> 00:09:40,133
把BN重构成为BN B

279
00:09:40,133 --> 00:09:43,500
还有BN A就把一个BN算子呢拆成两个

280
00:09:43,500 --> 00:09:46,733
首先BN A呢来算均值和方差

281
00:09:46,800 --> 00:09:49,766
BN B呢就完成归一化和缩放

282
00:09:49,966 --> 00:09:50,800
像刚才说到

283
00:09:50,800 --> 00:09:53,300
把这些参数呢统一的计算完之后呢

284
00:09:53,300 --> 00:09:54,966
输给一个BN B

285
00:09:55,133 --> 00:09:57,133
然后给ReLU进行执行

286
00:09:57,166 --> 00:09:58,166
ReLU执行完之后

287
00:09:58,166 --> 00:10:00,200
再给卷积进行执行

288
00:10:00,200 --> 00:10:01,666
卷积执行完之后呢

289
00:10:01,666 --> 00:10:03,400
再求误差和方差

290
00:10:03,800 --> 00:10:05,766
并把计算后的最终的结果

291
00:10:05,766 --> 00:10:07,600
写到内存里面

292
00:10:07,600 --> 00:10:10,466
那这种方式呢就是把算子进行融合

293
00:10:10,500 --> 00:10:12,800
可能看这个图你不是很直观

294
00:10:12,800 --> 00:10:15,133
下面呢来看看具体的计算公式

295
00:10:15,366 --> 00:10:17,466
假设计卷机公式呢是w

296
00:10:17,466 --> 00:10:18,366
乘以x加b

297
00:10:18,366 --> 00:10:19,100
等于z

298
00:10:19,100 --> 00:10:21,766
在BN的计算呢就是我把z计算完的

299
00:10:21,766 --> 00:10:22,599
然后减去mean

300
00:10:22,600 --> 00:10:24,333
然后做一个BN的计算

301
00:10:24,333 --> 00:10:26,133
这个呢就是BN计算公式

302
00:10:26,133 --> 00:10:27,000
得到y

303
00:10:27,266 --> 00:10:28,766
y呢再进行一个ReLU

304
00:10:28,766 --> 00:10:31,533
那可以把ReLU先抛开一边

305
00:10:31,766 --> 00:10:33,666
融合后的简化的算子

306
00:10:33,666 --> 00:10:34,066
融合

307
00:10:34,066 --> 00:10:37,266
简化的卷积和BN之间呢就是这种关系

308
00:10:37,266 --> 00:10:38,600
我先求W1撇

309
00:10:38,600 --> 00:10:39,666
再求B1撇

310
00:10:39,666 --> 00:10:42,400
最后呢再求y真正的输出

311
00:10:44,700 --> 00:10:46,166
下面来看看TVM

312
00:10:46,166 --> 00:10:48,700
定义的一个融合规则和具体的算法

313
00:10:48,766 --> 00:10:49,100
首先

314
00:10:49,100 --> 00:10:51,766
要了解一下TVM的一个支配树啊

315
00:10:51,966 --> 00:10:53,666
先来讲讲TVM吧

316
00:10:53,733 --> 00:10:54,266
TVM呢

317
00:10:54,266 --> 00:10:57,266
实际上是一个陈天奇主导的开源项目

318
00:10:57,400 --> 00:10:57,866
之前呢

319
00:10:57,866 --> 00:11:00,166
主要是用来做一个推理的AI编译器

320
00:11:00,266 --> 00:11:01,866
那现在呢来看看TVM

321
00:11:01,866 --> 00:11:03,733
其实整体的算子融合的策略

322
00:11:03,733 --> 00:11:05,533
是基于支配树来实现的

323
00:11:05,900 --> 00:11:08,066
嗯那什么是支配树吗

324
00:11:08,800 --> 00:11:09,533
支配树呢

325
00:11:09,533 --> 00:11:12,400
实际上是由各个支配点来构成的树

326
00:11:12,933 --> 00:11:15,533
嗯那什么又是支配点呢

327
00:11:16,333 --> 00:11:18,700
支配点就是所有能够达到当前

328
00:11:18,700 --> 00:11:20,400
节点路径的公共

329
00:11:20,500 --> 00:11:21,500
祖先节点

330
00:11:21,500 --> 00:11:23,566
叫做LCA

331
00:11:23,800 --> 00:11:25,700
那听上去有点拗口

332
00:11:26,066 --> 00:11:28,200
来看看一个真正的图

333
00:11:28,766 --> 00:11:31,166
右边的这个图呢就是支配树

334
00:11:31,166 --> 00:11:32,333
每个节点呢

335
00:11:32,333 --> 00:11:33,466
就是每个支配点

336
00:11:33,466 --> 00:11:35,600
其实都有一个对应的数值

337
00:11:35,666 --> 00:11:36,733
而这个数值呢

338
00:11:36,733 --> 00:11:38,733
跟这个数值可能会串起来

339
00:11:38,733 --> 00:11:41,533
像这个呢叫它叫做一个支配点

340
00:11:41,533 --> 00:11:44,466
像这个Inplace呢它就是一个支配点

341
00:11:44,466 --> 00:11:46,333
每一个算子都是一个支配点

342
00:11:46,333 --> 00:11:48,100
这么几个算子组合起来

343
00:11:48,100 --> 00:11:49,766
能到达支配点

344
00:11:49,766 --> 00:11:51,200
这几个算子组合起来

345
00:11:51,200 --> 00:11:52,866
也能到达支配点

346
00:11:52,933 --> 00:11:55,200
这几个算子这种方式组合

347
00:11:55,300 --> 00:11:57,466
也能够达到支配点

348
00:11:58,700 --> 00:12:01,000
那为什么需要支配树呢

349
00:12:01,333 --> 00:12:02,200
有了支配树

350
00:12:02,200 --> 00:12:03,400
就非常的

351
00:12:03,500 --> 00:12:05,533
方便的去检查每一个Node

352
00:12:05,666 --> 00:12:09,000
到支配点的Node是否符合融合的策略

353
00:12:09,166 --> 00:12:13,133
假设我现在把478做了一个融合

354
00:12:13,133 --> 00:12:14,500
那这个时候呢

355
00:12:14,500 --> 00:12:16,666
478这个算子已经融合掉了

356
00:12:16,733 --> 00:12:18,400
578这几个算子

357
00:12:18,400 --> 00:12:20,100
是没办法进行融合的了

358
00:12:20,100 --> 00:12:23,266
因为78我已经跟4融合成一个大的算子

359
00:12:23,400 --> 00:12:25,800
有了这个支配树和这个支配点呢

360
00:12:25,800 --> 00:12:26,766
就知道

361
00:12:26,800 --> 00:12:29,600
哪些节点能够符合融合规则

362
00:12:29,800 --> 00:12:31,333
哪些剩下的节点

363
00:12:31,333 --> 00:12:33,966
不会对融合规则产生问题

364
00:12:34,266 --> 00:12:36,500
而支配树的生成呢有几个步骤

365
00:12:36,500 --> 00:12:38,600
就是根据计算图

366
00:12:38,600 --> 00:12:41,266
或者计算图其实是一个DAG图嘛

367
00:12:41,366 --> 00:12:44,100
生成一个深度优先搜索的树

368
00:12:44,166 --> 00:12:44,866
那接着呢

369
00:12:44,866 --> 00:12:47,133
根据这个深度优先搜索的树呢

370
00:12:47,133 --> 00:12:48,466
产生边

371
00:12:48,800 --> 00:12:51,266
接下来会根据深度优先搜索树呢

372
00:12:51,266 --> 00:12:54,100
对应的边产生DOM数

373
00:12:54,133 --> 00:12:56,733
然后呢加入一个数据结构叫做Group

374
00:12:56,733 --> 00:13:00,200
来描述多个的Node能够被融合

375
00:13:01,200 --> 00:13:02,600
那这呢是一个Node

376
00:13:02,600 --> 00:13:04,300
这是一个Node这是一个Node

377
00:13:04,300 --> 00:13:05,733
这些都是一个节点

378
00:13:05,733 --> 00:13:08,600
通过这个树呢去描述那些Node

379
00:13:08,600 --> 00:13:10,266
节点算子

380
00:13:10,266 --> 00:13:10,666
这个呢

381
00:13:10,666 --> 00:13:13,266
就是整体TVM支配树的一个作用

382
00:13:13,266 --> 00:13:14,866
那了解完大概的作用之后呢

383
00:13:14,866 --> 00:13:17,733
看看TVM的算子的融合的流程

384
00:13:18,066 --> 00:13:19,800
首先呢会通过AST

385
00:13:19,800 --> 00:13:21,366
就是源码转换

386
00:13:21,366 --> 00:13:23,133
把计算图的IR

387
00:13:23,133 --> 00:13:25,166
转换成为Relay的IR

388
00:13:25,266 --> 00:13:28,100
那这个Relay的IR呢是tvm里面自己的IR

389
00:13:28,200 --> 00:13:30,466
接着呢去遍历这个Relay的IR

390
00:13:30,466 --> 00:13:31,933
遍历完这个Relay的IR呢

391
00:13:31,933 --> 00:13:33,200
会利用Relay的IR

392
00:13:33,466 --> 00:13:35,933
建立刚才所说的DAG树

393
00:13:35,933 --> 00:13:39,266
那DAG树呢主要是生成支配树的

394
00:13:39,333 --> 00:13:40,800
生成支配树之后呢

395
00:13:40,800 --> 00:13:43,700
就可以应用算子的融合的算法

396
00:13:43,700 --> 00:13:44,733
和规则了

397
00:13:44,866 --> 00:13:47,133
现在来打开呢具体看看

398
00:13:47,400 --> 00:13:49,966
第一个呢输入是一个DAG的图

399
00:13:49,966 --> 00:13:51,866
就是计算图是一个DAG嘛

400
00:13:51,866 --> 00:13:55,300
进行深度优先遍历生成DFS树

401
00:13:55,533 --> 00:13:57,133
那在这里面生成的过程当中呢

402
00:13:57,133 --> 00:13:58,666
除了单纯的记录

403
00:13:58,666 --> 00:14:00,966
每一个Node的节点的深度之外

404
00:14:01,100 --> 00:14:02,900
Node呢就是算子嘛

405
00:14:02,900 --> 00:14:05,066
还需要为每一个节点保存

406
00:14:05,066 --> 00:14:06,366
它相连的边

407
00:14:06,366 --> 00:14:07,966
保存的它相连的边呢

408
00:14:07,966 --> 00:14:09,700
就知道对应的索引

409
00:14:09,766 --> 00:14:11,933
才好找到LCA

410
00:14:11,966 --> 00:14:15,533
找到能够达到当前节点的公共子路径

411
00:14:17,400 --> 00:14:17,900
接着呢

412
00:14:17,900 --> 00:14:20,800
在第二步就是根据DFS树

413
00:14:20,800 --> 00:14:22,866
因为刚才已经建立了DFS树

414
00:14:22,866 --> 00:14:24,366
生成一个DOM的树

415
00:14:24,366 --> 00:14:26,500
那TVM呢就使用group这个概念

416
00:14:26,500 --> 00:14:27,800
那这个概念实际上呢

417
00:14:27,800 --> 00:14:30,366
它是一个数据结构来描述这些Node

418
00:14:30,666 --> 00:14:32,700
这些算子能不能做融合

419
00:14:32,700 --> 00:14:35,333
如果一个算子不能和其他算子融合

420
00:14:35,333 --> 00:14:37,733
那这个group呢由自己去组成

421
00:14:37,766 --> 00:14:39,266
如果这几个算子

422
00:14:39,266 --> 00:14:42,366
例如我遇到卷积 BN ReLU的时候呢

423
00:14:42,366 --> 00:14:44,933
我就把这几个算子合并成一个group

424
00:14:44,966 --> 00:14:46,466
这个时候呢只有一个group

425
00:14:46,966 --> 00:14:49,166
最后一步呢就是定好我的group之后呢

426
00:14:49,166 --> 00:14:50,700
去遍历每一个Node

427
00:14:50,700 --> 00:14:52,966
找到它的支配点所在的路径

428
00:14:52,966 --> 00:14:54,266
是否符合

429
00:14:54,366 --> 00:14:55,600
融合规则

430
00:14:55,600 --> 00:14:56,733
而这个时候呢

431
00:14:56,733 --> 00:14:59,266
融合规则就变得异常的重要

432
00:14:59,266 --> 00:15:02,266
现在来看看有哪几种融合规则

433
00:15:02,266 --> 00:15:06,466
首先呢有一种就是映射injective

434
00:15:06,800 --> 00:15:07,800
类似于加

435
00:15:07,800 --> 00:15:09,466
还有Pointwise这种节点呢

436
00:15:09,466 --> 00:15:11,133
就可以做一个injective

437
00:15:11,133 --> 00:15:12,533
然后做一个融合

438
00:15:12,700 --> 00:15:14,566
例如我这里是一个加号

439
00:15:14,566 --> 00:15:16,500
第二次也是一个加号这个

440
00:15:16,500 --> 00:15:18,300
时候呢就可以把两次的加号

441
00:15:18,300 --> 00:15:19,933
变成一次的相乘

442
00:15:20,333 --> 00:15:23,800
那现在还有reduction就是简约函数

443
00:15:23,933 --> 00:15:25,866
这个简约函数也比较好理解

444
00:15:25,866 --> 00:15:28,600
就是输入到输出具有降维的性质

445
00:15:28,700 --> 00:15:32,100
例如sum max min就是求最大值求最小值

446
00:15:32,200 --> 00:15:34,733
这些都可以把它进行融合

447
00:15:35,166 --> 00:15:37,066
例如我加完之后求最大值

448
00:15:37,066 --> 00:15:37,733
那这个时候呢

449
00:15:37,733 --> 00:15:38,933
可能可以把它

450
00:15:39,000 --> 00:15:40,933
合成一个具体的算子

451
00:15:41,966 --> 00:15:42,300
另外呢

452
00:15:42,300 --> 00:15:45,100
第三种就是人工定义的规则

453
00:15:45,100 --> 00:15:47,500
人工呢去发现一些例如卷积

454
00:15:47,566 --> 00:15:49,600
BN ReLU这是可以融合的

455
00:15:49,600 --> 00:15:52,700
人工呢去设定一些融合的规则

456
00:15:53,133 --> 00:15:54,500
当然呢人工融合的规则

457
00:15:54,500 --> 00:15:55,766
除了卷积 BN ReLU呢

458
00:15:55,766 --> 00:15:57,599
可能还有add sqrt

459
00:15:57,666 --> 00:16:00,000
还有那个相乘再相加

460
00:16:00,000 --> 00:16:00,766
那像这种呢

461
00:16:00,766 --> 00:16:03,466
会人工的定义了非常多的pass

462
00:16:03,733 --> 00:16:04,300
编译器呢

463
00:16:04,300 --> 00:16:07,300
虽然看上去很高大上很深奥的技术

464
00:16:07,300 --> 00:16:09,866
实际上啊在做到编译器底层的时候

465
00:16:09,866 --> 00:16:12,700
还是定义了很多人工的规则

466
00:16:12,866 --> 00:16:13,766
通过工程师呢

467
00:16:13,766 --> 00:16:16,333
去抽象了很多不同的规则出来

468
00:16:16,333 --> 00:16:19,066
然后把这些规则呢写到AI编译器里面

469
00:16:19,066 --> 00:16:21,066
最后变成一个通用的法则

470
00:16:21,066 --> 00:16:22,100
或者通用的范式

471
00:16:22,100 --> 00:16:23,600
或者通用的pass函数

472
00:16:23,600 --> 00:16:26,100
最后一种就是没有办法进行融合的

473
00:16:26,100 --> 00:16:28,733
例如sort这种排序的没办法进行融合

474
00:16:28,733 --> 00:16:31,400
那只能单独的把它拿出来

475
00:16:32,200 --> 00:16:34,400
那最后看看实验的结果呢

476
00:16:34,400 --> 00:16:35,733
像卷积 BN ReLU

477
00:16:35,733 --> 00:16:37,500
然后depthwise-conv bn relu

478
00:16:37,500 --> 00:16:38,966
还有rnn lstm

479
00:16:38,966 --> 00:16:41,133
这些都可以做算子的融合的

480
00:16:41,200 --> 00:16:42,166
融合之后呢

481
00:16:42,166 --> 00:16:43,133
就是蓝色

482
00:16:43,133 --> 00:16:46,066
这条线实际上执行的效率更高

483
00:16:46,066 --> 00:16:48,466
减少了访存减少了Kernel调用的

484
00:16:48,466 --> 00:16:51,533
次数所以说会比绿色的这个

485
00:16:51,700 --> 00:16:54,166
没有进行融合的时候效率更高

486
00:16:55,466 --> 00:16:57,100
好了进来回顾一下

487
00:16:57,100 --> 00:16:58,500
其实算子的融合呀

488
00:16:58,500 --> 00:17:00,866
分开横向融合和纵向融合

489
00:17:00,900 --> 00:17:01,766
但实际上呢

490
00:17:01,766 --> 00:17:03,533
会根据AI模型的排布

491
00:17:03,533 --> 00:17:06,366
衍生出更多不同的融合的策略

492
00:17:06,700 --> 00:17:08,200
那我后面第二个呢

493
00:17:08,200 --> 00:17:10,600
就是通过卷积 BN ReLU这个算子呢

494
00:17:10,600 --> 00:17:12,933
了解到对算子如何进行融合

495
00:17:12,933 --> 00:17:14,533
还有融合后的计算

496
00:17:14,600 --> 00:17:15,466
基本上可以

497
00:17:15,466 --> 00:17:17,766
极大的减少对访存的压力

498
00:17:17,766 --> 00:17:19,266
但是这个算子的融合呢

499
00:17:19,266 --> 00:17:22,533
需要在数学上保证语义的一致

500
00:17:22,600 --> 00:17:24,700
那第三个呢就是在AI编译器里面呢

501
00:17:24,700 --> 00:17:26,733
一般融合规则都是通过pass

502
00:17:26,733 --> 00:17:27,566
来承载的

503
00:17:27,566 --> 00:17:30,166
不同的pass处理不同的融合规则

504
00:17:30,166 --> 00:17:32,466
而融合规则呢就是需要人工

505
00:17:32,466 --> 00:17:34,666
工程师去预先定义

506
00:17:34,666 --> 00:17:37,133
好的好了今天到此为止

507
00:17:37,133 --> 00:17:37,866
谢谢各位

508
00:17:37,866 --> 00:17:38,733
拜了个拜

509
00:17:38,966 --> 00:17:40,733
卷的不行了卷的不行了

510
00:17:40,733 --> 00:17:42,466
记得一键三连加关注哦

511
00:17:42,533 --> 00:17:43,900
所有的内容都会开源

512
00:17:43,900 --> 00:17:45,700
在下面这条链接里面

513
00:17:46,133 --> 00:17:47,066
拜了个拜