Function definition and Function implementation does not match

[AshStuff]

@ line 246. The function starts with _bn_relu_conv() but you have implemented as conv --> batch_norm --> relu

[relh]

If you go here you can see that it was the way you described before @titu1994 changed it: f9eaf9a.

Presumably there was a reason for this.

[chuong98]

I am curiousl too, since the original paper proposed the Micro-block as: BN-Act-Conv2d. This is also replicated by other pytorch implementation, and also used in DenseNet.
However, the ResNeXt uses Conv2d-BN-Act.
So between the two implementations, the Conv2d is missing at the first block. Can you explain?

[relh]

When BN-Act comes first it adds batch norm and an activation to the input images.

At least that's one reason one might write your helper function thus. Obviously when tiled, these layers look similar for most of the network.

There's lots of interesting ways to style residual blocks. We may as well be doing it like NAS does it when we design networks.

[chuong98]

Hi thanks Relh.
Actually, in the paper "Identity mappings in deep residual networks", He et al. examined several structures, include BN-Act-Cov, and Cov-Bn-Act, and found that the structure BN-Act-Conv is better.
However, later in ResNeXt, they changed back to Conv-Bn-Act.
Now, there are discussion that Act-Bn is better than Bn-Act. I guess NAS can choose the best structure.