[FEATURE] convert model for 1D inputs

[pfeatherstone]

Is there a way to convert timm models for 1D inputs?
I realize that a 1D tensor with shape [B,C,S] can be reshaped to [B,C,1,S] or [B,C,S,1], but then the filters are unnecessarily large.
Maybe for transformer models this is easier since patch embeddings are linearized but it would be cool if CNNs could also be converted to 1D.
Cheers

[rwightman]

@pfeatherstone actually had a use case for doing this for 3d come up recently, think it might be a need extension, would make sense to cover 1d as well if I did that, curious what the use case is/was?

[pfeatherstone]

I regularly build classifier models and feature extractors for 1D time series

[pfeatherstone]

So far i have this conversion function:

def _SqueezeExciteForwardMethod(self, x):
    x_se = x.mean((2,), keepdim=True)
    x_se = self.conv_reduce(x_se)
    x_se = self.act1(x_se)
    x_se = self.conv_expand(x_se)
    return x * self.gate(x_se)
    
class GlobalAvgPool1d(nn.Module):
    def __init__(self):
        super(GlobalAvgPool1d, self).__init__()

    def forward(self, x):
        return x.flatten(2).mean(-1)
    
def convert2DModelTo1D(module):
    # Reset global functions
    timm.models.efficientnet_blocks.SqueezeExcite.forward = _SqueezeExciteForwardMethod

    module_output = module

    if hasattr(module, "global_pool"):
        module.global_pool = GlobalAvgPool1d()

    if isinstance(module, nn.BatchNorm2d):
        module_output = nn.BatchNorm1d(
            module.num_features,
            module.eps,
            module.momentum,
            module.affine,
            module.track_running_stats)
        if module.affine:
            with torch.no_grad():
                module_output.weight    = module.weight
                module_output.bias      = module.bias
        module_output.running_mean          = module.running_mean
        module_output.running_var           = module.running_var
        module_output.num_batches_tracked   = module.num_batches_tracked
        if hasattr(module, "qconfig"):
            module_output.qconfig = module.qconfig

    elif isinstance(module, nn.Conv2d):
        module_output = nn.Conv1d(
            module.in_channels,
            module.out_channels,
            module.kernel_size[0],
            module.stride[0],
            module.padding[0],
            module.dilation[0],
            module.groups,
            hasattr(module, "bias"),
            module.padding_mode)
        with torch.no_grad():
            module_output.weight.copy_(module.weight.mean(-1))
            if hasattr(module, "bias"):
                module_output.bias = module.bias
         
    for name, child in module.named_children():
        module_output.add_module(name, convert2DModelTo1D(child))
    del module
    return module_output

This works for efficientnet_b0 so far.

[pfeatherstone]

This works with a lot of CNN models. Maybe the better approach would be to have a flag in create_model() which dictated whether to build a 1D, 2D or 3D model. But for now this is sufficient.

[rwightman]

@pfeatherstone thanks for sharing that snippet, I think initial approach would be to try as a helper that mutates an existing model like above, adding arguments through the create_model fn is a bigger commitement that adds long term maintenance burden if it doesn't end up being used much...

[pfeatherstone]

I have to say, the results were really poor. The better option was to concert my 1D data into 2D by using torch.stft() then using a normal 2D model...

[pfeatherstone]

Tried this again optimistically thinking I would get better results. Nope...