TiDE dense skip connections support

[dbalabka]

Description

Nixtla's TiDE implementation only implements residual skip connections. Utilizing the dense skip connection can potentially enhance the forecasting quality metrics.

Use case

Minor model enhancement can improve the forecast metrics.

[elephaint]

Maybe I misunderstand, but if I look at the architecture and the paper the model has been implemented as in the paper which we generally stick to.

We use a residual block where they are in the architecture (in the dense encoders), and a simple linear block for the global skip connection. Note sure where you'd want to add a dense skip connection?

[github-actions]

This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one.

[dbalabka]

@elephaint, you're right. However, based on experiments, it can improve metrics slightly. Still, w/o the ablation study, there is no evidence that dense skip connection provides any benefit. Let's keep this ticket closed. I'll get back to it with the PR (I've already implemented it) and experiments that should prove my thoughts before making any changes.