[enhancement] Enhance positional encoding and introduce dynamic multi-scale patching for improved forecasting #232
Killer3048
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I propose the following concise enhancements:
Enhanced Positional Encoding:
Replace the standard sinusoidal positional encodings with relative or learnable encodings—such as Fourier features or RoPE—to better capture the relationships between tokens. This adjustment will allow the model to more effectively learn cyclic and seasonal patterns, independent of absolute positions.
Dynamic/Multi-Scale Patching:
Instead of using a fixed patch size, implement dynamic patching that adapts to local variability, or design a multi-scale framework that processes both small (fine-grained) and large (coarse-grained) patches in parallel. Merging these representations (via an attention-based fusion layer) will enable the model to capture both detailed local fluctuations and broad long-term trends.
These improvements are aimed at enhancing the model’s ability to model complex temporal dependencies,
Beta Was this translation helpful? Give feedback.
All reactions