Fix check-migrations / try-runtime job (when downloading too much data/keys)

[bkontur]

Actually, the check-migrations job for AssetHubPolkadot occasionally fails, and this is happening more frequently due to Transport: 429, which likely corresponds to HTTP 429 Too Many Requests.
Example: GitHub Actions Run

With the upcoming Plaza and the increasing shift of functionality to AssetHubPolkadot, optimizing this job is crucial.

Suggested solutions:

1. Implement rate limiting on the client side (e.g., based on the amount of data or time).
2. Make parallel requests to different nodes, for example:
   • Download keys at a specific block from one node.
   • Download data for those keys at the same block from several nodes.
3. Use some small snapshot of the last block (state/db) (refreshed/collected periodically and stored some where, so we just quickly download it here) - (something similar is needed here: Run storage benchmarks before release #491)
4. Investigate and fix issues with the try-runtime CLI, which currently uses HttpClient try-runtime: dynamic storage query sizes paritytech/substrate#13923.
5. Explore alternative ways to download the actual state/database at a specific block.
6. Other optimizations as needed?

This optimization is particularly important because the only current workaround involves making a new commit, which triggers the full CI jobs. This approach wastes CI time unnecessarily, as it's not possible to restart just one particular job (issue.

[bkchr]

Ha. I also looked into this today. The most easiest thing to improve this is to cache the snapshot once per day and make the snapshot downloading retry on error.

[ggwpez]

We actually have snapshot caching in the SDK, could re-use that pipeline.

For the Proxy: I think subway can be used to distribute requests among nodes.

[bkchr]

We actually have snapshot caching in the SDK, could re-use that pipeline.

yeah I looked into it, with some optimizations we can take it.