improve blocks syncing

[tdroxler]

Let's use this issue to list every issues related to improving blocks synchronization

• Optimize main_chain update of new blocks #350
  • Remove double reading a block in BlockFlowSyncService.insert #358
  • Reduce data read by BlockDao.updateMainChainAction #359
• Sync without requesting max timestamp from the node #351
• Make BlockFlowSyncService closer to database #352
• Migrate main chain update to raw sql #155
• Sync using full node's web socket #353
• Heuristic update of finalized blocks #354

[simerplaha]

Hey Thomas, I’ve been wanting to do some performance improvements to the sync process which kinda fits the title of this issue “Improve block syncing” but it does not overlap with the description, so we are not duplicating work, which is good.

I've tried creating a breakdown of tasks for this "performance improvement" but it's getting a bit too much, a lot of documentation. In a very informal way - I’d like to reduce the trips to BlockFlowClient and database to bare necessary and not use polling. Make it cost-effective, run only as fast as hardware resources available/allocated (massively reduced IO + increase/controlled sync speed), batch as much as possible and ensure things are transactional + other stuff, basically just performance improvements.

Yep, I'm not suggesting a very concrete task. Its a bit open-ended. I've just been pondering, haven't started on this yet, will wait for your feedback.

[tdroxler]

Hey @simerplaha I updated this issue to become the "main" issue for all the improvement in the syncing process. I started to create and gather few issues here. Feel free to add more in the description (can you edit it? otherwise post them as a comment).

[simerplaha]

Hey @tdroxler, I'm kinda having a tricky time trying to figure out how to implement any one of these sync related issues. I keep going back and forth with different implementations.

The problem I'm having is that the sync is a bit chunky with one big test-case and no unit test-cases. It's sensitive to any changes I make.

Seems like before working on any of the issues listed above, the sync need to be re-implemented such that each function is small/independent and is unit testable. This would make performance optimisations manageable and these issues can get resolved on the go.

My attempts so far are all sort of moving towards the PR I submitted before #362. If you are ok with general overall shape that PR was taking I can maybe start off small off with a PR just for init() (following what's done in that PR + unit tests)?

Without unit-tests I'm not sure how I can be helpful here.

[tdroxler]

Hi @simerplaha I agree we need better unit tests to be more confident when changing that part.

As you proposed we can start with multiple small PRs where we take one function and add unit tests. Let's try to keep it super simple for now without adding to much things like SyncRemoteAction, I think it's safer if we first split into multiple functions and unit test everything before adding new paradigm.

You can go ahead with the init(), thx a lot.

[simerplaha]

Man for a decent solution I need to create case classes defined in SyncRemoteAction so the functions are declarative and unit testable.

Will see if I can do without it.