Low fill rate (6/8/2024) investigation

[ghost-not-in-the-shell]

The fill rate is around 55 per cent on 06/08/2024.

The investigation is following:

1. According to rejected-blocks.log on seed-1, there're 67 blocks rejected due to invalid time (received 1 block too late) and 629 blocks were rejected due to duplicate producer (this would not caused the fill rate to drop). For those 67 blocks, they are all included in the transition frontier. Those records of rejection is caused by gossips that come too late.
2. Looking at the error logs on seed-1, there're some "valiation timeout". But again, those blocks which were reported to be timeout were also included in the transition frontier.
3. The thread timing metrics for seed-1 didn't show much difference for 6/8.

The conclusion is that everything looks normal on seed node side and we could just have some bad luck on vrf evaluation that the stakes online didn't win those slots. The drop from 60 to 55 could just be a blip.