Implement Data Collection and Visualization for Web3.Storage Measurement Batch

[pyropy]

Description:

We need to collect and visualize the size of measurement batches submitted to web3.storage by spark-publish. Existing metric publish is already being sent to InfluxDB here and should be simply extended by adding new point. The collected data should be sent to InfluxDB for storage and further analysis.

Task Breakdown:

1. Data Collection:

   • Capture the size of each measurement batch submitted to web3.storage by spark-publish.
   • Extend existing InfluxDB "publish" metric with new point that captures batch size.

2. Data Visualization:

   • Create a chart to display size of measurement batches over time.
   • Create a chart to display size of measurement batches per round.

Acceptance Criteria:

• The system should accurately capture and store the size of each measurement batch submitted to web3.storage.
• The visualization dashboard should provide real-time insights into the measurement batch sizes.

[Winter-Soren]

Hey @pyropy, I would love to contribute to this issue. This is very interesting. Can you assign this issue to me?

[pyropy]

Sure thing @Winter-Soren ! I'm afraid that you wouldn't be able to visualise the data yourself since that is supposed to be done in our internal Grafana dashboard but I can help you with that part.

I'm going to assinge the issue to you and If there's anything that you would need help with, let me know.

[Winter-Soren]

Sure thing @Winter-Soren ! I'm afraid that you wouldn't be able to visualise the data yourself since that is supposed to be done in our internal Grafana dashboard but I can help you with that part.

I'm going to assinge the issue to you and If there's anything that you would need help with, let me know.

Hey @pyropy,

Thanks for assigning me the issue! I appreciate the heads-up about the visualization part—I'll focus on the data collection and storage aspects first. Further I'll jump on visualisation part. :)