[Data] Additional metrics for tracking block times

[scottjlee]

Description

To gain more insight into internal Ray Data operations, we need metrics with finer granularity and a Histogram-like view (currently all Ray Data metrics are Gauge forms, in order to display the last known value on the Ray Dashboard).

Some other useful metrics: time taken to read the block, deserialize the block, move blocks between workers and generate batches.

• General overview of the stats / metrics code path:
  • op_runtime_metrics.py defines metrics which are observed during dataset execution
  • stats.py contains the classes which collect and summarize stats from runtime metrics.
  • _StatsActor._create_prometheus_metrics_for_execution_metrics() converts the raw metrics from op_runtime_metrics.py into Gauge prometheus-type metrics.
  • Currently, a major restriction is that all of our metrics are defined as Gauge, in order to display the last known value on the ray dashboard.

As an example, let's consider the block generation time, defined as the time taken to generate blocks per task.

• For block generation time, we have an existing metric block_generation_time under op_runtime_metrics.py , which is incremented in on_task_output_generated()
• We can use this existing metric and define a new Histogram metric, which tracks the size/number of events.
• Currently i don't think ray dashboard has a way to display the entire histogram, but should be able to show single-value metrics like avg, p90, etc., if you also want to add a chart.

Use case

No response