[FEATURE] Collect the shuffle reader different phase times

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

As title described

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zuston]

Based on my observation of many Spark jobs, decompression operations contribute to the majority of the time spent in the reader phase. However, the current shuffle read fetchWaitTime metric only captures the time spent on data fetching and does not reflect the actual read duration, even though this behavior is consistent with Spark’s current design.

Given this, I believe we should introduce clearer and more comprehensive metrics to capture the different stages of the reader phase, like shuffle write phase has did.

[SalvadorRomo]

Hey!! I'll be happy to help in this one!

[zuston]

Hey!! I'll be happy to help in this one!

Please go ahead.

[SalvadorRomo]

HI! @zuston, i found this snippet, were you refering to this incFetchWaitTime? and did you meant something like enhance suffleReadMetrics in order to capture fetch, decompression and finish metrics?

[zuston]

to capture fetch, decompression and finish metrics

That makes sense, but I think these metrics are more appropriate to be part of Uniffle’s own shuffle read metrics, rather than being integrated into Spark’s ShuffleReadMetrics. You could refer to the uniffle's TaskShuffleWriteTimes

[SalvadorRomo]

@zuston I saw how TaskShuffleWriteTimes works, correct me if I wrong, it sends writes statistics to Coordinator server with gRPC once its has finished and this statistic are sent fromRssShuffleWriter. Similarly, TaskShuffleReadMetric in RssShuffleReader. But RssShuffleDataIterator it's another phase in the reading process that iteratively gets blocks processed from Uniffle Server. But here, we are not tracking fecthWaitTime and more important metrics to Coordinator, so proposal here is maybe enable another grpc method, to send this metric the same way as the other two?

[zuston]

I saw how TaskShuffleWriteTimes works, correct me if I wrong, it sends writes statistics to Coordinator server with gRPC once its has finished and this statistic are sent fromRssShuffleWriter. Similarly, TaskShuffleReadMetric in RssShuffleReader. But RssShuffleDataIterator it's another phase in the reading process that iteratively gets blocks processed from Uniffle Server. But here, we are not tracking fecthWaitTime and more important metrics to Coordinator, so proposal here is maybe enable another grpc method, to send this metric the same way as the other two?

The understanding is inaccurate.

The shuffle write times will be collected in the spark executor shuffleWriter side, and then will be sent to the spark driver's UniffleShuffleManager, and then all the metrics will be aggregated in the UniffleListener, after that, the metrics will be shown in the spark Uniffle UI tab, like the following screenshot.

And so, for the shuffle read times, I hope the following phase could be collected

1. Uncompression times (this may be as the majority part of the whole progress)
2. Deser time
3. Fetch wait time
4. buffer copy times
5. ...

The shuffle read times metrics aggregation logic is like the write times, so you can follow the same style. BTW, the grpc transfer could be reused by ShuffleManagerGrpcService.reportShuffleReadMetric , the thing you should do is that to extend the existing protobuf definition .

Feel free to reach me if you have any questions @SalvadorRomo

[SalvadorRomo]

@zuston ok, yes i do have another question, how do you enable the Uniffle tab in spark shell? Im deploying the overall solution using docker, according to the deploy folder.

[zuston]

@zuston ok, yes i do have another question, how do you enable the Uniffle tab in spark shell? Im deploying the overall solution using docker, according to the deploy folder.

Please see https://github.com/apache/uniffle/blob/master/docs/client_guide/spark_client_guide.md#spark-ui

[zuston]

I think I will take this for the read performance improvement, there will be more small tasks to do, if you have interest on this, please feel free to tell me