Repository navigation
[FEATURE] Collect the shuffle reader different phase times #2569
Description
Activity
Based on my observation of many Spark jobs, decompression operations contribute to the majority of the time spent in the reader phase. However, the current shuffle read fetchWaitTime metric only captures the time spent on data fetching and does not reflect the actual read duration, even though this behavior is consistent with Spark’s current design.
Given this, I believe we should introduce clearer and more comprehensive metrics to capture the different stages of the reader phase, like shuffle write phase has did.
Hey!! I'll be happy to help in this one!
Hey!! I'll be happy to help in this one!
Please go ahead.
Reacted by SalvadorRomoHI! @zuston, i found this snippet, were you refering to this incFetchWaitTime? and did you meant something like enhance suffleReadMetrics in order to capture fetch, decompression and finish metrics?
to capture fetch, decompression and finish metrics
That makes sense, but I think these metrics are more appropriate to be part of Uniffle’s own shuffle read metrics, rather than being integrated into Spark’s ShuffleReadMetrics. You could refer to the uniffle's
TaskShuffleWriteTimesReacted by SalvadorRomo@zuston I saw how
TaskShuffleWriteTimesworks, correct me if I wrong, it sends writes statistics to Coordinator server with gRPC once its has finished and this statistic are sent fromRssShuffleWriter. Similarly,TaskShuffleReadMetricin RssShuffleReader. ButRssShuffleDataIteratorit's another phase in the reading process that iteratively gets blocks processed from Uniffle Server. But here, we are not tracking fecthWaitTime and more important metrics to Coordinator, so proposal here is maybe enable another grpc method, to send this metric the same way as the other two?I saw how TaskShuffleWriteTimes works, correct me if I wrong, it sends writes statistics to Coordinator server with gRPC once its has finished and this statistic are sent fromRssShuffleWriter. Similarly, TaskShuffleReadMetric in RssShuffleReader. But RssShuffleDataIterator it's another phase in the reading process that iteratively gets blocks processed from Uniffle Server. But here, we are not tracking fecthWaitTime and more important metrics to Coordinator, so proposal here is maybe enable another grpc method, to send this metric the same way as the other two?
The understanding is inaccurate.
The shuffle write times will be collected in the spark executor shuffleWriter side, and then will be sent to the spark driver's UniffleShuffleManager, and then all the metrics will be aggregated in the
UniffleListener, after that, the metrics will be shown in the spark Uniffle UI tab, like the following screenshot.
And so, for the shuffle read times, I hope the following phase could be collected
- Uncompression times (this may be as the majority part of the whole progress)
- Deser time
- Fetch wait time
- buffer copy times
- ...
The shuffle read times metrics aggregation logic is like the write times, so you can follow the same style. BTW, the grpc transfer could be reused by
ShuffleManagerGrpcService.reportShuffleReadMetric, the thing you should do is that to extend the existing protobuf definition .Feel free to reach me if you have any questions @SalvadorRomo
Reacted by SalvadorRomo@zuston ok, yes i do have another question, how do you enable the Uniffle tab in spark shell? Im deploying the overall solution using docker, according to the deploy folder.
@zuston ok, yes i do have another question, how do you enable the Uniffle tab in spark shell? Im deploying the overall solution using docker, according to the deploy folder.
Please see https://github.com/apache/uniffle/blob/master/docs/client_guide/spark_client_guide.md#spark-ui
Reacted by SalvadorRomoI think I will take this for the read performance improvement, there will be more small tasks to do, if you have interest on this, please feel free to tell me
Code of Conduct
Search before asking
Describe the feature
As title described
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?