Repository navigation
[FEATURE] Dedicated faster serialization when shuffle writing/reading #2596
Description
Activity
cc @jerqi
cc @zhengchenyu
- added a commit that references this issue
on Aug 28, 2025 - linked a pull request that will close this issue[#2596] feat(spark): Introduce fory serializer #2597
on Aug 28, 2025
Code of Conduct
Search before asking
Describe the feature
After overlapping compression is applied, the majority of shuffle write times is the seralization. Is it possible to replace the kyro serialization with the fast implementation like apache fory? It looks it has a impressive benchmark result .
https://github.com/apache/fory/blob/main/docs/benchmarks/serialization/bench_serialize_STRUCT2_to_array_tps.png
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?