Spark性能优化：Shuffle性能优化

new SparkConf().set(“spark.shuffle.consolidateFiles”, “true”)web spark.shuffle.consolidateFiles：是否开启shuffle block file的合并，默认为false缓存 spark.reducer.maxSizeInFlight：reduce task的拉取缓存，默认48msvg spark.shuff

>>阅读原文<<