https://www.infoq.com InfoQ MapReduce News feed https://www.infoq.com/news/2022/08/uber-shuffle-spark/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=MapReduce-news <img src="https://res.infoq.com/news/2022/08/uber-shuffle-spark/en/headerimage/generatedHeaderImage-1660451681157.jpg"/><p>Uber engineering has recently open-sourced its highly scalable and reliable shuffle as a service for Apache Spark. Spark is one of the most important tools and platforms in data engineering and analytics. It is shuffling data on local machines by default and causes challenges while the scale is getting very large. Shuffle as a service is a solution developed at Uber for this problem.</p> <i>By Reza Rahimi</i> MapReduce Data Warehousing Uber Apache Spark Architecture & Design AI, ML & Data Engineering news Sun, 14 Aug 2022 10:05:00 GMT https://www.infoq.com/news/2022/08/uber-shuffle-spark/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=MapReduce-news Reza Rahimi 2022-08-14T10:05:00Z /news/2022/08/uber-shuffle-spark/en