申请试用
HOT
登录
注册
 
Improving Apache Spark Downscaling
Improving Apache Spark Downscaling

Improving Apache Spark Downscaling

Spark开源社区
/
发布于
/
3606
人观看

As more workloads move to severless-like environments, the importance of properly handling downscaling increases. While recomputing the entire RDD makes sense for dealing with machine failure, if your nodes are more being removed frequently, you can end up in a seemingly loop-like scenario, where you scale down and need to recompute the expensive part of your computation, scale back up, and then need to scale back down again.

Even if you aren’t in a serverless-like environment, preemptable or spot instances can encounter similar issues with large decreases in workers, potentially triggering large recomputes. In this talk, we explore approaches for improving the scale-down experience on open source cluster managers, such as Yarn and Kubernetes-everything from how to schedule jobs to location of blocks and their impact (shuffle and otherwise).

6 点赞
2 收藏
0下载
确认
3秒后跳转登录页面
去登陆