申请试用
HOT
登录
注册
 
Tangram: Distributed Scheduling Framework for Apache Spark at Facebook

Tangram: Distributed Scheduling Framework for Apache Spark at Facebook

Spark开源社区
/
发布于
/
8392
人观看
Tangram is a state-of-art resource allocator and distributed scheduling framework for Spark at Facebook with hierarchical queues and a resource based container abstraction. We support scheduling and resource management for a significant portion of Facebook’s data warehouse and machine learning workloads that equates to running millions of jobs across several clusters with tens of thousands of machines. In this talk, we will describe Tangram’s architecture, discuss Facebook’s need for a custom scheduler, and explain how Tangram schedules Spark workloads at scale. We will specifically focus on several important features around improving Spark’s efficiency, usability and reliability: 1. IO-rebalancer (Tetris) Support 2. User-Fairness Queueing 3. Heuristic-Based Backfill Scheduling Optimizations.
0 点赞
0 收藏
1下载
确认
3秒后跳转登录页面
去登陆