申请试用
HOT
登录
注册
 
Dynamic Partition Pruning in Apache Spark
Dynamic Partition Pruning in Apache Spark

Dynamic Partition Pruning in Apache Spark

Spark开源社区
/
发布于
/
4195
人观看

In data analytics frameworks such as Spark it is important to detect and avoid scanning data that is irrelevant to the executed query, an optimization which is known as partition pruning. Dynamic partition pruning occurs when the optimizer is unable to identify at parse time the partitions it has to eliminate. In particular, we consider a star schema which consists of one or multiple fact tables referencing any number of dimension tables. In such join operations, we can prune the partitions the join reads from a fact table by identifying those partitions that result from filtering the dimension tables. In this talk we present a mechanism for performing dynamic partition pruning at runtime by reusing the dimension table broadcast results in hash joins and we show significant improvements for most TPCDS queries.

6 点赞
2 收藏
4下载
确认
3秒后跳转登录页面
去登陆