申请试用
HOT
登录
注册
 
Supporting Over a Thousand Custom Hive User Defined Functions

Supporting Over a Thousand Custom Hive User Defined Functions

Spark开源社区
/
发布于
/
8173
人观看
Over the years, Facebook has used Hive as the primary query engine to be used by our data engineers. Since Hive uses SQL-like query language called HQL, the list of built-in User Defined Functions (UDFs) did not always satisfy our customer requirements and as a result, an extensive list of custom UDFs was developed over time. As we started migrating pipelines from Hive to Spark SQL, a number of custom UDFs appeared incompatible with Spark, and many others showed bad performance. In this talk will first take a deep dive into how Hive UDFs work with Spark. We will then share what challenges we overcame on the way to support 99.99% of the custom UDFs in Spark.
0 点赞
0 收藏
1下载
确认
3秒后跳转登录页面
去登陆