ETL Made Easy with Azure Data Factory and Azure Databricks

下载 1

Spark开源社区

发布于

8341

人观看

#信息技术

Data Engineers are responsible for data cleansing, prepping, aggregating, and loading analytical data stores, which is often difficult and time-consuming. Azure Data Factory makes this work easy and expedites solution development. We’ll demonstrate how Azure Data Factory can enable a new UI-driven ETL design paradigm on top of Azure Databricks for building scaled-out data transformation pipelines.

展开查看详情

1 .WIFI SSID:SparkAISummit | Password: UnifiedAnalytics

2 .ETL Made Easy with Azure Data Factory & Azure Databricks Mark Kromer Sr. Azure Data Program Manager Microsoft #UnifiedAnalytics #SparkAISummit

3 .Cloud ETL Patterns with ADF Azure Data Factory #UnifiedAnalytics #SparkAISummit 3

4 .Nightly ETL Data Loads Code-free

5 .Slowly Changing Dimension Scenario

6 .Load Star Schema DW Scenario

7 .Data Lake Data Science Scenario

8 .Workflow Data Pipelines/Control Flow Azure Data Factory #UnifiedAnalytics #SparkAISummit 9

9 .

10 .

11 .

12 .Mapping Data Flows Azure Data Factory #UnifiedAnalytics #SparkAISummit 13

13 .What is ADF Mapping Data Flow? • Transform Data, At Scale, in the Cloud, Zero-Code – Cloud-first, scale-out ELT – Code-free dataflow pipelines • Serverless scale-out transformation execution engine • Maximum Productivity for Data Engineers – Does NOT require understanding of Spark / Scala / Python / Java • Resilient Data Transformation Flows – Built for big data scenarios with unstructured data requirements – Operationalize with Data Factory scheduling, control flow and monitoring

14 .Code-free Data Transformation At Scale • Does not require understanding of Spark, Big Data Execution Engines, Clusters, Scala, Python … • Focus on building business logic and data transformation – Data cleansing – Aggregation – Data conversions – Data prep – Data exploration

15 .Transformation Function Expression Language

16 .Debug Data Flows with Data Preview and Data Sampling

17 .Deep Monitoring Introspection of Data Transformations

18 .DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT

0点赞

1收藏

1下载