使用Delta Lake构建批流一体数据仓库

1 . Delta Lake yuanjian.li@databricks.com 2019.11.30 

2 .

3 .Index • Delta Lake • Delta Lake • Live Demo 

4 . Data Lake Data Lake 

5 . Real-time Streaming, Data Science and ML • Recommendation Engines • Risk, Fraud, & Intrusion Detection • Customer Analytics IoT & Predictive Maintenance Data Lake • • Genomics & DNA Sequencing 

6 . Real-time Streaming, Data Science and ML • Recommendation Engines • Risk, Fraud, & Intrusion Detection • Customer Analytics IoT & Predictive Maintenance Data Lake • • Genomics & DNA Sequencing 

7 . Data Lake Events ? Streaming Analytics Data Lake AI & Reporting 

8 . Data Lake Events Streaming Analytics Data Lake AI & Reporting 

9 . Data Lake #1 λ-arch 1 λ-arch Events 1 1 λ-arch Streaming Analytics Data Lake AI & Reporting 

10 . Data Lake #2 λ-arch 1 λ-arch Events 1 2 Validation 1 λ-arch Streaming Analytics 2 Validation Data Lake AI & Reporting 

11 . Data Lake #3 λ-arch 1 λ-arch Events 1 2 Validation 1 λ-arch Streaming 3 Reprocessing Analytics 2 Validation Partitioned 3 Reprocessing Data Lake AI & Reporting 

12 . Data Lake #4 λ-arch 1 λ-arch Events 1 2 Validation 1 λ-arch Streaming 3 Reprocessing Analytics 4 Updates 2 Validation Partitioned 3 4 Scheduled to Avoid Modifications Reprocessing Data Lake 4 UPDATE & AI & Reporting MERGE 

13 . Data Lake • • • • 

14 . Kinesis CSV, ? AI & Reporting JSON, TXT… Data Lake 

15 . Kinesis CSV, ? AI & Reporting JSON, TXT… Data Lake 1. Reader Writer 

16 . Kinesis CSV, ? AI & Reporting JSON, TXT… Data Lake 1. Reader Writer 2. Spark 

17 . Kinesis CSV, ? AI & Reporting JSON, TXT… Data Lake 1. Reader Writer 2. Spark 3. Time Travel 

18 . Kinesis CSV, ? AI & Reporting JSON, TXT… Data Lake 1. Reader Writer 2. Spark 3. Time Travel 4. 

19 . Kinesis CSV, JSON, TXT… AI & Reporting Data Lake 1. Reader Writer 2. Spark 3. Time Travel 4. 5. 

20 .Delta Lake Streaming Kinesis Analytics CSV, JSON, TXT… Data Lake AI & Reporting 

21 .