展开查看详情
1 . Continuous Applications
at the Scale of 100 Teams
with Databricks Delta
and Structured Streaming
Viacheslav Inozemtsev
Max Schultze
April 25, 2019
2 .OUTLINE
Introduction of Zalando
Zalando’s Processing Platform
Databricks Use Cases
Lessons Learned
3 . Who we are
Viacheslav Inozemtsev Max Schultze
● Data Engineer ● Data Engineer
● Degrees in Applied Math and in ● MSc in Computer
Computer Science Science
● Working with Spark since 0.9.2 ● Took part in early
development of Apache Flink
3
4 .Introduction of Zalando
4
5 . Zalando’s Data Lake
Ingestion Serving
Storage
5
6 . Zalando’s Data Lake
Ingestion Serving
Data Center Storage
DWH
Event Bus
6
7 . Zalando’s Data Lake
Ingestion Serving
Data Center Storage
DWH
Event Bus
Metastore
7
8 . Zalando’s Data Lake
Ingestion Serving
Ad-Hoc querying
Data Center Storage
DWH
Event Bus
Metastore
8
9 . Zalando’s Data Lake
Ingestion Serving
Ad-Hoc Querying
Data Center Storage
DWH
Processing Platform
Event Bus
Metastore
9
10 . Zalando’s Data Lake
Ingestion Serving
Ad-Hoc Querying
Data Center Storage
DWH
Processing Platform
Event Bus
Metastore
10
11 .Zalando’s Databricks
Processing Platform
11
12 . Zalando’s Databricks Processing Platform - Technical Setup
12
13 . Zalando’s Databricks Processing Platform - Technical Setup
13
14 . Zalando’s Databricks Processing Platform - Technical Setup
14
15 . Zalando’s Databricks Processing Platform - Technical Setup
15
16 . Zalando’s Databricks Processing Platform - Technical Setup
16
17 . Zalando’s Databricks Processing Platform - Organizational Setup
Introduction to Databricks
● RSA
● Office Hours
17
18 . Zalando’s Databricks Processing Platform - Organizational Setup
Introduction to Databricks Initial Setup
● RSA ● Inner Source
● Office Hours Configuration
18
19 . Zalando’s Databricks Processing Platform - Organizational Setup
Introduction to Databricks Initial Setup
● RSA ● Inner Source
● Office Hours Configuration
Development Phase
● Office Hours
● Guest Developer
19
20 . Zalando’s Databricks Processing Platform - Organizational Setup
Introduction to Databricks Initial Setup
● RSA ● Inner Source
● Office Hours Configuration
Development Phase Productionizing
● Office Hours ● 24/7 Support
● Guest Developer
20
21 .Databricks Use Cases
21
22 . Batch Ingestion from Data Warehouse
Ingestion Serving
Ad-Hoc Querying
Data Center Storage
DWH
Processing Platform
Event Bus
Metastore
22
23 . Batch Ingestion from Data Warehouse
23
24 . Batch Ingestion from Data Warehouse
● Problem 1: extraction from databases via JDBC can be slow
24
25 . Batch Ingestion from Data Warehouse
● Problem 1: extraction from databases via JDBC can be slow
● Solution:
○ use parallelism of Spark JDBC reader
25
26 . Batch Ingestion from Data Warehouse
● Problem 1: extraction from databases via JDBC can be slow
● Solution:
○ use parallelism of Spark JDBC reader
○ for partitioned tables a view with a column PARTITION_ID can be
created
26
27 . Batch Ingestion from Data Warehouse
● Problem 1: extraction from databases via JDBC can be slow
● Solution:
○ use parallelism of Spark JDBC reader
○ for partitioned tables a view with a column PARTITION_ID can be
created
○ works especially well for tables partitioned on multiple machines
27
28 . Batch Ingestion from Data Warehouse
● Problem 2: data warehouse is still often on premises
28
29 . Batch Ingestion from Data Warehouse
● Problem 2: data warehouse is still often on premises
● Solution:
○ resolve this early!
29
