郑鲁-类SQL在实时流处理上的探索及应用final

注脚

展开查看详情

1.

2.第七届 全球软件案例研究峰会 AI/ AIOps DevOps AI 2018 11 30 -12 3 | 100+

3. 上海精品公开课 管理3.0认证课程 时间:12月22-23日 | 地点:上海 | 讲师:林伟丹 大数据及AI挖掘技术 时间:12月22-23日 | 地点:上海 | 讲师:风清扬 高可用架构与设计 时间:01月12-13日 | 地点:上海 | 讲师:沈老师 K8S与service mesh 时间:01月12-13日 | 地点:上海 | 讲师:Jim 备注:扫码查看课程详情,两人以上报名有优惠,详情咨询:15802217295

4.Ø Ø Ø Ø Ø

5.01 02 03 04

6.RD • • •

7.SQL

8.01 02 03 & 04

9.l Apache Beam l Apache Samza ü l Apache Apex ü l Apache Flink ü l Apache Strom ü SQL l Apache Strom trident l Apache Spark

10. Native Micro-batching Micro-batching Native At-least-once Exactly-once Exactly-once Exactly-once ACKS RDD based Checkpointing Checkpointing 1 Lineage Chandy-Lamport 2 Checkpoint Metadata operator DStream operator l mapWithState l updateStateBykey l Keyed State l Operator State Process time SQL Spark sql table api SQL java Java/Scala Java/Scala

11.Why Choose Flink? Lambda Kappa

12.Why Choose Flink? Event Time 3 Ingestion Time Kafka Flink Processing Time eg: <<Flink >> l === Event Time l == Ingestion Time l == Processing Time

13.Why Choose Flink? Tumbling Window Sliding Window Session Window ,( 1 ) 1 30 30 1

14.Why Choose Flink? ü Event-driven Appliacations ü Watermark ü Keyed State Operator State ü Checkpoint

15.01 & 02 03 04

16. SQL Ø ETL Flink Mysql Kakfa Eagle OLAP Ø

17.

18.SQL ETL

19.SQL ETL

20.SQL ETL

21. SQL ETL Flink JSONSource POJO Row JSON Table Flink

22. SQL ETL Ø json Ø SQL UDF Ø kafka mysql Eagle SnappyData

23.SQL ETL- 8 7 6 5 4 3 2 1 0 DSL

24.SQL ETL -

25.SQL

26.V1 Flink +siddhi V2: Flink SQL match recognize

27.CEP Complex Event Processing Storm+Esper Storm+avi Apache Flink CEP + ator Eagle esper query matching Storm+Siddhi cpu

28.•Multi-threading •Queues and use of pipelining •Nested queries and chaining streams

29.V1

30.

31.Stream

32.V2

33.Match_Recognize

34.SQL Spark sql SQL Calcite Calcite SQL SQL Logical Tree Spark catalyst Janino Code Physical Flink Table Generation Planning DataSet

35.Stream SQL Table API SQL DataSet DataStream Calls Query Table API Calcite Calcite Validator Catalog Parse & Validator Calcite Logical Plan DataSet Calcite DataStream Opt.Rules Optimizer Opt.Rules DataSet DataStream Plan Plan DataSet DataStream

36.Flink SQL MatchRecognize ü Flink table API match ü match ü Calcite Code Generate

37.

38.01 02 03 & 04

39.

40.Eat Better, Live Better Thanks

41.