Apache Flink+Apache Kafka在Vipkid的实践

介绍了vipkid的大数据技术架构,以及Flink在其中的位置,并对Flink的基本功能,生产实践中的运维指标和使用方式进行案例分析,并对其中碰到的技术问题和解决方式进行总结梳理。
展开查看详情

1.Apache Kafka x Apache Flink·

2.

3.

4.

5.

6. Yarn Kafka HDFS Mysql Hadoop Spark Tidb Hbase ES Flink Kafka

7.l l l l

8.l l

9.l l

10.

11.l l

12.l l l l l

13.l l

14. PROCESS SINK MYSQL LOG KAFKA SOURCE DB MQ FLINK ES DATA HDFS

15.

16.

17.

18.

19.{"id":1,"stu":123,"status":"book"} (true,123,1) ---------- INSERT {"id":2,"stu":123,"status":"book"} {"id":1,"stu":123,"status":"finish"} (false,123,1) ---------- DELETE UPDATE {"id":2,"stu":123,"status":"finish"} (true,123,2) ---------- INSERT

20.

21.

22.{"id":1,"stu":123,"status":"book"} (true,123,1) ---------- INSERT {"id":2,"stu":123,"status":"book"} {"id":1,"stu":123,"status":"finish"} {"id":2,"stu":123,"status":"finish"} (false,123,1) ---------- DELETE UPDATE (true,123,2) ---------- INSERT (false,123,2) ---------- DELETE (true,123,1) ---------- INSERT UPDATE (false,123,1) ---------- DELETE DELETE

23.

24.

25.