徐骁_Apache Flink 和 Elasticsearch 助⼒实时 OLAP 平台

Apache Flink 和 Elasticsearch 助⼒实时 OLAP 平台
展开查看详情

1.Flink Forward China 2018 公司:去哪⼉儿⽹网 职位:实时计算负责⼈人 演讲者:徐骁

2.Apache Flink 和 Elasticsearch 助⼒力力实时 OLAP 平台 Apache Flink and Elasticsearch power realtime OLAP platform &

3. ⾃自我介绍 About me 2016 加⼊入 Qunar Joined Qunar in 2016 主要负责: Focus on: ELK ELK Elasticsearch Elasticsearch Kafka Kafka Flink 实时计算平台 Flink platform

4. ⽬目录 Outline 1. 起因 1. Motivation 2. 如何写⼊入 ES 2. How to sink to ES 3. Flink 和 ES 构建实时 OLAP 平台 3. Flink & ES OLAP platform 4. 实时⽹网络拓拓扑 4. Realtime network topology

5.1 Motivation

6. Motivation Log Kafka Logstash Elasticsearch Kibana

7. Motivation 基于 Lucene 上⼿手容易易, 易易编程 Based on Lucene Easy to start 分⽚片机制 灵活 Sharding Flexible ⾼高可⽤用 伸缩性 Logstash Elasticsearch High available Scalable Kibana Kibana

8. Motivation 可靠性 High available 性能 Performance Logstash < 5.0 复杂需求 Complex needs

9. Motivation 可靠性 Exactly Once High available 性能 JVM Performance 复杂需求 Streaming Model Flink Complex needs Stateful Window Failover

10. Motivation Flink

11. Motivation Logstash Log Kafka Elasticsearch Kibana Flink

12.2 How to sink to ES?

13. How to sink to ES Config ES address ElasticsearchSinkFunction https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/elasticsearch.html

14. How to sink to ES bulk.flush.max.actions Config bulk.flush.max.size.mb bulk.flush.interval.ms bulk processor

15. How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

16. How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

17. How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend OffsetCommitMode: • DISABLED • ON_CHECNKPOINTS Offset:0 Offset:0 • KAFKA_PERIODIC

18. How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

19. How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

20. How to sink to ES Enable checkpoint: At least once bulk.flush.max.actions=2 Kafka Source bulk processor ES flush real offset Zookeeper Statebackend Offset:0 Offset:0

21. How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

22. How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

23. How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Trigger checkpoint Kafka Source bulk processor ES barrier real offset Zookeeper Statebackend Offset:0 Offset:0

24. How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

25. How to sink to ES Enable checkpoint: At least once CheckpointCoordinator ack Kafka Source bulk processor ES barrier real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

26. How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

27. How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier flush() snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

28. How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

29. How to sink to ES Enable checkpoint: At least once CheckpointCoordinator ack Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)