申请试用
HOT
登录
注册
 
徐骁_Apache Flink 和 Elasticsearch 助⼒实时 OLAP 平台
Flink China中文社区
/
发布于
/
3226
人观看
Apache Flink 和 Elasticsearch 助⼒实时 OLAP 平台
展开查看详情

1 .Flink Forward China 2018 公司:去哪⼉儿⽹网 职位:实时计算负责⼈人 演讲者:徐骁

2 .Apache Flink 和 Elasticsearch 助⼒力力实时 OLAP 平台 Apache Flink and Elasticsearch power realtime OLAP platform &

3 . ⾃自我介绍 About me 2016 加⼊入 Qunar Joined Qunar in 2016 主要负责: Focus on: ELK ELK Elasticsearch Elasticsearch Kafka Kafka Flink 实时计算平台 Flink platform

4 . ⽬目录 Outline 1. 起因 1. Motivation 2. 如何写⼊入 ES 2. How to sink to ES 3. Flink 和 ES 构建实时 OLAP 平台 3. Flink & ES OLAP platform 4. 实时⽹网络拓拓扑 4. Realtime network topology

5 .1 Motivation

6 . Motivation Log Kafka Logstash Elasticsearch Kibana

7 . Motivation 基于 Lucene 上⼿手容易易, 易易编程 Based on Lucene Easy to start 分⽚片机制 灵活 Sharding Flexible ⾼高可⽤用 伸缩性 Logstash Elasticsearch High available Scalable Kibana Kibana

8 . Motivation 可靠性 High available 性能 Performance Logstash < 5.0 复杂需求 Complex needs

9 . Motivation 可靠性 Exactly Once High available 性能 JVM Performance 复杂需求 Streaming Model Flink Complex needs Stateful Window Failover

10 . Motivation Flink

11 . Motivation Logstash Log Kafka Elasticsearch Kibana Flink

12 .2 How to sink to ES?

13 . How to sink to ES Config ES address ElasticsearchSinkFunction https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/elasticsearch.html

14 . How to sink to ES bulk.flush.max.actions Config bulk.flush.max.size.mb bulk.flush.interval.ms bulk processor

15 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

16 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

17 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend OffsetCommitMode: • DISABLED • ON_CHECNKPOINTS Offset:0 Offset:0 • KAFKA_PERIODIC

18 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

19 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

20 . How to sink to ES Enable checkpoint: At least once bulk.flush.max.actions=2 Kafka Source bulk processor ES flush real offset Zookeeper Statebackend Offset:0 Offset:0

21 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

22 . How to sink to ES Enable checkpoint: At least once Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:0

23 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Trigger checkpoint Kafka Source bulk processor ES barrier real offset Zookeeper Statebackend Offset:0 Offset:0

24 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

25 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator ack Kafka Source bulk processor ES barrier real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

26 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

27 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES barrier flush() snapshotState() real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

28 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

29 . How to sink to ES Enable checkpoint: At least once CheckpointCoordinator ack Kafka Source bulk processor ES real offset Zookeeper Statebackend Offset:0 Offset:5(Pending)

0 点赞
1 收藏
4下载
确认
3秒后跳转登录页面
去登陆