申请试用
HOT
登录
注册
 
8.魏彬-Apache Pulsar 在日志场景的最佳实践
1 点赞
5 收藏
16下载
示说网官方
/
发布于
/
473
人观看

魏彬,Apache Pulsar Contributor,StreamNative 解决方案工程师

议题简介

ELK+Apache Kafka是一种常见的日志场景的架构。然而,如今情况发生了变化,云原生变得流行,微服务架构被到处采用。这带来了更多的服务,日志数量和类别也越来越多。Apache Kafka不能满足云原生日志场景的所有要求,如操作简单、百万主题管理、租赁资源隔离等。Apache Pulsar是一个更好的解决方案,具有云原生架构和更好的性能。本演讲将与大家分享Apache Pulsar作为一种新的日志消息解决方案,包括对日志消息系统的要求、Kafka与Pulsar解决方案对比、Pulsar最佳实践和Pulsar Functions/连接器介绍。

展开查看详情

1.Apache Pulsar 在⽇志场景的最佳实践 魏彬@StreamNative streamnative.i o

2. StreamNative is the company behind the Pulsar community. streamnative.i o

3.Executive Team ✓ Data veterans with extensive industry experienc Sijie Guo Matteo Merli Jia Zhai ✓ Original creators of Apache Pulsar Founder and CEO CTO Co-Founder & BookKeepe Apache Software ✓ Apache Software Pulsar/BookKeeper PMC Foundation Membe Pulsar/BookKeeper PMC Foundation Membe Operated the largest Pulsar/ Pulsar PMC Chair BookKeeper cluster streamnative.i r r r o e

4. StreamNative: Host of the Pulsar Summits 2020 202 ● 2 Global Pulsar Summit ● 3 Global Pulsar Summit ● 1,600 attendee ○ North America – June 16-1 ● 80+ speaker ○ Europe – Sept. 2 ● Attendees included: ○ Asia – Nov 28-2 ● Speakers from: streamnative.i 1 s o s 9 1 s s 7

5.Apache Pulsar: Github Stars 9000+ 6X Growth Sep 2018: Apache Top-level project streamnative.i o

6.Apache Pulsar: Contributors 400+ 11X Growth Sep 2018: Apache Top-level project streamnative.i o

7.Apache Pulsar: Monthly Active Contributors streamnative.i o

8. 魏彬 (@rockybean) Solution Engineer@StreamNative Elastic Certified Engineer & Analyst 阿⾥云 MVP streamnative.i o

9.⽬录 I. 常⻅⽇志架构(ELK II. ⽇志场景的常⻅挑战 III. 解决⽅案(Apache Pulsar IV. Q&A streamnative.i o ) )

10.I. 常⻅⽇志架构 streamnative.i o

11.常⻅⽇志架构 ELK Elasticsearch Beats Logstash Kafka FILEBEAT Kibana WINGLOGBEAT AUDITBEAT Messaging Queue PACKETBEAT streamnative.i o

12.消息队列在⽇志场景中的作⽤ 削峰解耦 数据分发 streamnative.i o

13.II. ⽇志场景的常⻅挑战 streamnative.i o

14.⽇志场景的常⻅挑战 ⽇志分级 流量不确定性 种类繁多 如何保证 如何实现 如何灵活地适配 核⼼业务⽇志的可⽤性? ⽣产和消费处理能⼒的即时扩容? 复杂多变的⽇志种类? streamnative.i o

15.消息队列在⽇志场景中的更多功能性要求 ⽇志分级 流量不确定性 种类繁多 限速 即时扩缩容能⼒ ⽆限 Topic/Partition Auto Scaling streamnative.i o

16.III. 解决⽅案(Apache Pulsar) streamnative.i o

17.Apache Pulsar • 计算和存储分离的分层架构 • Broker - 计算节点 • Bookie - 存储节点 • 计算层⽆状态,即时扩容 • 存储层具有低延时、⾼吞吐、持久化、强⼀ 致的特性 • 即时故障转移和容错机制 • 天⽣适合 K8S 等容器编排管理平台 streamnative.i o

18.限速 •仅⽀持 user/client-id 级别设定 •namespace/topic 级别 •Broker 级别 streamnative.i o

19.即时扩缩容 •增加新节点后,需要 reassign partition 才能使⽤ •存储与计算分离,可以按需增加计算或存储节点,增加即 •消费能⼒受 partition 数⽬限制,⽽且有 consumer ⽣效,不需要 reassign rebalance 的问题 •消费能⼒不受 partition 数⽬限制,可以不断增加 consumer 来提升消费能⼒ streamnative.i o

20.⽆限 Topic/Partition •随 Partition 数增多,请求延迟下降严重 •⽀持百万级别 topic,请求延迟稳定 •追加写模式退化为随机写 •topic/partition 仅是逻辑概念,保证追加写模式 streamnative.i o

21.⽆限 Topic/Partition https://www.infoq.cn/article/xeyeEeNNY5CG0PGyxeVD streamnative.i o

22.ELK 架构 Filebeat Filebeat Filebeat Filebeat Kafka Logstash Elasticsearch Kibana Grafana Monitoring API streamnative.i o

23.Kafka on Pulsar 兼容 kafka 协议 > 1.0 https://github.com/streamnative/kop streamnative.i o

24.Pulsar 引⼊后的架构 V1 Filebeat Filebeat Filebeat Filebeat 架构不需要变动! kop Apache Pulsar kop * ⽆法使⽤部分 Pulsar 的特性 Logstash * 限速限流 Elasticsearch Kibana Grafana Monitoring API streamnative.i o

25.Pulsar 引⼊后的架构 V2 Filebeat * Filebeat * Filebeat * Filebeat * Filebeat Output Pulsar Plugin * Apache Pulsar Logstash * Logstash/Pulsar Input Plugin Input Pulsar Plugin Elasticsearch Kibana Grafana Monitoring API streamnative.i o

26.Pulsar 引⼊后的架构 V2 ● https://github.com/streamnative/pulsar-beat-outpu ● https://github.com/streamnative/logstash-input-pulsa ● https://github.com/streamnative/logstash-output-pulsar t r

27.Pulsar Function 轻量级计算引擎:ETL、数据丰富、数据过滤、数据脱敏、动态路由、统计分析等 streamnative.i o

28.Pulsar Function 轻量级计算引擎:ETL、数据丰富、数据过滤、数据脱敏、动态路由、统计分析等 streamnative.i o

29.Pulsar Function Function Pipeline -> Function Mesh streamnative.i o

1 点赞
5 收藏
16下载