2019-06-22_ Serverless Event Streaming with Pulsar Functions

● What is Apache Pulsar? ● Event Stream - Pulsar view on Data ● When Event Streaming meets serverless ○ Programming Model ○ Architecture ○ Use cases
展开查看详情

1.Serverless Event Streaming with Pulsar Functions Sijie Guo (@sijieg) 2019.06.22

2. Who am I ● Apache Pulsar PMC Member ● Apache BookKeeper PMC Chair ● Twitter, Yahoo Alumni ● Founder of StreamNative ● Interested in technologies around Event Streaming

3. Agenda ● What is Apache Pulsar? ● Event Stream - Pulsar view on Data ● When Event Streaming meets serverless ○ Programming Model ○ Architecture ○ Use cases

4.What is Apache Pulsar?

5.What is Apache Pulsar? “Flexible pub/sub messaging backed by a durable stream storage”

6.Pulsar - Pub/Sub

7.Pulsar - Multi Tenancy

8.Pulsar - Flexible Messaging

9. Pulsar - Flexible Messaging ● One data, different ways to consume ● Queuing (aka stateless messaging) ○ Shared (* RabbitMQ) ● Streaming (aka stateful messaging) ○ Exclusive ○ Failover (* Kafka) ○ Key_Shared

10.Pulsar - Cloud Native Architecture Layered Architecture ❏ Independent scalability ❏ Instance failure recovery ❏ Balance-free on cluster expansions

11.A Pulsar view on Data

12.Pulsar View - Topic

13.Pulsar View - Partition

14.Pulsar View - Segment

15.Pulsar View - Event Stream

16.Event Stream is the right foundation for your data

17.M(essaging), S(torage), P(rocessing)

18.MSP - Interactive Queries

19.MSP - Stream & Batch Processing

20.MSP - What is next? What is next?

21.When Event Streaming meets Serverless

22.Introduce Pulsar Functions

23. Pulsar Functions ● A serverless event streaming framework ● Lightweight computation ● Event-first, Stream-first ● Multi languages ● Multi runtimes ● SDK-less & SDK

24. Function Elements ● Input Topics ● Output Topics ● Function ● State ● Log Topics

25.API - Native Java / Python / Go Function Java Python Golang

26. API - Function Context ● Logger ● State ● Metrics ● Security / Secrets ● ...

27.Context - Logger

28.Context - State

29. Context - State ● Global Managed State * ● Mutable by functions and admin-cli ● Queryable by functions and admin-cli ● State are stored at storage layer ● State are implemented using streams + snapshot

StreamNative 是一家围绕 Apache Pulsar 和 Apache BookKeeper 打造下一代流数据平台的开源基础软件公司。秉承 Event Streaming 是大数据的未来基石、开源是基础软件的未来这两个理念,专注于开源生态和社区的构建,致力于前沿技术。