Serverless Event Streaming with Pulsar Functions

Pulsar 如何基于 serverless 计算架构提供强大的计算能力。
作为 Pulsar 的计算组件,Pulsar Functions 是消息平台和计算平台在 serverless 方向上的融合和创新。通过支持多种语言和运行环境,为用户编写、运行和部署功能提供了极大的灵活性,让构建消息触发式流平台变得简单。

展开查看详情

1.Serverless Event Streaming with Pulsar Functions Xiaolong Ran (@wolf4j1) 09/12/2019

2. Who am I ● Go programmer ● PingCAP > Bitmain > StreamNative ● Pulsar committer ○ pulsar-client-go ○ Go Functions ○ … ● https://github.com/wolfstudy

3. Agenda ● What is Apache Pulsar? ● Event Stream - Pulsar view on Data ● When Event Streaming meets serverless ○ Programming Model ○ Architecture ○ Use cases

4. Agenda ● What is Apache Pulsar? ● Event Stream - Pulsar view on Data ● When Event Streaming meets serverless ○ Programming Model ○ Architecture ○ Use cases

5.A Pulsar view on Data

6.Pulsar View - Topic

7.Pulsar View - Partition

8.Pulsar View - Segment

9.Pulsar View - Event Stream

10.Event Stream is the right foundation for your data

11.M(essaging), S(torage), P(rocessing)

12.MSP - Interactive Queries

13.MSP - Stream & Batch Processing

14.MSP - What is next? What is next?

15.When Event Streaming meets Serverless

16.

17.Introduce Pulsar Functions

18. Pulsar Functions ● A serverless framework for data processing ● Lightweight computation ● Event-first, Stream-first ● Support both stateless and stateful computation ● Multi languages ● Multi runtimes ● SDK-less & SDK

19. Function Elements ● Input Topics ● Output Topics ● Function ● State ● Log Topics

20.API - Native Java / Python / Go Function Java Python Golang

21. API - Function Context ● Logger ● State ● Metrics ● Security / Secrets ● ...

22.Context - Logger

23.Context - State

24. Context - State ● Global Managed State * ● Mutable by functions and admin-cli ● Queryable by functions and admin-cli ● State are stored at storage layer ● State are implemented using streams + snapshot

25. Context - State API ● Key/Value State API ○ putState ○ getState ● Counter State API ○ getCounter ○ incrCounter

26. Context - Metrics ● API - recordMetric(String metricName, double value) ● Exposed in prometheus format ● Collected by prometheus

27. Flexible Runtime ● Colocate with Broker - Thread & Process ● Managed Function Workers - Thread & Process ● External Schedulers - Container ○ Kubernetes

28.Colocate with Brokers

29.Managed Function Workers