申请试用
HOT
登录
注册
 
Function Mesh
0 点赞
0 收藏
4下载
StreamNative
/
发布于
/
220
人观看

由 StreamNative 翟佳为大家带来「Apache Pulsar 2021 规划与展望」主题分享。

展开查看详情

1.Function Mesh Complex Streaming Jobs In A Simple Way Neng Lu

2.Data Processing With Apache Pulsar

3.Pulsar Functions Pulsar Functions are lightweight compute processes that: ● consume messages from Pulsar topics ● apply a user-supplied processing logic to each message ● publish results to another Pulsar topic

4.Pulsar Functions Overview

5.Pulsar Functions IS & IS NOT Pulsar Functions IS NOT: ● Another Full-Power Streaming Processing Engine ● A New Computation Abstraction Layer Pulsar Functions IS: ● Lambda-style functions that are specifically designed to integrate with Pulsar

6.Pulsar Functions Use Case ● ETL Jobs ● Real-time Aggregation ● Microservices ● Reactive Services ● Event Routing ● ...

7.Pulsar Functions API Hello Pulsar Function Hello!

8.Pulsar Functions Semantics ● ATMOST_ONCE ○ Message is ACKed to Pulsar once received ● ATLEAST_ONCE ○ Message is ACKed to Pulsar after the function completes -- Default ● EFFECTIVELY_ONCE ○ Utilizes Pulsar’s Effectively Once Semantics

9.Pulsar Functions State ● Built-In State Management ○ Provide `Context` object for users to access State ○ Stores state in Bookkeeper ○ Support Server-Side Operations like Counters

10.Pulsar Functions CLI

11.Pulsar Functions Summary ● Developer productivity ○ Intuitive API: `func apply(input) output {}` ○ Multiple Language Support: Java, Python, Golang ● Operational simplicity ○ Fully Integrated with Pulsar ○ No Extra System/Service Setup Needed ● Easy troubleshooting ○ Convenient Local Runtime ○ Easy to Use log topics

12.Function Mesh Function Mesh is a collection of functions collaborate together to: ● Accomplish a final data processing goal ● With clearly defined stages Function Mesh IS NOT full power Streaming Engines

13.Function Mesh Pulsar Pulsar Function Function 3 5 Pulsar Function 1 Pulsar Function 4 Pulsar Function 6 Pulsar Function 2

14.Function Mesh -- BUT �� ● Redundant to manage ● Hard to track as an integrity ● Really difficulty to know upstream/downstream functions

15.Function Mesh Function Mesh Pulsar Pulsar Function Function 3 5 Pulsar Function 1 Pulsar Function 4 Pulsar Function 6 Pulsar Function 2

16.Pulsar Native Function Mesh PIP-66 🙂

17.Function Mesh -- Scheduling

18.Kubernetes Function Mesh 😀

19.Kubernetes Function Mesh - Custom Resource - Function - Mesh - Source - Sink

20.Kubernetes Function Mesh -- Scheduling

21.Pulsar v.s. Kubernetes ● Utilizes Kubernetes’ Scheduling Power ● Function as a First Class Citizen in Cloud Environment ● Open the potential talking to different messaging system

22.Demo

23.Function & Function Mesh

24.Connector - Source

25.Future Planning ● Cloud Native Support ● Self-Contained Function Runtime ● Function registry for reusing function unit ● Better Tools/Frontend to Manage & Inspect Function Meshes ● Smartly Group Function Unit Together If No Shuffle Used ● Auto-scaling based on Pulsar metrics

26.Thank You!

0 点赞
0 收藏
4下载