Function Mesh

1 .Function Mesh Complex Streaming Jobs In A Simple Way Neng Lu 

2 .Data Processing With Apache Pulsar 

3 .Pulsar Functions Pulsar Functions are lightweight compute processes that: ● consume messages from Pulsar topics ● apply a user-supplied processing logic to each message ● publish results to another Pulsar topic 

4 .Pulsar Functions Overview 

5 .Pulsar Functions IS & IS NOT Pulsar Functions IS NOT: ● Another Full-Power Streaming Processing Engine ● A New Computation Abstraction Layer Pulsar Functions IS: ● Lambda-style functions that are specifically designed to integrate with Pulsar 

6 .Pulsar Functions Use Case ● ETL Jobs ● Real-time Aggregation ● Microservices ● Reactive Services ● Event Routing ● ... 

7 .Pulsar Functions API Hello Pulsar Function Hello! 

8 .Pulsar Functions Semantics ● ATMOST_ONCE ○ Message is ACKed to Pulsar once received ● ATLEAST_ONCE ○ Message is ACKed to Pulsar after the function completes -- Default ● EFFECTIVELY_ONCE ○ Utilizes Pulsar’s Effectively Once Semantics 

9 .Pulsar Functions State ● Built-In State Management ○ Provide `Context` object for users to access State ○ Stores state in Bookkeeper ○ Support Server-Side Operations like Counters 

10 .Pulsar Functions CLI 

11 .Pulsar Functions Summary ● Developer productivity ○ Intuitive API: `func apply(input) output {}` ○ Multiple Language Support: Java, Python, Golang ● Operational simplicity ○ Fully Integrated with Pulsar ○ No Extra System/Service Setup Needed ● Easy troubleshooting ○ Convenient Local Runtime ○ Easy to Use log topics 

12 .Function Mesh Function Mesh is a collection of functions collaborate together to: ● Accomplish a final data processing goal ● With clearly defined stages Function Mesh IS NOT full power Streaming Engines 

13 .Function Mesh Pulsar Pulsar Function Function 3 5 Pulsar Function 1 Pulsar Function 4 Pulsar Function 6 Pulsar Function 2 

14 .Function Mesh -- BUT �� ● Redundant to manage ● Hard to track as an integrity ● Really difficulty to know upstream/downstream functions 

15 .Function Mesh Function Mesh Pulsar Pulsar Function Function 3 5 Pulsar Function 1 Pulsar Function 4 Pulsar Function 6 Pulsar Function 2 

16 .Pulsar Native Function Mesh PIP-66 🙂 

17 .Function Mesh -- Scheduling 

18 .Kubernetes Function Mesh 😀 

19 .Kubernetes Function Mesh - Custom Resource - Function - Mesh - Source - Sink 

20 .Kubernetes Function Mesh -- Scheduling 

21 .Pulsar v.s. Kubernetes ● Utilizes Kubernetes’ Scheduling Power ● Function as a First Class Citizen in Cloud Environment ● Open the potential talking to different messaging system 

22 .Demo 

23 .Function & Function Mesh 

24 .Connector - Source 

25 .Future Planning ● Cloud Native Support ● Self-Contained Function Runtime ● Function registry for reusing function unit ● Better Tools/Frontend to Manage & Inspect Function Meshes ● Smartly Group Function Unit Together If No Shuffle Used ● Auto-scaling based on Pulsar metrics 

26 .Thank You!