申请试用
HOT
登录
注册
 
TGIP-CN-018: Pulsar Functions Deep Dive
StreamNative
/
发布于
/
1091
人观看
展开查看详情

1 . © 2020 SPLUNK INC. Pulsar Functions A Deep Dive | Pulsar Summit 2020 Sanjeev Kulkarni sanjeevk@splunk.com

2 . © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Agenda Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work

3 . © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Agenda Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work

4 . © 2020 SPLUNK INC. Pulsar Functions:- A Brief Introduction Core Concept Bringing Serverless concepts to the Abstract View streaming world. Execute processing logic per message on input topic Function output goes to an output topic • Optional

5 . © 2020 SPLUNK INC. Pulsar Functions:- A Brief Introduction Simple API Emphasis on simplicity SDK-less API Great for 90% use-cases on streams • Filtering import java.util.function.Function; public class ExclamationFunction implements Function<String, String> { • Routing @Override • Enrichment public String apply(String input) { return input + "!"; } } Not meant to replace Spark/Flink

6 . © 2020 SPLUNK INC. Pulsar Functions:- A Brief Introduction Function lifecycle Flexible execution environments • Pulsar managed – Thread – Process • Externally managed – Kubernetes CRUD based Rest API

7 . © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Agenda Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work

8 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function Representation Submit to any worker FunctionConfig public class FunctionConfig { Json repr of FunctionConfig private String tenant; private String namespace; • tenant/namespace/name private String name; • Input/Output private String className; private Collection<String> inputs; • configs private String output; • lot more knobs …. private ProcessingGuarantees processingGuarantees; private Map<String, Object> userConfig; private Map<String, Object> secrets; private Integer parallelism; Function Code private Resources resources; • jars/.py/zip/etc ... }

9 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Submission Checks AuthN/AuthZ checks FunctionMetaData FunctionConfig validation message FunctionMetaData { • missing parameters FunctionDetails functionDetails; • Incorrect parameters PackageLocationMetaData packageLocation; • Local Configs uint64 version; uint64 createTime; map<int32, FunctionState> instanceStates; Function Code Validation FunctionAuthenticationSpec functionAuthSpec; • class presence, etc } Copy Code to Bookeeper

10 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager Function MetaData Manager System of record Stores all Functions • map from <FQFN, FunctionMetaData> foo -> {functionDetails : {...}, MetaData Topic Tailer version: 2, …} FQFN:- Fully Qualified Function Name Backed by Pulsar Topic MetaData Topic • Function MetaData Topic Contains a MetaData Topic Tailer

11 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- Update State Machine Just before Function creation/update/delete foo -> {functionDetails : {...}, MetaData Topic Tailer version: 2, …} MetaData Topic

12 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- Update State Machine Make a copy of the current state foo -> {functionDetails : {...}, version: 2, …} foo -> {functionDetails : {...}, MetaData Topic Tailer version: 2, …} MetaData Topic

13 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- Update State Machine Make a copy of the current state foo -> {functionDetails : {......}, version: 2, …} Merge the updates foo -> {functionDetails : {...}, MetaData Topic Tailer version: 2, …} MetaData Topic

14 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- Update State Machine Make a copy of the current state foo -> {functionDetails : {......}, version: 3, …} Merge the updates foo -> {functionDetails : {...}, Increment the version MetaData Topic Tailer version: 2, …} MetaData Topic

15 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- Update State Machine Make a copy of the current state Merge the updates foo -> {functionDetails : {...}, Increment the version MetaData Topic Tailer version: 2, …} Write to MetaData Topic MetaData Topic

16 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- Update State Machine Make a copy of the current state Merge the updates foo -> {functionDetails : {...}, Increment the version MetaData Topic Tailer version: 2, …} Write to MetaData Topic Tailer reads and verifies MetaData Topic

17 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- Update State Machine Make a copy of the current state Merge the updates foo -> {functionDetails : {.....}, Increment the version MetaData Topic Tailer version: 3, …} Write to MetaData Topic Tailer reads and verifies MetaData Topic Upon no conflict, tailer updates

18 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- When do conflicts occur? Worker 2 Worker 1 Multiple Workers foo -> {functionDetails : {...}, foo -> {functionDetails : {...}, version: 2, version: 2, …} …} MetaData Topic Tailer MetaData Topic Tailer MetaData Topic

19 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- When do conflicts occur? Worker 2 Worker 1 Multiple Workers foo -> {functionDetails : {...}, foo -> {functionDetails : {...}, Concurrent updates to same function version: 2, …} version: 2, …} MetaData Topic Tailer MetaData Topic Tailer MetaData Topic

20 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- When do conflicts occur? Worker 2 Worker 1 Multiple Workers foo -> {functionDetails : {...}, foo -> {functionDetails : {...}, Concurrent updates to same function version: 3, …} version: 3, …} First Writer Wins MetaData Topic Tailer MetaData Topic Tailer MetaData Topic

21 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Function MetaData Manager:- When do conflicts occur? Worker 2 Worker 1 Multiple Workers foo -> {functionDetails : {...}, foo -> {functionDetails : {...}, Concurrent updates to same function version: 3, …} version: 3, …} First Writer Wins MetaData Topic Tailer MetaData Topic Tailer Others are rejected MetaData Topic

22 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Advantages Worker 2 Worker 1 Submit to any worker foo -> {functionDetails : {...}, foo -> {functionDetails : {...}, Validation load scales linearly version: 3, …} version: 3, …} Deterministic State Machine MetaData Topic Tailer MetaData Topic Tailer MetaData Topic is audit log MetaData Topic

23 . © 2020 SPLUNK INC. Pulsar Functions:- Submission Workflow Pitfalls Worker 2 Worker 1 MetaData topic topic growth foo -> {functionDetails : {...}, foo -> {functionDetails : {...}, MetaData Topic compaction version: 3, version: 3, …} …} non-trivial Worker Start time MetaData Topic Tailer MetaData Topic Tailer All Workers know everything MetaData Topic

24 . © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Agenda Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work

25 . © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Pluggable Scheduler IScheduler Interface Abstracts out Scheduler Executed only on a Leader public interface IScheduler { List<Assignment> schedule(<List<Instance> unassigned, List<Instance> current, Invoked when Set<String> workers); • Function CRUD operations } – create/update – delete • Worker Changes – Unresponsive/dead workers – New workers – Periodic – Leadership changes

26 . © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Leader Election Leader Election Empty Coordination Topic Failover Subscription Worker 3 Worker 2 Worker 1 Active Consumer is the Leader Coordination Topic

27 . © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Function Assignments Worker 3 Worker 2 Worker 1 Assignment Topic {foo, 1} : worker-1, {foo, 1} : worker-1, {foo, 1} : worker-1, ... ... ... Written by the Leader Compacted based on key(FQFN + Assignment Tailer Assignment Tailer Assignment Tailer Instance Id) All workers know about all assignments Assignment Topic

28 . © 2020 SPLUNK INC. Pulsar Functions:- Scheduling Workflow Assignment Topic Assignment Stores Assignment message Instance { Compacted FunctionMetaData functionMetaData = 1; int32 instanceId = 2; Key -> (FQFN + InstanceId) } message Assignment { Instance instance = 1; string workerId = 2; }

29 . © 2020 SPLUNK INC. Pulsar Functions:- A Deep Dive Agenda Brief introduction to Pulsar Functions Deep Dive into internals • Submission workflow • Scheduling workflow • Execution workflow • Java Instance concepts Current/Future Work

2 点赞
2 收藏
3下载
确认
3秒后跳转登录页面
去登陆