How Splunk Mission Control leverages various Pulsar subscription types——Pranav Dharma

1 . © 2020 SPLUNK INC. How Splunk Mission Control leverages Pulsar Pulsar Summit Pranav Dharma June 17, 2020 

2 . Splunk Mission Control A cloud native, unified experience for modernizing the Security Operation Center (SOC) © 2020 SPLUNK INC. 

3 . © 2020 SPLUNK INC. Core 1) Targeted for the Security Operations Center (SOC) Requirements Reliability – Can not lose a single security event or its details during the security event lifecycle driving technology 2) Multi tenant SAAS product Data isolation of tenant data is a very big deal choices for Mission 3) Micro services based architecture Control Performance and latency when communicating with downstream services is important. Event investigation, Automation and collaboration need to happen with minimal latency 

4 . © 2020 SPLUNK INC. Security Event Lifecycle 

5 . © 2020 SPLUNK INC. Messaging use cases in Mission Control • Sending user notifications as part of user collaboration and approval workflows • Populating data for dashboard panels • Triggering downstream services for automation and security metadata enrichment • Generating audit trail • Publishing Web Socket messages for UI refresh • User triggered resource provisioning • Providing playbook (automation) debug log • Broadcasting important settings and ACL changes to pods 

6 . © 2020 SPLUNK INC. Why Pulsar ? Core Pulsar team now part of Splunk as part of the Streamlio acquisition (Drops Mic) 

7 . © 2020 SPLUNK INC. Why Pulsar ? • Native multitenancy – Satisfies our core requirement of data isolation • Message level acks instead of using only offset level acks – Satisfies our core requirement of reliability • Improve performance and scalability by adding more consumers without adding partitions – Satisfies our core requirement for performance and latency • Unified messaging – We can use both queueing and streaming without the need to operate and maintain different products for each purpose • TTL – Our use cases have varied TTL requirements from none (WebSocket) to high (audit) 

8 . © 2020 SPLUNK INC. Why Pulsar ? • Simple producers and clients – Reduced complexity for developers and increased productivity • Operational overhead – Several benefits, but adding new brokers, additional storage etc. worth calling out • Topic creation is lightweight – Easier and light weight to create new topics if needed (We evaluated Redis and Kafka when making this design decision) 

9 . © 2020 SPLUNK INC. Use case for Exclusive subscription Service Service Service Pod Pod Pod Pod Pod Pod Pod Pod Pod Each pod has a consumer subscribing to a topic with the subscription type ‘Exclusive’ – streaming or pub-sub paradigm Critical Settings / ACL changes Producer Pulsar 

10 . © 2020 SPLUNK INC. Use case for Shared subscription Service Consumer service Pod Pod Pod Pod Pod Service Pod Pod Data for use cases like websocket notification, send emails, dashboard data, audit record etc. can be generated by any of the Service services and is published to Pulsar. Pulsar Consumers in the consumer service subscribe using a ‘shared’ Pod Pod subscription (queueing paradigm) 

11 . © 2020 SPLUNK INC. Use case for Key Shared subscription First some background • Automation consists of playbooks – Playbooks consist of discrete units of work called actions – Example of actions: “block user”, “create ticket”, “restart server” – Actions can run sequentially or concurrently – A final “on_finish” handler (part of playbook) called when playbook completes – should be called only once – Only one pod should be able to call the “on_finish” – Consumer with Key shared subscription on the playbook initiating pod will ensure only this pod will call the ‘on_finish’ handler • Action runs – Action runs can be cancelled – Cancel action run messages need to be routed to the pod running the action • Used as message bus between the automation services 

12 . © 2020 SPLUNK INC. Use case for Key Shared subscription Service Run action for <playbook_id> Pod Pod Message published with Message published with <playbook_id> key <playbook_id> key Service The pod initializing the playbook run creates a key_shared Pulsar subscription to decide when Pod Pod Message with <playbook_id> key consumed playbook run is complete 

13 . © 2020 SPLUNK INC. Use case for Key Shared subscription Run action for <action_id> Pod Pod Cancel action message Service consumed with key <action_id> Key_shared subscription used to route cancel action run messages to the pod running Pod Pod Cancel action message Pulsar the action published with key <action_id> 

14 . © 2020 SPLUNK INC. What’s next ? • So far, Pulsar integration has been pretty painless relatively • Still learning, tweaking and optimizing • Unbundle various queueing based consumers into their own service • Opens possibility of event based design 

15 . © 2020 SPLUNK INC. Thank You