EP-003: Topic Discovery, ServiceURL, Cluster

1 .

2 .Topic Discovery, ServiceURL, Cluster TGIP-CN Episode 003 

3 .How does Pulsar assign topics to brokers? 

4 .Tenant & Namespace persistent :// my-tenant / my-namespace / my-topic Topic Domain Tenant Namespace Topic 

5 .Tenant, Namespace and Topics Pulsar Instance Tenant Tenant Tenant (Data Serving) (Marketing) (Product Safety) NS NS NS NS NS (Microservice) (ETL) (Campaigns) (ETL) (Fraud Detection) Topic Topic Topic Topic Topic Topic Topic (Customer (Location (Demographic (Budget (Acct (User (Risk Authentication) Resolution) classification) spend) history) clusterin) classification) 

6 .Namespace Tenant Namespace T0 T1 T2 T3 T4 T5 

7 .Namespace Bundles 0x00000000 Tenant Namespace 0x40000000 0xc0000000 T0 T1 T2 T3 T4 T5 0x80000000 

8 .Namespace Bundles 0x00000000 T2 Tenant T0 T5 Namespace 0x40000000 0xc0000000 T3 T0 T1 T2 T3 T4 T5 T1 T4 0x80000000 

9 .Bundle Assignment 0x00000000 T2 T0 T5 0x40000000 0xc0000000 T3 T1 T4 0x80000000 

10 .Load Manager 0x00000000 T2 T0 T5 0x40000000 0xc0000000 T3 Broker 0 Broker 1 Broker 2 Broker 3 T1 T4 0x80000000 

11 .Load Manager - Leader Election 0x00000000 T2 T0 T5 0x40000000 0xc0000000 T3 Broker 0 Broker 1 Broker 2 Broker 3 T1 T4 0x80000000 Load Manager (Leader) 

12 .Load Manager - Assign bundles to brokers 0x00000000 T5 T4 T2 T2 T3 T1 T0 T0 T5 0x40000000 0x00000000 0x40000000 0x80000000 0xc0000000 0xc0000000 - - - - 0x40000000 0x80000000 0xc0000000 0xffffffff T3 Broker 0 Broker 1 Broker 2 Broker 3 T1 T4 0x80000000 Load Manager assigns Load Manager namespace bundles to brokers (Leader) based on load reports. 

13 .How does Pulsar client locate owner brokers? 

14 .Topic Owners T2 T4 Broker 0 Broker 1 Broker 2 Broker 3 T5 T3 T1 T0 ZooKeeper ZooKeeper ZooKeeper 

15 .Topic Lookup Pulsar Client T2 T4 Broker 0 Broker 1 Broker 2 Broker 3 T5 T3 T1 T0 ZooKeeper ZooKeeper ZooKeeper 

16 .Topic Lookup Pulsar Client ) ic (T1 p To lt 2) ookup Resu ker 3. Establish the persist L up Bro TCP connection to the 1. ook er = L owner broker for 2. Brok subsequent requests wner (O T2 T4 Broker 0 Broker 1 Broker 2 Broker 3 T5 T3 T1 T0 ZooKeeper ZooKeeper ZooKeeper 

17 .HTTP Topic Lookup Pulsar Client ) ic (T1 p To up lt L ook Resu r 2) 3. Establish the persist P up ke H TT ook = Bro TCP connection to the 1. L TP er owner broker for . HT Brok 2 er subsequent requests n (Ow T2 T4 Broker 0 Broker 1 Broker 2 Broker 3 T5 T3 T1 T0 ZooKeeper ZooKeeper ZooKeeper 

18 .Topic Lookup Summary ❏ Topic Lookup Requests can be served by any broker ❏ Client doesn’t have to know all the broker’s address ahead of time ❏ DNS, Load Balancer or Multi-hosts service url can be used ❏ Persistent TCP connection is established with owner broker for subsequent requests ❏ Client requires connectivity to the owner broker address returned from lookup result 

19 .Service URL ❏ Service URL is the access point for Pulsar clients to lookup topic ownerships ❏ Service URL protocol ❏ Binary ❏ HTTP ❏ Service URL host ❏ DNS: pulsar://example.pulsar.local:6650 ❏ Load Balancer: pulsar://1.2.3.4:6650 ❏ Multi-Hosts: pulsar://broker-0:6650,broker-1:6650,broker-3:6650 

20 .If a client can not access the broker address directly? 

21 . Add Proxies Proxies allow for load balancing across Pulsar brokers Producer Proxies Consumer Producers send data through the proxies Consumers receive data through the and do not interact with brokers proxies and do not interact with brokers Brokers Bookies 

22 .Topic Lookup with Proxies Pulsar Client ) ic (T1 p To Proxies ookup 3. Establish the persist L TCP connection to the 1. owner broker for lt o o kup Resu ker 2) subsequent requests 2. L Bro e r B roker = (Own T2 T4 Broker 0 Broker 1 Broker 2 Broker 3 T5 T3 T1 T0 ZooKeeper ZooKeeper ZooKeeper 

23 .Cluster ❏ Identified by a cluster-name ❏ Cluster name is used for configuring geo-replication ❏ Cluster configuration ❏ Web service url (http://) ❏ Web service url tls (https://) ❏ Broker service url (pulsar://) ❏ Broker service url tls (pulsar+ssl://) 

24 .Cluster Setup $ bin/pulsar initialize-cluster-metadata \ --cluster pulsar-cluster-1 \ --zookeeper zk1.us-west.example.com:2181 \ --configuration-store zk1.us-west.example.com:2181 \ --web-service-url http://pulsar.us-west.example.com:8080 \ --web-service-url-tls https://pulsar.us-west.example.com:8443 \ --broker-service-url pulsar://pulsar.us-west.example.com:6650 \ --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651 

25 .Cluster Operations ❏ Create ❏ Get ❏ Update ❏ List