TGIP-CN-019:Interactive Querying Of Streams Using Apache Pulsar


1. StreamNative TGIP 019 Interactive Querying Of Streams Using Apache Pulsar Gaoran 2020.8.2

2.Community PR 7646 - Support Set Backlog Quota On Topic Level Http Rest API - GET: {tenant}/{namespace}/{topic}/backlogQuotaMap - POST: {tenant}/{namespace}/{topic}/backlogQuota - DELETE: {tenant}/{namespace}/{topic}/backlogQuota - PIP 39 - Namespace Change Events - PR 4955 - Introduce system topic and topic policies service

3.Community PR 7647 - Allow Ability To Specify Retain Key Ordering In Functions - FunctionCon ig: retainKeyOrdering f

4.Community PR 7655 - Fix Backward Compatibility Issues With Batch Index Acknowledgment - Consumer: batchIndexAckEnabled - Broker: acknowledgmentAtBatchIndexLevelEnabled

5.Pulsar SQL Presto Distributed SQL Query Engine for Big Data What Presto Is Not ? What Presto Is ?

6.Pulsar SQL Pulsar SQL Based On Presto - PulsarConnector -> Connector - PulsarSplitManager -> ConnectorSplitManager - PulsarSplit -> ConnectorSplit - PulsarRecordCursor -> RecordCursor - PulsarMetadata -> ConnectorMetadata - PulsarColumnMetadata -> ColumnMetadata

7.Pulsar SQL Namespace Retention - bin/pulsar-admin namespaces set-retention --size 100M --time 7d public/default - bin/pulsar-admin namespaces get-retention public/default

8.Pulsar SQL Produce Messages To Pulsar - Primitive Type: BOOLEAN, INT16, DOUBLE, STRING, etc. - Complex Type: KeyValue, Struct ( Avro, Json)

9.Pulsar SQL JavaBean

10.Pulsar SQL 1. Use JsonSchema 2. Set batching max messages 3. Set batching max publish delay 4. Set compressionType 5. Send in async way

11.Pulsar SQL Query

12.Pulsar SQL bin/pulsar sql

13.Pulsar SQL Presto JDBC

14.Pulsar SQL Presto Http API http://ip:port/v1/statement http header: X Presto-User -

15.Pulsar SQL Metabase Metabase is the easy, open source way for everyone in your company to ask questions and learn from data.

16.Pulsar SQL

17.Pulsar SQL

18.Pulsar SQL

19.Pulsar SQL

20.Pulsar SQL Query Performance Optimized

21.Pulsar SQL Add Topic Partition Number

22.Pulsar SQL Use Meta Property __publish_time__

23.Pulsar SQL How Pulsar SQL Use __publish_time__ isLowerUnbounded = true

24. Pulsar SQL Pulsar SQL Con ig ${PULSAR_HOME}/conf/presto/con coordinator=true node-scheduler.include-coordinator=true ${PULSAR_HOME}/conf/presto/catalog/ Of load related con igurations pulsar.bookkeeper-throttle-value = 0 f f f f

25.Pulsar SQL localhost:8081

26.Pulsar SQL New Feature Pulsar SQL Read Compaction Data

27.Pulsar SQL Thank You

StreamNative 是一家围绕 Apache Pulsar 和 Apache BookKeeper 打造下一代流数据平台的开源基础软件公司。秉承 Event Streaming 是大数据的未来基石、开源是基础软件的未来这两个理念,专注于开源生态和社区的构建,致力于前沿技术。