Pulsar Architectural Patterns for CI/CD Automation and Self-Service——Devin Bost

展开查看详情

1.Pulsar Architectural Patterns for CI/CD Data-Driven CI/CD Automation for Pulsar Function Flows and Pub/Sub Includes on-prem, AWS, and GCP architectures By Devin Bost, Senior Data Engineer at Overstock Every pattern shown here has been developed and implemented with my team at Overstock + Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/

2.Legend & Referenced Technologies Pulsar Function Amazon S3 Flink Job AWS IAM Pulsar Topic ReactJS Sonotype Nexus Pulsar Beam Pulsar Brokers AWS CodePipeline Kubernetes GCP IAM GCP Cloud Storage Golang Docker CouchDB GCP Cloud Build Google Cloud Functions

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.Data + Contact = Modular design +

16.Modular Design Reusable functions

17.Modular Design Reusable functions

18.

19.

20.

21.

22.Might need to manually satisfy contract at first

23.Might need to manually satisfy contract at first

24.Until you can get to where the data is originated Might need to manually satisfy contract at first

25. Push to deployment Filter to Push to gate Build tool Artifact Storage Storage data Store pipeline for desired artifact data keeping system environment Push to deployment Filter to Push to gate Build tool (1) Artifact Storage Store pipeline for desired artifact data keeping system environment (2) Build data

26.

27.

28.{ "type": "function", "artifactPathOrUrl": "http://path-to-artifact/example-ignite-function-1.0.1-20200125.003935-3- jar-with-dependencies.jar", "tenant": "exampleTenant", "namespace": "exampleNamespace", "name": "exampleIgniteFunction-backfill”, "className": "com.yourcompany.pulsar.functions.ExampleIgniteFunction", "userConfig": { "username": "igniteUser", "password": "exampleHashedPass", "cache_name": "example-ignite-cache-backfill”, "hosts_with_ports": "igniteserver1.domain.com:10800, igniteserver2.domain.com:10800, igniteserver3.domain.com:10800, igniteserver4.domain.com:10800 }, "inputs": [ "persistent://feeds/exampleProject/data-to-dump-into-ignite-backfill” ], "output": "persistent://exampleTenant/exampleNamespace/data-enriched-from-ignite-backfill”, "logTopic": "persistent://public/default/function-log-topic-backfill” }

29. Using the Java Admin API to consume from a Pulsar topic { "type": "function", "artifactPathOrUrl": "http://path-to-artifact/example-ignite- function-1.0.1-20200125.003935-3-jar-with-dependencies.jar", "tenant": "exampleTenant", "namespace": "exampleNamespace", "name": "exampleIgniteFunction", Pulsar Brokers "className": "com.yourcompany.pulsar.functions.ExampleIgniteFunction", "userConfig": { "username": "igniteUser", "password": "exampleHashedPass", via Java Pulsar REST "cache_name": "example-ignite-cache", Admin API Admin API "hosts_with_ports": "igniteserver1.domain.com:10800, igniteserver2.domain.com:10800, igniteserver3.domain.com:10800, Consumer/Producer igniteserver4.domain.com:10800 }, "inputs": [ "persistent://feeds/exampleProject/data-to-dump-into-ignite" ], "output": "persistent://exampleTenant/exampleNamespace/data- enriched-from-ignite", "logTopic": "persistent://public/default/function-log-topic" }

StreamNative 是一家围绕 Apache Pulsar 和 Apache BookKeeper 打造下一代流数据平台的开源基础软件公司。秉承 Event Streaming 是大数据的未来基石、开源是基础软件的未来这两个理念,专注于开源生态和社区的构建,致力于前沿技术。