申请试用
HOT
登录
注册
 
Pulsar Architectural Patterns for CI/CD Automation and Self-Service——Devin Bost
3 点赞
1 收藏
0下载
StreamNative
/
发布于
/
1303
人观看
展开查看详情

1.Pulsar Architectural Patterns for CI/CD Data-Driven CI/CD Automation for Pulsar Function Flows and Pub/Sub Includes on-prem, AWS, and GCP architectures By Devin Bost, Senior Data Engineer at Overstock Every pattern shown here has been developed and implemented with my team at Overstock + Email: dbost@overstock.com Twitter: DevinBost LinkedIn: https://www.linkedin.com/in/devinbost/

2.Legend & Referenced Technologies Pulsar Function Amazon S3 Flink Job AWS IAM Pulsar Topic ReactJS Sonotype Nexus Pulsar Beam Pulsar Brokers AWS CodePipeline Kubernetes GCP IAM GCP Cloud Storage Golang Docker CouchDB GCP Cloud Build Google Cloud Functions

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.Data + Contact = Modular design +

16.Modular Design Reusable functions

17.Modular Design Reusable functions

18.

19.

20.

21.

22.Might need to manually satisfy contract at first

23.Might need to manually satisfy contract at first

24.Until you can get to where the data is originated Might need to manually satisfy contract at first

25. Push to deployment Filter to Push to gate Build tool Artifact Storage Storage data Store pipeline for desired artifact data keeping system environment Push to deployment Filter to Push to gate Build tool (1) Artifact Storage Store pipeline for desired artifact data keeping system environment (2) Build data

26.

27.

28.{ "type": "function", "artifactPathOrUrl": "http://path-to-artifact/example-ignite-function-1.0.1-20200125.003935-3- jar-with-dependencies.jar", "tenant": "exampleTenant", "namespace": "exampleNamespace", "name": "exampleIgniteFunction-backfill”, "className": "com.yourcompany.pulsar.functions.ExampleIgniteFunction", "userConfig": { "username": "igniteUser", "password": "exampleHashedPass", "cache_name": "example-ignite-cache-backfill”, "hosts_with_ports": "igniteserver1.domain.com:10800, igniteserver2.domain.com:10800, igniteserver3.domain.com:10800, igniteserver4.domain.com:10800 }, "inputs": [ "persistent://feeds/exampleProject/data-to-dump-into-ignite-backfill” ], "output": "persistent://exampleTenant/exampleNamespace/data-enriched-from-ignite-backfill”, "logTopic": "persistent://public/default/function-log-topic-backfill” }

29. Using the Java Admin API to consume from a Pulsar topic { "type": "function", "artifactPathOrUrl": "http://path-to-artifact/example-ignite- function-1.0.1-20200125.003935-3-jar-with-dependencies.jar", "tenant": "exampleTenant", "namespace": "exampleNamespace", "name": "exampleIgniteFunction", Pulsar Brokers "className": "com.yourcompany.pulsar.functions.ExampleIgniteFunction", "userConfig": { "username": "igniteUser", "password": "exampleHashedPass", via Java Pulsar REST "cache_name": "example-ignite-cache", Admin API Admin API "hosts_with_ports": "igniteserver1.domain.com:10800, igniteserver2.domain.com:10800, igniteserver3.domain.com:10800, Consumer/Producer igniteserver4.domain.com:10800 }, "inputs": [ "persistent://feeds/exampleProject/data-to-dump-into-ignite" ], "output": "persistent://exampleTenant/exampleNamespace/data- enriched-from-ignite", "logTopic": "persistent://public/default/function-log-topic" }

3 点赞
1 收藏
0下载