展开查看详情
1 . Introduction to
Apache DolphinScheduler 3.3.0
2 .GitHub: ruanwenjun
WhaleOps/eBay/Pinduoduo
Focus on distributed system and microservices middleware
PMC Member of Apache DolphinScheduler/Seatunnel/EventMesh
Live in Wuhan
3 .Agenda
• Introduction to DolphinScheduler
• New changes in DolphinScheduler 3.3.0
• How to participate in the community
4 .What is DolphinScheduler
Open-source system for workflow orchestrating, triggering and management
• Workflow is first-class citizen
• No/Low code
• Scale without data migration
• Pluginable architecture
• …
5 .Workflow is first-class citizen
• Workflow contain tasks and “workflows”
• Each task should belong to a workflow
• Workflow can dependent on other workflow
• Workflow is minimal execution unit
• Workflow can be manual or atomatic trigger
6 .No/Low code
• Manage the workflow by portal website
• Manage the workflow by OpenAPI or PyDolphinScheduler
7 .Scale without data migration
• Metadata are stored in a shared relational database
• Automatic fault tolerance
• Load balance and task automatic shifting
8 .Pluginable architecture
• Components outside the workflow-engine are implemented as plugins
• Alert-Plugins: Email, Http, WeChat, Slack…
• TaskPlugins: Shell, SQL, Spark, K8s…
• DatasourcePlugins: Mysql, PG, Oracle…
• Registry: Zookeeper, JDBC, ETCD
• Storage: HDFS, S3, OSS…
• …
9 .Architecture in 3.3.0
• API：Manage metadata
• Master：Orchestrate workflow
• Worker：Execute task
10 .New Task Plugin Interface
WorkerServer
• Used to manage the task plugin instance lifecycle
• Lifecycle: Running、Pause、Kill、Failover、Success、Failed
Problem
• There is no way to do failover in Task plugin
• Different task can support different failover operation：take-over、resubmit、kill-resubmit、failed
• The current task interface is complicated
11 .New Task SPI
Clearer interfaces
ITaskApplicationContext: Represent the third part task instance info, e.g. processId, applicationId
ITaskListenerManager: Can register listener used to listen the task info changed
12 . Improved fault tolerance
MasterFailover
• Failover the workflows which hold by the crashed master
• Insert workflow failover command
• Workflow failover will take over the running tasks from work server
WorkerFailover
• Failover the tasks which hold by the crashed worker
• Dispatch a failover task to new worker
• Failover operation might be different in different task plugin
• Failover support exactly-once、at-least-once、at-most-once
13 .Parameter priority
Parameter type
• Startup parameter
• Workflow definition parameter
• Task definition parameter
Problem
• Parameter priority is different in different parameters
• Some parameters can't be changed at runtime.
News
• Parameter priority is unify.
• Follow proximity principle: Startup parameter > task definition parameter > workflow definition parameter
14 .Decoupled Workflow Trigger
Workflow Trigger
• All workflow trigger are represented by command
Problem
• Command can only be consumed from DB
• No trigger command schema
News
• Provide standard command schema
• CommandEngine used to detective pending command
• Repository can be streaming system
15 .Event thread model
Two event thread pool
• There are exist two event thread pool in master server, one used to deal
with workflow event, another used to deal with task state change event.
Problem
• It’s difficult to coordinate these two thread pool
• The sequence of events might be lost
• It’s difficult to set the thread pool size
News
• Unify the event thread pool
• All event belong to one workflow will FIFO in one queue
• More event metrics
• Event visualization
16 .Multiple hadoop/s3 cluster
Configure the cluster from common.properties
• Configure the hadoop/s3/.. configuration in common.properties
Problem
• It’s difficult to add new cluster in runtime
• The configuration file is too large
News
• Manage the configuration in metadata store
• Support multiple cluster in runtime
• Support add new cluster in runtime
• Different task can use different cluster
17 . Alert Server
AlertServer
• AlertServer is a HA server
• AlertServer use single thread to send alert
Problem
• Need to use additional machine to deploy AlertServer
News
• The AlertServer can be embedded at API Server
• Use multiple thread model at AlertServer
18 .How to participate in the community
• Mail is the most suggested way to discuss problem
• Github is the most simplest way to participate DolphinScheduler
19 . Mail
• users@dolphinscheduler.apache.org for users
• dev@dolphinscheduler.apache.org for developer
• Discuss usage problem, design proposal, new idea…
• Discuss before do
• Vote for the new release is the simplest way
20 .Github
• https://github.com/apache/dolphinscheduler
• https://github.com/apache/dolphinscheduler-website
• https://github.com/apache/dolphinscheduler-operator
• https://github.com/apache/dolphinscheduler-sdk-python
• DSIP is the only way to import a new feature
• Report a bug is as important as fixing them
• The simpler PR the better PR
21 .End
• Patience is the most important thing
热门城市
