1.夏晨- Apache Linkis(Incubating) 对接多样性计算存储引擎的技术实现和功能特性

1 .

2 . How does Apache Linkis Solve Computational Governance Problems and Support Diversified Big Data Engines Xia Chen Apache Linkis(incubating) Committer 

3 .Agenda 1. What’s Computation Governance Problems? 2. What’s Linkis? 3. Computation Governance Problems & Solutions 4. Support Diversified Big Data Engine---EngineConn 5. Use Cases & Community 6. Future Planning 

4 .What’s Computation Governance Problems? Today's data infrastructure is increasingly dynamic and innovative. Apache Software Foundation big data related project num 2018 around 30 2022 around 80 10+ per year *From FIRSTMARK 

5 .What’s Computation Governance Problems? More data platform components, more difficulty for development and maintenance. Data Governance Data Analysis Tools Machine Learning Tools Data Applications Tools • IDE • Data Dictionary • Notebook • Reports • Data Exchange • Data Lineage • Data Cleansing • Ad Hoc Queries … • Workflow Scheduler • Data Privilege • Model Evaluation • User behaviors • BI Tools • Data Quality • Model Deployment • MDM OLAP Engines OLTP Engines Streaming Engines ML Engines • Hive • HBase • Flink • Anaconda • Spark • ElasticSearch • Spark Streaming • Tensorflow … • Kylin • TiDB • Kafka Streams • Pytorch • Presto • NebulaGraph • Pulsar • Cafee 

6 .What’s Computation Governance Problems? Increasing computation governance problems: connectivity ,extendability ,controllability, orchestration, reusability, etc. APP A APP B APP APP Active Standby ① Hybrid Engine A Engine B Engine Engine A Engine B 

7 .What’s Linkis? Linkis: A computation middleware to solve computation governance problems Before Linkis After Linkis Functional FunctionalApp& App& tools tools Connectivity Extendability Controllability Orchestration Reusability Computation& Computation& Storage Storageengine engine OLAP OLTP Streaming Machine Learning 

8 .What’s Linkis? Linkis is consists of 3 service groups, CGS, PES and MGS. • CGS: Computation Public Enhancement Context Material Data Public Governance Services to Services (PES) Service Service Source Service enhance the request submit →prepare → Applications Computation Governance Engines &Tools Services (CGS) execute → return process Orchestrator LinkisManager Engine A APP 01 Cluster • PES: Public Enhancement Engine Requests Parse AppManager Conn Requests APP 02 E EngineConn n t Plan Manager Engine Engine B Services such as data r Label Cluster Results a n Optimize Manager Conn source, material, and c Results e Execute Resource Engine context services. Conn APP N Manager EngineConn Reheat Engine N Manager Validator ECPlugin Engine Conn Cluster • MGS: Microservice JobHistory Governance Services Microservice Governance Services (MGS) Spring Cloud Eureka Open Feign Gateway 

9 .Linkis Computation Governance Services Introduction CGS is designed to serve and enhance the submit →prepare → execute → return process. Engine Connector(EC): to do the real connecting and result fetching to the underlying engines. Submit Prepare Execute Applications& Computation Governance Engines Tools Services (CGS) Orchestrator LinkisMaster Engine A APP 01 Cluster Engine Request Parse AppManager Conn APP 02 E EngineConn n Plan Manager Engine B t Engine r Label Cluster a Optimize Manager Conn n c e Execute Engine Resource Conn APP N Manager EngineConn Reheat Manager Engine N Engine Cluster Validator ECPlugin Conn JobHistory Return 

10 .Linkis Computation Governance Services Introduction Engines could be classified into 4 classes, which can be covered by 3 Engine Connector (EC) types. EC Type Major Functions Comment OLAP submit() getLog() OlapEC EC Type getProgress() For OLAP, Machine Machine Learning (OLAP Engine Connector) getResult() Learning engines getStatus() kill() submit() Streaming StreamEC getLog() For Streaming engines (Streaming Engine Connector) getStatus() kill() OLTP OltpEC ODBC/JDBC For OLTP engines (OLTP Engine Connector) 

11 .OLAP&ML Engine Computation Governance Problems A data tool/application system developer/maintainer could encounter following problems while dealing with OLAP&ML engines. OlapEC is designed to solve OLAP&ML engine computation governance problems. Environment Prepare Submit Execute Developer • Need to install and • Need to submit various types of • Need to customize parameters maintain multiple tasks: SQL, Python, shell, and rules engine clients and Scala, Java, etc. runtime environments • Need to obtain the task • Need to provide diversified execution result set, task status interface: and logs Restful/JDBC/Python/Shell Operator Operation • Problems such as multi-tenant, high concurrency, high availability, fine-grained resource control, and audit need to be solved 

12 .OLAP&ML Engine Computation Governance Problems-Environment Prepare • Need to install and maintain multiple engine clients and runtime environments. • Need to submit various types of tasks: SQL, Python, shell, Scala, Java, etc. • Need to provide diversified interface: Restful/JDBC/Python/Shell. 

13 .Linkis Computation Governance Solution-Environment Prepare • Decouple the upper and underlying layers, simplify environment maintenance. • Support various types of tasks: SQL, Python, Shell, Scala, Java, etc. • Support diversified interface: Restful/JDBC/Python/Shell. Flexible and convenient. SQL, Python, Shell, Scala, Java Restful/JDBC/Python/Shell 

14 .Linkis Computation Governance Solution-Submit • Decouple the upper and underlying layers, simplify environment maintenance. • Support various types of tasks: SQL, Python, Shell, Scala, Java, etc. • Support diversified interface: Restful/JDBC/Python/Shell. Flexible and convenient. SparkEC PrestoEC SparkEC Restful submit pySpark job JDBC submit presto job PythonClient submit Spark scala job Restful API JDBC Driver Linkis Client //1. Load Linkis jdbc driver linkisClient = LinkisClient( /api/rest_j/v1/entrance/submit Class.forName("org.apache.linkis.ujes.jdbc "hadoop", { .LinkisSQLDriver"); "127.0.0.1:9001", executionContent: { //2. Construct JDBC Connection "pwd") code: "df=spark.sql('select * from table01');show(df)", Connection connection = DriverManager.getConnection( job = Job.builder() runType: "py", "jdbc:linkis://127.0.0.1:9001?limit=false& .setEngineType("spark-2.4.3") }, EngineType=presto", .setRunType("scala") labels:{ "hadoop", .setCreator("qml") engineType:"spark-2.4.3", "123456"); .setCode("val df=spark.sql('select * userCreator:"hadoop-IDE" //3. execute query from table01');show(df)") }, Statement st = .setExecuteUser("hadoop") .... connection.createStatement(); .addLabel("jobRunningTimeout","5000") } ResultSet rs =st.executeQuery("select * .setMaxRetry(2) from table01"); 

15 .Linkis Computation Governance Solution-Execute • Support customize parameters and rules, for example job timeout, job retry, fixed driver node, etc. Job.addLabel(timeout/retry times/fixed node …) Timeout detection& auto kill Auto retry Job assign to specified node … 

16 .Linkis Computation Governance Solution-Execute • Support customize parameters and rules, for example job timeout, job retry, fixed driver node, etc. // 1. build job val job =LinkisJobClient.interactive() .builder() .setEngineType("hive-2.3.3") .setRunType("sql") .setCreator("IDE") .setCode("select * from table01") .setExecuteUser("hadoop") .addLabel("jobRunningTimeout", "1000") // job timeout .addLabel("jobRetryCount", "3") //job retry .addLabel("bindEngine","127.0.0.1:3457") //fixed driver node .addLabel("tenant", "tenant1") //set tenant .build(); // 2. Submit Job to Linkis job.submit(); 

17 .OLAP&ML Engine Computation Governance Problem-Operation • Need to solve high concurrency, scalability and high availability problems. Hundreds of Hundreds of Hundreds of … clients and drivers … clients and drivers … clients and drivers Dozens of servers Dozens of servers Dozens of servers 

18 .Linkis Computation Governance Solution-Operation • A reusable resource pool, easier to maintain. • Microservice architecture and Layered asynchronous to support mega concurrency and high availability. Thousands of Linkis … clients and drivers Master Hundreds of servers 

19 .Linkis Computation Governance Solution-Operation • Running engine connector list 

20 .Linkis Computation Governance Solution-Operation • Multi-tenant, label-based multilevel fine-grained resource control. • Multilevel parameter configuration. Tenant→User → App/Tool → EC Type → EC (User, App/Tool) （EC Type, Version） EC 

21 .Linkis Computation Governance Solution-Operation • Multi-tenant, label-based multilevel fine-grained resource control. • Multilevel parameter configuration. Tenant→User → App/Tool → EC Type → EC App/Tool Level Configuration EC Type Level->EC Configuration User Level Configuration 

22 .Linkis Computation Governance Solution-Operation • Multi-tenant, label-based multilevel fine-grained resource control. • Multilevel parameter configuration. /api/rest_j/v1/entrance/submit { executionContent:{ code:"show tables;", runType:"sql“ }, params:{variable:{},configuration:{}}, source:{scriptPath:"file:///tmp/hadoop/1.sql"}, labels:{ engineType:"spark-2.4.3", userCreator:"hadoop-IDE", route:"entrance1", #optional tenant:"hadoopECM" #optional } } 

23 .OLAP&ML Engine Computation Governance Problems-Operation • Lack of verification, bad Jobs and high-risk operations could be submitted freely. select * from trans_history a join users b on a.uid = b.uid; --driver-memory 256g --num-executors 2048 --executor-cores 8 --executor-memory 32g drop database sme_loans cascade; 

24 .Linkis Computation Governance Solution-Operation • Converged engine entrance, unified identity authentication • High-risk prevention and validation • Unified audit records Authenticator & Validator & Auditor 

25 .Linkis Computation Governance Solution-Operation Example of Validation rule: AlterTableSetLocationCheckRule: For Hive tables, it is not allowed to manually modify the Location of the table Submit a job that violates this rule Restful API /api/rest_j/v1/entrance/submit { executionContent: { code: "alter table test_linkis_check set location 'hdfs:///tmp/test_linkis_check1';", runType: "hql" }, ..... } Error response { "status": 1 "message": "Didn't pass rule : AlterTableSetLocationCheckRule", "data": { "errorMsg": { "serviceKind": "linkis-cg-entrance", "desc": "For Hive tables, it is not allowed to manually modify the Location of the table" }, }, } 

26 .Linkis Public Enhancement Services Introduction PES: Public Enhancement Services such as data source, material, and context services. Public Enhancement Context Material Data Public Services (PES) Service Service Source Service Applications Computation Governance Engines &Tools Services (CGS) Orchestrator LinkisManager Engine A APP 01 Cluster Engine Requests Parse AppManager Conn Requests APP 02 E EngineConn n Plan Manager Engine B t Engine r Label Cluster a Optimize Manager Conn Results n Results c e Execute Engine Resource Conn APP N Manager EngineConn Reheat Engine N Manager Engine Cluster Validator ECPlugin Conn JobHistory Microservice Governance Services (MGS) Spring Cloud Eureka Open Feign Gateway 

27 .Linkis Public Enhancement Services Introduction Break through application silos, sharing user resources & runtime environments. Variable Variable Function Function File File Result set … Result set … Connection connection = DriverManager.getConnection( Context Data Source Material Set dataSourceName Metadata =“ds-presto01- "jdbc:linkis://127.0.0.1:9001?limit=false&EngineType=presto", jdbc” Service "hadoop","123456"); Service Service Query Service OLAP OLTP Streaming Machine Learning 

28 .Support Diversity Computing Storage Engine - EngineConn EngineConn: Engine Connector Positioning: • The module unit that connects and interacts with the underlying computing storage engine (Hive/Spark/Flink...) • For the underlying computing storage engine, it plays the role of a connected client • Can quickly support the expansion of different types of computing storage engines Engine Engine Engine Engine Engine Engine Conn Conn Conn Conn Conn Conn 

29 .Support Diversified Big Data Engine---EngineConn By matching the parameters to the corresponding engine, by obtaining resource materials, the engine is started and the task is executed. Request Task engineType:“hive-1.2.1" engineType:"spark-2.4.3" engineType:“hive-2.3.3" Ec type hive hive spark … Jars & Material Service Version 1.2.1 2.3.3 2.4.3 … conf files Enginer Plugin RersourceId resourceId1 resourceId2 resourceId3 …. Resource