Hadoop与Spark社区Roadmap与腾讯云开源实践

1 .& & : @ . . & . . 

2 . About Me • (TOSA) • Apache Hadoop Committer & PMC • Hadoop 2.6, 2.8 Release • 10 

3 .: 3 2 3 1 

4 .&& 

5 . vr 8 l K AP GH S d? M l l spP o e l Cv Ik e l K sp e l GUta u 

6 . - c M ) ( i Ah e B - )( ) ) () a a f d 

7 .

8 .• Erasure Coding • GPU/FPGA • Node Attribute • Docker image • Shell script rewrite • YARN Native • Hadoop Submarine management • Global Scheduling Service on YARN • OZone • Multiple Resource • Placement • Storage Policy • Integrate with types Constraints Satisfier Kubenetes • New YARN UI • Timeline service V2 13 Dec ’17 3.0.3 02 Aug ’18 3.1.2 16 Jan ‘19 3.2.0 3.0.0 3.1.0 3.2.0 3.3.0+ GPU /AI 

9 .• Spark on K8S • Barrier Execution • Project Hydrogen • Data source API v2 Mode to support Deep • Spark History • Scala 2.12 Support Learning Server v2 • Built-in Avro data • Spark on • Vectorized ORC source Kubernetes GA Reader • SQL Compatibility enhancement 4 May ‘18 2.3.3 2 Nov ‘18 2.4.0 Q3 2019 estimated) 2.3 2.4 3.0 AI 

10 .

11 . P /,. TSP c ü 80143 A üHO 02778 B.80 6 üL a c B 02778 d.80 6 d 0 3d 5 3d-0 3 5 

12 .业务价值促进开发 开发者群体促进项 者资源增长 目开发 优质的项目促进业 务价值 

13 . Sparkling – 腾讯云新一代云数仓与数据科学平台 , Flume Hive Zeppelin Metastore Livy YARN 以开源组件为核心，打 Alluxio Spark ATLAS …… 造新一代云端数仓与数 Parquet ORC 据科学平台 Sqoop HDFS Airflow 

14 .!