Challenges and Opportunities of Apache Flink® Ecosystem Part I

王绍翾 章剑锋 Challenges and Opportunities of Apache Flink® Ecosystem

1.Challenges and Opportunities of Apache Flink® Ecosystem ‫ݪل‬ғᴨ᯾૬૬ ‫ݪل‬ғᴨ᯾૬૬ ᘳ֖ғᩒႮದ๞ӫਹ ᘳ֖ғṛᕆದ๞ӫਹ ᄍᦖᘏғሴᕨ孖 य़ဉ ᄍᦖᘏғᒍ‫ڻ‬᲋ ᓌ᲋

2. ۹Ղय़਍ ᗦࢵ‫ے‬૞य़਍ࣀࣈԵߵ‫໊ړ‬ EECS Computer Engineering ‫ܗ‬᭗ Broadcom ᚜ԡ Facebook ᴨ᯾૬૬ High-Perf Platform Social Graph Real-Time Data Infra Storage E A ᨮᨱय़හഝਫ෸ᦇᓒଘ‫ݣ‬ . - - ᓒဩૡᑕ҅޾ದ๞ኞா @ -. - - . - - . . - .

3.Apache Flink Ecosystem - Present Messaging Messaging Queue Apache SAMOA Queue Yarn Apache Flink is the most sophisticated open-source Stream Processor

4.,QWHOOLJHQW%LJ'DWD&RPSXWLQJ Business Intelligence Artificial Intelligence Big Data Infrastructure Can Apache Flink become an unified engine for intelligent big data computing?

5.%XLOG,QWHOOLJHQW%LJ'DWD3ODWIRUPZLWK$SDFKH)OLQN ၞᦇᓒ୚ක ᕹӞጱय़හഝ ಢᦇᓒ୚ක ฬᚆᦇᓒ୚ක samza Apex Apache Flink $,ᦇᓒ୚ක

6.$SDFKH)OLQN (FRV\VWHP )XWXUH Messaging Queue AI Processing Flink ML Flink TableAPI Apache SAMOA Messaging Queue Batch Processing Enhance User Yarn Experience Container-centric Management

7.)OLQN Ń 3RZHUIXODQG8QLILHG&RPSXWH(QJLQH $3,ጱᕹӞ unified batch and stream processing, same query, same results The Practice and Challenges in Building a Stream-Based Unified Big Data Processing Engine 4XHU\3URFHVVLQJጱᕹӞ unified query optimization and query execution framework Kurt Yang, Staff Engineer at Alibaba Jark Wu, Senior Software Engineer at Alibaba 3:20 – 4:00pm ଫአጱᕹӞ ‫ړ‬տ࣋ӣ 310 switch between batch processing and streaming seamlessly &DQREWDLQ[SHUIRUPDQFHLPSURYHPHQWLQ EDWFKSURFHVVLQJZLWKUHFHQWRSWLPL]DWLRQ

8.,QWHJUDWH)OLQN ZLWK+LYH(FRV\VWHP )/,1. Flink DML or Hive DML Zeppelin / Flink SQL_Cli Table Source/Sink New Catalog API Connector API HiveCatalog Hive Data Connector FlinkHiveMetaStoreCatalog Hive Metadata is 100% compatible Hive Data is 100% (Data types, Tables, Views, UDFs) accessible

9.,QWHJUDWH)OLQN ZLWK+LYH(FRV\VWHP )/,1. Flink SQL_Cli Catalog DML Support Support Hive Data Most Hive Connector Data Types

10.$,3URFHVVLQJ Transformation Training Predication Predication Data Process and Model Testing and Data Ingestion Online Serving Transformation Training Validation Change model or parameters

11.$,3URFHVVLQJ Data Engineering Data Science / AI Production Data Process and Visualization Reports, Transformation and Analysis Dashboards Acquisition Governance Change model Model Training Production Online or parameters & Testing Data Pipelines Serving

12.7DEOH$3, IRU$,3URFHVVLQJ Simplify Machine Learning With Apache Flink® TableAPI Jiangjie Qin, Staff Engineer at Alibaba Jincheng Sun, Staff Engineer at Alibaba 2:40 – 3:20pm ‫ړ‬տ࣋Ӟ 309B