Challenges and Opportunities of Apache Flink® Ecosystem Part I

1 .Challenges and Opportunities of Apache Flink® Ecosystem ‫ݪل‬ғᴨ᯾૬૬ ‫ݪل‬ғᴨ᯾૬૬ ᘳ֖ғᩒႮದ๞ӫਹ ᘳ֖ғṛᕆದ๞ӫਹ ᄍᦖᘏғሴᕨ孖 य़ဉ ᄍᦖᘏғᒍ‫ڻ‬᲋ ᓌ᲋ 

2 . ۹Ղय़਍ ᗦࢵ‫ے‬૞य़਍ࣀࣈԵߵ‫໊ړ‬ EECS Computer Engineering ‫ܗ‬᭗ Broadcom ᚜ԡ Facebook ᴨ᯾૬૬ High-Perf Platform Social Graph Real-Time Data Infra Storage E A ᨮᨱय़හഝਫ෸ᦇᓒଘ‫ݣ‬ . - - ᓒဩૡᑕ҅޾ದ๞ኞா @ -. - - . - - . . - . 

3 .Apache Flink Ecosystem - Present Messaging Messaging Queue Apache SAMOA Queue Yarn Apache Flink is the most sophisticated open-source Stream Processor 

4 .,QWHOOLJHQW%LJ'DWD&RPSXWLQJ Business Intelligence Artificial Intelligence Big Data Infrastructure Can Apache Flink become an unified engine for intelligent big data computing? 

5 .%XLOG,QWHOOLJHQW%LJ'DWD3ODWIRUPZLWK$SDFKH)OLQN ၞᦇᓒ୚ක ᕹӞጱय़හഝ ಢᦇᓒ୚ක ฬᚆᦇᓒ୚ක samza Apex Apache Flink $,ᦇᓒ୚ක 

6 .$SDFKH)OLQN (FRV\VWHP )XWXUH Messaging Queue AI Processing Flink ML Flink TableAPI Apache SAMOA Messaging Queue Batch Processing Enhance User Yarn Experience Container-centric Management 

7 .)OLQN Ń 3RZHUIXODQG8QLILHG&RPSXWH(QJLQH $3,ጱᕹӞ unified batch and stream processing, same query, same results The Practice and Challenges in Building a Stream-Based Unified Big Data Processing Engine 4XHU\3URFHVVLQJጱᕹӞ unified query optimization and query execution framework Kurt Yang, Staff Engineer at Alibaba Jark Wu, Senior Software Engineer at Alibaba 3:20 – 4:00pm ଫአጱᕹӞ ‫ړ‬տ࣋ӣ 310 switch between batch processing and streaming seamlessly &DQREWDLQ[SHUIRUPDQFHLPSURYHPHQWLQ EDWFKSURFHVVLQJZLWKUHFHQWRSWLPL]DWLRQ 

8 .,QWHJUDWH)OLQN ZLWK+LYH(FRV\VWHP)/,1. Flink DML or Hive DML Zeppelin / Flink SQL_Cli Table Source/Sink New Catalog API Connector API HiveCatalog Hive Data Connector FlinkHiveMetaStoreCatalog Hive Metadata is 100% compatible Hive Data is 100% (Data types, Tables, Views, UDFs) accessible 

9 .,QWHJUDWH)OLQN ZLWK+LYH(FRV\VWHP)/,1. Flink SQL_Cli Catalog DML Support Support Hive Data Most Hive Connector Data Types 

10 .$,3URFHVVLQJ Transformation Training Predication Predication Data Process and Model Testing and Data Ingestion Online Serving Transformation Training Validation Change model or parameters 

11 .$,3URFHVVLQJ Data Engineering Data Science / AI Production Data Process and Visualization Reports, Transformation and Analysis Dashboards Acquisition Governance Change model Model Training Production Online or parameters & Testing Data Pipelines Serving 

12 .7DEOH$3, IRU$,3URFHVVLQJ Simplify Machine Learning With Apache Flink® TableAPI Jiangjie Qin, Staff Engineer at Alibaba Jincheng Sun, Staff Engineer at Alibaba 2:40 – 3:20pm ‫ړ‬տ࣋Ӟ 309B