Challenges and Opportunities of Apache Flink® Ecosystem Part II

Challenges and Opportunities of Apache Flink® Ecosystem Part II
展开查看详情

1.,QWHUDFWLYH3URJUDPPLQJ )OLQN Flink Env Cached/Materialized TableAPI Flink Cluster Table Submit job1 points job uses generat es lines Result Submit ses job2 u job generates surfaces S ub Result mi t es job job3 us … Result

2.,QWURGXFH5RZEDVHG7DEOH$3, )/,3)OLQN

3.,PSURYH)OLQN 0/3LSHOLQH )OLQN HPEUDFLQJ7DEOH$3,ғ ๅ਻ฃቘᥴ޾ӤಋҔಢၞᕹӞҔ(7/॒ቘ޾$,ᦇᓒ‫ݢ‬զ෫ᖲᵞ౮ VXSSRUWVWUHDPLQJ0/ ඪ೮ਫ෸ᦒᕞ҅ਫ෸ᶼၥ SURYLGHDEHWWHU$,H[SHULHQFH

4.7HQVRUIORZ RQ)OLQN Data Process and Model Testing and Data Ingestion Deployment Transformation Training Validation Change model or parameters

5.$ERXW0HŃ ᒍ‫ڻ‬᲋ -HII=KDQJ Ø Joined Alibaba in 2018/10 Ø Worked in Hortonworks Pivotal & eBay Ø Apache member PMC of Apache Tez, Livy & Zeppelin

6.$SDFKH=HSSHOLQ चԭZHEጱԻ԰ୗ‫ړ‬ຉ1RWHERRNඪ೮ 6TO 6FDODᒵᒵ :HEEDVHGQRWHERRNWKDWHQDEOHV GDWDGULYHQ LQWHUDFWLYHGDWDDQDO\WLFV DQGFROODERUDWLYHGRFXPHQWVZLWK64/ 6FDODDQGPRUH

7.(QKDQFH)OLQN ([SHULHQFHLQ$SDFKH=HSSHOLQ =HSSHOLQ පሲ ᵞ౮ մӱᕆᇙ௔ ҁ(IILFLHQF\҂ ҁ,QWHJUDWLRQ҂ ҁ,QWHJUDWLRQ҂

8. පሲ (IILFLHQF\ Ø ள᭛୏‫ݎ‬ӱ‫ۓ‬᭦ᬋ҅෫ᵱ಑۱҅᮱ᗟ҅ᯈᗝᒵᒵ 4XLFNGHYHORSPHQWQREXLOGQRSDFNDJLQJDQGHDV\FRQILJXUDWLRQ Ø ள᭛ਧ֖Კ᧏҅෫ᵱ᪡᫨‫)ک‬OLQNᵞᗭ๐‫࢏ۓ‬ ,GHQWLI\HUURUIDVWQRQHHGWRVZLWFKWR-0GDVKERDUGIRUH[FHSWLRQ Ø ṛපᓕቘ඙֢)OLQN -REҁ&DQFHO -RE҅ 6DYH3RLQW ᒵᒵ҂ 0DQLSXODWH)OLQN MREHDVLO\LQFOXGLQJVXEPLVVLRQFDQFHODQGUHVXPHIURPVDYHSRLQW

9.ᵞ౮ ,QWHJUDWLRQ Ø හഝ‫ݢ‬ᥤ۸ ҁᶉாහഝ  ۖாහഝ҂ 'DWD 9LVXDOL]DWLRQ ҁ6WDWLF  '\QDPLF҂ Ø ٌ՜ૡٍጱᵞ౮ҁ+LYH҅3LJ҅ 3\WKRQ҅0DUNGRZQ ᒵ҂ ,QWHJUDWLRQ ZLWK RWKHU GDWD WRROV ҁ+LYH3LJ3\WKRQ0DUNGRZQDQGHWF Ø ๢࢏਍ԟ໛ຝጱᵞ౮ҁ7HQVRUIORZ҅ .HUDV ᒵ҂ ,QWHJUDWLRQZLWK0DFKLQHOHDUQLQJIUDPHZRUNV 7HQVRUIORZ.HUDV DQGHWF

10.մӱᕆᇙ௔ (QWHUSULVH)HDWXUH Ø ग़ᑄಁҁ0XOWLWHQDQF\҂ Ø ਞ‫ق‬ḵᦤ ҁ$XWKHQWLFDWLRQ $XWKRUL]DWLRQ Ø ᶱፓդᎱᓕቘ 3URMHFW 0DQDJHPHQW Ø Өԯ๐‫ۓ‬ጱᵞ౮ҁ,QWHJUDWLRQ ZLWK &ORXG҂

11.7DNH$ZD\ Batch Processing • Unified Batch and Stream Processing • Integrate Flink with Hive Ecosystem (FLINK-10556) AI Processing • Interactive Programming (FLINK-11199) • Iterative Processing • Row-based TableAPI (FLIP29, FLINK-10972) • Tensorflow on Flink • Flink ML pipeline (FLINK-11095) Zeppelin • Enhance Flink Experience in Apache Zeppelin (ZEPPELIN-3913)

12. Demo • Query Hive Meta and Data via Flink SQL_CLI • Flink Batch and Stream Processing on Zeppelin

13.