Flink Streaming SQL 2018 Part I

【分会场二04-Piotr Nowojsk】Flink Streaming SQL 2018
展开查看详情

1. Flink Streaming SQL 2018 ‫ݪل‬ғ data Artisans ᘳ֖ғ Software Engineer ᄍᦖᘏғ Piotr Nowojsk

2. About data Artisans ‫ى‬ԭ data Artisans PLATFORM Original Creators of Real-Time Stream Processing Apache Flink® Enterprise Ready Apache Flink ܻত֢ᘏ մӱᕆਫ෸ၞୗ॒ቘ

3.Flink Streaming SQL 2018

4. ፓ୯ $JHQGD ԅ֜ᭌೠ 64/Ҙ Why SQL? ᭌೠ 6WUHDPLQJ64/ਖ਼ᥝᶎ੒ጱ೴౴ Challenges in Streaming SQL ࣁ 6WUHDPLQJ64/Ӿᬳളᤒጱӧ‫ݶ‬ොୗ Various ways to join tables in Streaming SQL ཛྷୗᦩ‫ڦ‬ Pattern recognition ٌ՜ᬪ๗౮ຎ Other recent improvements.

5. ԅ֜ᭌೠ SQL Why SQL? ռಅޮᎣጱള‫ݗ‬ :HOONQRZQLQWHUIDFH ෫ᵱᖫᑕńńฃԭӤಋ 1RSURJUDPPLQJLVUHTXLUHG HDVLHUWROHDUQ ኩᦫୗ᧍᥺ᤒᬡ֦ጱࠟӱ᭦ᬋ 'HFODUDWLYHZD\WRH[SUHVV\RXUEXVLQHVVORJLF ٖୌս۸ 2XWRIWKHER[RSWLPL]DWLRQ

6.ᭌೠ Streaming SQL ਖ਼ᥝᶎ੒ጱ೴౴ Challenges in Streaming SQL

7. ಢེ॒ቘਫֺ Batch example SELECT a.id FROM A a, B b WHERE a.id = b.id

8. SELECT a.id FROM A a, B b WHERE a.id = b.id Table A Table B Result 1 42 1 42 7 42 2 3 3 3 1 6

9. ୭ଚᬳളᓒဩ Sort-Merge Join Table A Table B 1 42 42 7 2 3 3 1 6

10. ᒫӞྍ ń ‫ړ‬ᔄ First step - Sort Table A Table B 1 1 2 3 3 7 6 42 42

11. ᒫԫྍ ‫ݳ‬ଚ݊ᬳള Second step Merge and Join Table A Table B Result 1 1 1 2 3 3 3 7 42 6 42 42

12. ग़ᤒᬳളᓒဩ Hash Join Table A Table B Result 1 42 1 42 7 42 2 3 3 3 1 6

13. ᬳളᬳᖅັᧃ Join in continuous queries Table A Table B Result ... 42 ... ...

14. ᬳളᬳᖅັᧃ Join in continuous queries Table A Table B Result 1 42 ... ... ...

15. ᬳളᬳᖅັᧃ Join in continuous queries Table A Table B Result 1 42 42 42 ... ... ...

16. ᬳളᬳᖅັᧃ Join in continuous queries Table A Table B Result 1 42 42 42 7 ... ... ...

17. ᬳളᬳᖅັᧃ Join in continuous queries Table A Table B Result 1 42 42 42 7 ... ... 3 ...

18. ᬳളᬳᖅັᧃ Join in continuous queries Table A Table B Result 1 42 42 42 7 1 ... 3 ... 1 ...

19. ᬳളᬳᖅັᧃ Join in continuous queries Table A Table B Result 1 42 42 42 7 1 2 3 3 3 1 ... 6 ... ...

20. ෸ᳵᑻ‫ݗ‬ᬳള Time-windowed Join

21. ࿜֖ᕚ Watermarks Ԫկ෸ᳵ౿ Ԥၞ Event timestamp Stream (out of order) 21 19 20 17 22 12 14 12 9 15 11 7 ࿜֖ᕚ ࿜֖ᕚ Watermark 17 Watermark 11 Ԫկ Event

22. ෸ᳵᑻ‫ݗ‬ᬳള Time-windowed Join SELECT * FROM Orders o, Shipments s WHERE o.id = s.orderId AND s.shiptime BETWEEN o.ordertime AND o.ordertime + INTERVAL '4' HOUR

23.෸ᳵᑻ‫ݗ‬ᬳള Time-windowed Join

24. ܲ‫ݥ‬ᤒ Temporal Tables