【主会场04-Stephan】Flink Forward China keynote

1 .Stream Processing takes on Everything Stephan Ewen Flink Forward China 2018 

2 . Streams as a Paradigm bounded stream bounded stream start of past now future the stream unbounded stream unbounded stream 

3 . Stream Processing takes on Everything Batch Continuous Processing & Event-driven Processing Streaming Analytics Applications offline real-tim e 

4 . Some new Streaming Features in Flink 1.7 SELECT * FROM TaxiRides A X Y MATCH_RECOGNIZE ( PARTITION BY driverId ORDER BY rideTime MEASURES S.rideId as sRideId AFTER MATCH SKIP PAST LAST ROW PATTERN (S M{2,} E) DEFINE S AS S.isStart = true, M AS M.rideId <> S.rideId, A B E AS E.isStart = false AND E.rideId = S.rideId) Time-versioned Joins MATCH_RECOGNIZE Schema Upgrades S3 exactly-once 

5 .Applications and Services 

6 .Request/Response Applications Event-sourced / Streaming Applications Application Streaming Application Database State 

7 .Streaming Applications Interface Streaming Streaming Application Application Event Sourcing State State Command Query Streaming Responsibility Segregation Application State 

8 .What about Transactional Applications? 

9 . Example: Accounts and Transfers ACID Acct. $$ Atomicity Consistency A > 100? A Acct. $$$ A -= 50 B += 50 C Acct. $ E Isolation Durability Acct. $$ G 

10 . Streaming Transaction Processing Acct. $$ A Acct. $$$ A -= 50 C Acct. $ E Acct. $$ A -= 50 G B += 50 Acct. $ B += 50 B Acct. $$$ D Acct. $$ F Acct. $$$ H 

11 . Streaming Transaction Processing A -= 50 B += 50 Acct. $$ A Acct. $$$ A C Acct. $ E Acct. $$ A > 50 ? G A -= 50 A > 50 ? B += 50 Acct. $ B Acct. $$$ D Acct. $$ F Acct. $$$ H 

12 . Streaming Transaction Processing Acct. $$ A Acct. $$$ A -= 50 C Acct. $ E Acct. $$ A > 50 ? G A -= 50 B += 50 Acct. $ B += 50 B Acct. $$$ D Acct. $$ F Acct. $$$ H 

13 . Streaming Transaction Processing A > 50? Acct. $$ How to resolve this A > 50? A > 100? A Acct. $$$ consistently? A -= 50 A -= 50 C Acct. $ B += 50 A -= 100 E Acct. $$ G Event-time to the rescue! A > 100? A -= 100 B += 50 Acct. $ B D += 100 Acct. $$$ D += 100 D Acct. $$ F Acct. $$$ H 

14 .Serializable Isolation in Streaming Transactions A B C AC Txn: (A,B,C) -> C Txn: (A,B,C) A B C A -> A D E D F Txn: (D,E) -> (D,F) transaction define reorder events ordering events 

15 . Serializable Isolation in Streaming Transactions (0xf876ab78|0x0b7cc7a3) Logical Clocks event re-ordering & Iterative streaming to define schedule out-of-order processing dataflows 

16 .dA Streaming Ledger ACID Exactly-once At-least-once 

17 . Applications against Shared Consistent State Appl. Deposits Deposit: A: $ shared state Account Balance Asset Qty. event streams Appl. Appl. Xfer: A → B: $ SUCCESS Swap: X →Y A=$, B=$ Transfers 

18 . Performance Scale Out Hot Keys 100% update queries 4 rows written/query Extreme contention: 800,000 200 million rows updates/sec 100% update on 1,000 keys queries 4 rows Slowdown, but written/query stable 

19 .Performance in a Geo-distributed Setup 

20 . Apache Flink: The Powerful Foundation This technology is possible, because Apache Flink offers such powerful building blocks • Continuous processing • Iterative flows • Flexible state abstraction • Asynchronous checkpoints • Sophisticated event-time/watermarks 

21 . Stream Processing takes on Everything Batch ✔ Continuous Processing & ✔ Event-driven ✔ Processing Streaming Analytics Applications offline real-tim e 

22 . Stream Processing takes on Everything Batch ✔ Continuous Processing & ✔ Event-driven ✔ Transacional ✔ Processing Streaming Analytics Applications Processing analytical operational …with the right framework ;-) 

23 .Apache Flink