【主会场04-Stephan】Flink Forward China keynote

Flink Forward China keynote
展开查看详情

1.Stream Processing takes on Everything Stephan Ewen Flink Forward China 2018

2. Streams as a Paradigm bounded stream bounded stream start of past now future the stream unbounded stream unbounded stream

3. Stream Processing takes on Everything Batch Continuous Processing & Event-driven Processing Streaming Analytics Applications offline real-tim e

4. Some new Streaming Features in Flink 1.7 SELECT * FROM TaxiRides A X Y MATCH_RECOGNIZE ( PARTITION BY driverId ORDER BY rideTime MEASURES S.rideId as sRideId AFTER MATCH SKIP PAST LAST ROW PATTERN (S M{2,} E) DEFINE S AS S.isStart = true, M AS M.rideId <> S.rideId, A B E AS E.isStart = false AND E.rideId = S.rideId) Time-versioned Joins MATCH_RECOGNIZE Schema Upgrades S3 exactly-once

5.Applications and Services

6.Request/Response Applications Event-sourced / Streaming Applications Application Streaming Application Database State

7.Streaming Applications Interface Streaming Streaming Application Application Event Sourcing State State Command Query Streaming Responsibility Segregation Application State

8.What about Transactional Applications?

9. Example: Accounts and Transfers ACID Acct. $$ Atomicity Consistency A > 100? A Acct. $$$ A -= 50 B += 50 C Acct. $ E Isolation Durability Acct. $$ G

10. Streaming Transaction Processing Acct. $$ A Acct. $$$ A -= 50 C Acct. $ E Acct. $$ A -= 50 G B += 50 Acct. $ B += 50 B Acct. $$$ D Acct. $$ F Acct. $$$ H

11. Streaming Transaction Processing A -= 50 B += 50 Acct. $$ A Acct. $$$ A C Acct. $ E Acct. $$ A > 50 ? G A -= 50 A > 50 ? B += 50 Acct. $ B Acct. $$$ D Acct. $$ F Acct. $$$ H

12. Streaming Transaction Processing Acct. $$ A Acct. $$$ A -= 50 C Acct. $ E Acct. $$ A > 50 ? G A -= 50 B += 50 Acct. $ B += 50 B Acct. $$$ D Acct. $$ F Acct. $$$ H

13. Streaming Transaction Processing A > 50? Acct. $$ How to resolve this A > 50? A > 100? A Acct. $$$ consistently? A -= 50 A -= 50 C Acct. $ B += 50 A -= 100 E Acct. $$ G Event-time to the rescue! A > 100? A -= 100 B += 50 Acct. $ B D += 100 Acct. $$$ D += 100 D Acct. $$ F Acct. $$$ H

14.Serializable Isolation in Streaming Transactions A B C AC Txn: (A,B,C) -> C Txn: (A,B,C) A B C A -> A D E D F Txn: (D,E) -> (D,F) transaction define reorder events ordering events

15. Serializable Isolation in Streaming Transactions (0xf876ab78|0x0b7cc7a3) Logical Clocks event re-ordering & Iterative streaming to define schedule out-of-order processing dataflows

16.dA Streaming Ledger ACID Exactly-once At-least-once

17. Applications against Shared Consistent State Appl. Deposits Deposit: A: $ shared state Account Balance Asset Qty. event streams Appl. Appl. Xfer: A → B: $ SUCCESS Swap: X →Y A=$, B=$ Transfers

18. Performance Scale Out Hot Keys 100% update queries 4 rows written/query Extreme contention: 800,000 200 million rows updates/sec 100% update on 1,000 keys queries 4 rows Slowdown, but written/query stable

19.Performance in a Geo-distributed Setup

20. Apache Flink: The Powerful Foundation This technology is possible, because Apache Flink offers such powerful building blocks • Continuous processing • Iterative flows • Flexible state abstraction • Asynchronous checkpoints • Sophisticated event-time/watermarks

21. Stream Processing takes on Everything Batch ✔ Continuous Processing & ✔ Event-driven ✔ Processing Streaming Analytics Applications offline real-tim e

22. Stream Processing takes on Everything Batch ✔ Continuous Processing & ✔ Event-driven ✔ Transacional ✔ Processing Streaming Analytics Applications Processing analytical operational …with the right framework ;-)

23.Apache Flink