Apache Flink® 1.7 and Beyond Part I

Apache Flink® 1.7 and Beyond Part I
展开查看详情

1.Apache Flink ® 1.7 and Beyond ‫ݪل‬ғ data Artisans ᘳ֖ғ Solutions Architect ᄍᦖᘏғ Konstantin Knauf (Slides by Till Rohrmann) @snntrable

2. What is Apache Flink? Data Stream Event-driven Batch Processing Processing process static and realtime results Applications historic data from data streams data-driven actions and services Stateful Computations Over Data Streams

3.Flink 1.7: What happened so far?

4. Flink 1.7.0 in Numbers • Contributors: 112 • Resolved issues: 430 • Commits: 970 • Changes LOC: +103824/-63124

5. Flink Applications Need to Evolve • E.g. changing requirements, new algorithms, better serializers, bug fixes, etc. • Expensive to restart application from scratch (maintain state)

6. State Schema Evolution • Support for changing state schema “Upgrading Stateful Flink Streaming Applications: State of • Adding/Removing fields the Union” by Tzu-Li Tai Today @ 5:20 pm Room 1 • Changing type of fields • Currently fully supported when using Avro types

7. Converting Currencies 7:12pm 9:37am 8:45am € 1 $ 1.13 CN¥ 7.8

8. Temporal Tables and Joins 13 11 7 7 4 Currency Rate Time CN¥ 7.8 3 CN¥ 7.89 5 15 14 12 CN¥ 7.75 9

9.SQL for Pattern Analysis SELECT * from ?

10.SELECT * MATCH_RECOGNIZE FROM TaxiRides MATCH_RECOGNIZE ( PARTITION BY driverId ORDER BY rideTime MEASURES S.rideId as sRideId AFTER MATCH SKIP PAST LAST ROW PATTERN (S M{2,} E) DEFINE S AS S.isStart = true, M AS M.rideId <> S.rideId, E AS E.isStart = false AND E.rideId = S.rideId )

11. More SQL Improvements • ElasticSearch 6 Table Sink “Flink Streaming SQL 2018” by Piotr Nowojski Today @ • Support for views in SQL Client 4:00 pm Room 1 • More built-in functions: TO_BASE64, LOG2, REPLACE, COSH,…

12. Other Notable Features • Scala 2.12 Support • Exactly-once S3 StreamingFileSink • Kafka 2.0 connector • Versioned REST API • Removal of legacy mode

13.Flink 1.8+: What is happening next?

14. End-to-end SQL Only Pipelines Hive Meta Store • Support for external catalogs (Confluent Schema Registry, Hive Meta Store) Input schema information Output schema information • Data definition language (DDL) SQL Table Source Table Sink Query