Designing distributed applications with streams

DriveTribe工程师分享基于Apache Flink实现的关于时间数据流应用程序,采用了函数式编程和Flink进行深度集成,实现了分布式和高可扩展程序和数据平台。
展开查看详情

1. DRIVETRIBE ENGINEERING PANTA RHEI: DESIGNING DISTRIBUTED APPLICATIONS WITH STREAMS

2.DRIVETRIBE 2 DRIVETRIBE ▸ The world biggest motoring community. ▸ A social platform for petrolheads. ▸ By Clarkson, Hammond and May.

3.DRIVETRIBE 3 DRIVETRIBE ▸ A content destination at the core. ▸ Users consume feeds of content: images, videos, long-form articles. ▸ Content is organised in homogenous categories called “tribes”. ▸ Different users have different interests and the tribe model allows to mix and match at will.

4.DRIVETRIBE 4 DRIVETRIBE ARTICLE ▸ Single article by Richard Hammond. ▸ Contains a plethora of content and engagement information. ▸ What kind of data do we need in order to compute an aggregation like this?

5.DRIVETRIBE 5 DRIVETRIBE ARTICLE ▸ getUser(id: Id[User])

6.DRIVETRIBE 6 DRIVETRIBE ARTICLE ▸ getUser(id: Id[User]) ▸ getTribe(id: Id[Tribe])

7.DRIVETRIBE 7 DRIVETRIBE ARTICLE ▸ getUser(id: Id[User]) ▸ getTribe(id: Id[Tribe]) ▸ getArticle(id: Id[Article])

8.DRIVETRIBE 8 DRIVETRIBE ARTICLE ▸ getUser(id: Id[User]) ▸ getTribe(id: Id[Tribe]) ▸ getArticle(id: Id[Article]) ▸ countViews(id: Id[Article])

9.DRIVETRIBE 9 DRIVETRIBE ARTICLE ▸ getUser(id: Id[User]) ▸ getTribe(id: Id[Tribe]) ▸ getArticle(id: Id[Article]) ▸ countViews(id: Id[Article]) ▸ countComments(id: Id[Article])

10.DRIVETRIBE 10 DRIVETRIBE ARTICLE ▸ getUser(id: Id[User]) ▸ getTribe(id: Id[Tribe]) ▸ getArticle(id: Id[Article]) ▸ countViews(id: Id[Article]) ▸ countComments(id: Id[Article]) ▸ countBumps(id: Id[Article])

11.DRIVETRIBE 11 DRIVETRIBE FEED OF ARTICLES ▸ rankArticles(forUserId).flatMap { a => … } ▸ getUser(id: Id[User]) ▸ getTribe(id: Id[Tribe]) ▸ getArticle(id: Id[Article]) ▸ countViews(id: Id[Article]) ▸…

12.DRIVETRIBE 12 QUINTESSENTIAL PREREQUISITES ▸ Scalable. Jeremy Clarkson has 7.2M Twitter followers. Cannot really hack it and worry about it later. ▸ Performant. Low latency is key and mobile networks add quite a bit of it. ▸ Flexible. Almost nobody gets it right the first time around. The ability to iterate is paramount. ▸ Maintainable. Spaghetti code works like interest on debt.

13.DRIVETRIBE 13 THREE TIER APPROACH ▸ Clients interact with a fleet of stateless servers (aka “API” servers or “Backend”) via HTTP (which is stateless). ▸ Global shared mutable state (aka the Database). ▸ Starting simple: Store data in a DB. ▸ Starting simple: Compute the aggregated views on the fly.

14.DRIVETRIBE 14 DRIVETRIBE ARTICLE ▸ getUser(id: Id[User]) ▸ getTribe(id: Id[Tribe]) ▸ getArticle(id: Id[Article]) ▸ countComments(id: Id[Article]) ▸ countBumps(id: Id[Article]) ▸ countViews(id: Id[Article])

15.DRIVETRIBE 15 DRIVETRIBE FEED OF ARTICLES ▸ (6 queries per Item) x (Y items per page) ▸ Cost of ranking and personalisation. ▸ Quite some work at read time. ▸ Slow. Not really Performant. ▸ Also very expensive to scale.

16.DRIVETRIBE 16 WRITE TIME AGGREGATION ▸ Compute the aggregation at write time. ▸ Then a single query can fetch all the views at once. That scales.

17.DRIVETRIBE 17 WRITE TIME AGGREGATION ▸ Compute the aggregation at write time. ▸ Then a single query can fetch all the views at once. That scales.

18.DRIVETRIBE 18 WRITE TIME AGGREGATION ▸ Compute the aggregation at write time. ▸ Then a single query can fetch all the views at once. That scales.

19.DRIVETRIBE 19 WRITE TIME AGGREGATION ▸ Compute the aggregation at write time. ▸ Then a single query can fetch all the views at once. That scales.

20.DRIVETRIBE 20 WRITE TIME AGGREGATION ▸ Compute the aggregation at write time. ▸ Then a single query can fetch all the views at once. That scales.

21.DRIVETRIBE 21 WRITE TIME AGGREGATION - EVOLUTION ▸ sendNotification

22.DRIVETRIBE 22 WRITE TIME AGGREGATION - EVOLUTION ▸ sendNotification ▸ updateUserStats

23.DRIVETRIBE 23 WRITE TIME AGGREGATION - EVOLUTION ▸ sendNotification ▸ updateUserStats. ▸ What if we have a cache? ▸ Or a different database for search?

24.DRIVETRIBE 24 WRITE TIME AGGREGATION - FAN OUT ▸ Requests may need to fan out and update multiple views. ▸ Eg: Authors can update their names. ▸ This needs to be propagated to all materialised views.

25.DRIVETRIBE 25 WRITE TIME AGGREGATION ▸ A simple user action is triggering an unbound sequence of side effects. ▸ Most of which need network IO. ▸ Many of which can fail.

26.DRIVETRIBE 26 ATOMICITY Atomicity? ▸ What happens if one of them fails? What happens if the server fails in the middle? ▸ Inconsistent.

27.DRIVETRIBE 27 CONCURRENCY Concurrency ▸ Concurrent mutations on a global shared state entail race conditions. ▸ State mutations are destructive and can not be (easily) undone. ▸ A bug can corrupt the data permanently.

28.DRIVETRIBE 28 ITALIAN PASTA Extensibility ▸ Model evolution becomes difficult. Reads and writes are tightly coupled. ▸ Migrations are scary. ▸ This is neither Extensible nor Maintainable.

29.DRIVETRIBE 29 DIFFERENT APPROACH ▸ Let’s take a step back and try to decouple things. ▸ Clients send events to the API: “John liked Jeremy’s post”, “Maria updated her profile” ▸ Events are immutable. They capture a user action at some point in time. ▸ Every application state instance can be modelled as a projection of those events.