【主会场06-罗李】滴滴实时计算平台架构与实践 部分1

1 .ᄆᄆਫ෸ᦇᓒଘ‫ݣ‬ຝ຅Өਫ᪢ The Application and Architecture of the Real-Time Computing Platform in DiDi ‫ݪل‬ғᄆᄆ‫ڊ‬ᤈ Company: DiDi Chuxing ᘳ֖ғದ๞Ꮈᑪާ Title Director of BigData Infrastructure Department ᄍᦖᘏғᗔ๫ Lecturer Luo Li 

2 . Agenda Ø ᙧว݊ሿᇫ Ø Background Ø ਫ෸ᥢ‫܃ڞ‬ᯈਫ᪢ Ø The CEP practice Ø 6WUHDP64/Օᕨ Ø StreamSQL Ø ਫ෸ᦇᓒଘ‫ݣ‬໛ຝ Ø Stream Platform Infrastructure Ø ௛ᕮ޾ᥢ‫ښ‬ Ø Future plans 

3 . ӱ‫ۓ‬ᇙᅩ characteristic of business 

4 . Ø ᄆᄆ๜᫝ฎӞӻਫ෸Իฃ୚ක Ø DiDi is a realtime Trade Engine Ø ᄆᄆጱහഝ޾࣋วॠᆐฎਫ෸ጱ Ø Didi's data and scenes are naturally real-time Special Shared /X[XU\ ள᫣ ӫ᫣ սՁ ᨟܏᫣ … ‫ڊ‬ᑄ᫣ Fast Cars … Taxi Cars Cars &DUV Transaction Collection ᫥ᭃහഝ Իฃහഝ ईᅩහഝ ෭பහഝ Trace Data Log Data Data point Data ਫ෸᯻ᵞ ਫ෸ᦇᓒ Real-time ETC & Collection Real-time process 

5 . ၞᦇᓒದ๞ᄍᬰ Didi’s Real-time develop steps ᒫӞᴤྦྷғӱ‫ۓ‬ොᛔୌੜᵞᗭ First Step A lot of some Clusters ᒫԫᴤྦྷғᵞӾୗय़ᵞᗭ̵ଘ‫ݣ‬۸ Second Step Single Big Shared Cluster ᒫӣᴤྦྷғၞᦇᓒ୏‫ݎ‬64/۸ Third Step SQL On Stream process चԭFlinkጱStreamSQLӤᕚ StreamSQL On Flink चԭFlink CEPጱਫ෸ᥢ‫܃ڞ‬ ᯈ๐‫ۓ‬Ӥᕚ Flink CEP Flink Streaming ಭ‫ف‬ଫአ Flink Streaming Spark Streaming Storm/Jstorm/ ๐‫ۓ‬۸ଘ‫ݣ‬۸ Storm/Jstorm/ Spark Streaming Spark/Samza Spark/Samza ӱ‫ۓ‬ොᛔୌੜᵞᗭ some clusters 

6 . ଫአ࣋ว Scenarios Ø ਫ෸‫̵ڭݎ‬ਫ෸ᷚഴ̵ Ø හഝ՗ၾ௳ᴚ‫کڜ‬HBase Ø Ø HBase ਫ෸୑ଉ༄ၥᒵ Druid ESᒵਂ‫ؙ‬ Druid ES - -- Ø Իฃፊഴ̵๐‫ۓ‬ፊ Ø ਫ෸ԙਮᇙ஄̵‫ݪ‬๢ᇙ஄̵ Ø Ø ഴ̵ૡ‫ܔ‬ፊഴᒵ ᦈ‫ܔ‬ᇙ஄ᒵ 

7 . ଫአᥢཛྷ Data Scale ๐‫ۓ‬ӱ‫ۓ‬ᕚ ᵞᗭᥢཛྷ ၞᦇᓒձ‫ۓ‬හ ྯ෭॒ቘහഝᰁ Applications Cluster Scale Streaming Jobs Data Scale per day 

8 .ਫ෸ᥢ‫܃ڞ‬ᯈҁCEP҂ਫ᪢ The CEP practice 

9 . ਫ෸ᥢ‫܃ڞ‬ᯈҁCEP҂ Pattern .begin(s1) ӻ௔۸០ᲀ .where(key=bubble && city=166) Ø ʼn຤उ૱ԙਮٛု‫ݸ‬ᑁဌํӥ‫ܔ‬Ŋ .notFollowedBy(s2) .where(key=order_send && city = 166) Ø ʼn຤उ૱ӥ‫ݸܔ‬ᑁဌՈള‫ܔ‬Ŋ .timeEnd(s3) Ø શ꧊ᥢ‫ڞ‬ .timeShiftMore(s1, Time.ms(5000)) ᥢ‫ڜڞ‬ᤒ$3, ୑ଉૡ‫ܔ‬༄ၥ Ø ᦈ‫ܔ‬ᇫாᳩ෸ᳵ‫ړ‬ᰦ؊ኸࣁ‫ݪ‬๢૪ಶ‫ܔ‬Ŋ හഝრ ᶼ॒ቘ DD-CEP ‫܃‬ᯈ Øʼnᦈ‫ܔ‬ᇫாᳩ෸ᳵੜ෸؊ኸࣁ୏তᦇᩇŊ ਫ෸ᥢ‫܃ڞ‬ᯈ๐‫ۓ‬ Øʼn೪᫣ӨGRVᦈ‫ܔ‬ᇫாӧӞᛘ‫ݪ‬๢૪ಶ‫ܔ‬Ŋ 

10 .CEP‫ۑ‬ᚆ ௔ᚆදᬰ 

11 .StreamSQL 

12 . StreamSQLୌᦡ      ಘ઀''/ ӿ੄8') ‫݌‬ၞ-2,1 ᖌᤒ-2,1 64/ ,'( 

13 . ಘ઀''/ Ø ಑᭗ᄆᄆٖ᮱Ԇၞၾ௳ᴚ‫޾ڜ‬ਫ෸ਂ‫ؙ‬ Ø ᭗ᬦಘ઀''/᧍‫ݙ‬ൈᬿ6RXUFH޾6LQN MQ Ø ٖᗝ-621̵ຽ‫ٵ‬෭ப̵3%໒ୗᥴຉ Kafka Druid Kafka HBase StreamSQL Phoenix MQ ElaticSearch MYSQL KVStore