展开查看详情
1.Near Real-Time Recommendations - Spark
Streaming
Elliot Chow, Netflix
Nitin Sharma, Netflix
#ML7SAIS
#ML7SAIS
2.Agenda
● Recommendations @ Netflix
● The Need for Near Real Time
● Use Cases
● Common Infrastructure
● Scaling Challenges
#ML7SAIS
3.Recommendations at Netflix
● Personalize the Netflix experience for each member
○ Goal: Quickly help members find content they’d like to
watch
○ Risk: Member may lose interest and abandon the service
○ Challenge: Recommending at scale
#ML7SAIS
4.Scale @ Netflix
● 125M+ active members
● 190 countries
● 450B+ unique events/day
● 700+ Kafka topics
#ML7SAIS
5.Typical Data Pipelines @ Netflix
● Data stored in Hive/S3
● Batch ETLs using Spark/Hive
● Table partitioning by day or hour
● Job scheduling by both CRON or data availability
● Latency often is on the order of days
#ML7SAIS
6.The Need for Near Real Time (NRT)
● Dynamic catalog
● Growing member base
● Time sensitivity
○ Content popularity changes
○ Member interests evolve
#ML7SAIS
7.The Need for Near Real Time (NRT)
● Increasing amount of data
○ Process data as soon as possible to keep latencies low
○ Minimize amount of data to reprocess in case of failure
● Multi-Armed Bandits Adoption
#ML7SAIS
8.Use Cases
● Video Insights
● ML Pipelines for Recommendations
#ML7SAIS
9.NRT for Video
Insights
#ML7SAIS
10.Video Insights
● New title launches
● Early reads on title performance
● Slice by various dimensions
#ML7SAIS
11.Video Insights
Service
#ML7SAIS
12.Video Insights
Service
#ML7SAIS
13.Video Insights
Service
#ML7SAIS
14.Video Insights
Service
#ML7SAIS
15.Video Insights Client
Service
#ML7SAIS
16.Video Insights - State
● Counts maintained in Spark
● Custom state management
○ Based on mapWithState implementation
input.scan(initRDD)((currentRDD, batchRDD) => f(currentRDD, batchRDD))
○ Easier to re-use the function f in batch mode
○ Lower-level control over state management
○ scanByKey alternative for keyed state
#ML7SAIS
17.NRT for Recommendations
#ML7SAIS
18.Billboard Recommendations
● Recommend a relevant title to each
member
● Right time
● Respond quickly to member feedback
#ML7SAIS
19. Artwork Personalization
● Personalized Image
● Visual Evidence
● Quickly adapting - Title
launches, member tastes
● Rapid learning - Cold start
#ML7SAIS 19
20. Traditional Recommendations
DataStore/
Millions of ETL Training Online
Play related Layers pipelines caches
Signals
Few days Rankings
Models
Precompute/Live
Compute
#ML7SAIS
21. NRT Recommendations
DataStore/Online
Millions of Streaming caches
Play related layer
Signals
NRT Rankings
Training
pipelines
Models
Precompute/Live
Compute
#ML7SAIS
22.Required Data
● Impressions, Plays, etc.
● Attribution
● Explore/Exploit Metadata
#ML7SAIS
23.Attribution Infrastructure
Stream Processing Batch Processing
Query API
#ML7SAIS
24.Stream Processing - Zoomed in
Impression
Play
Enrich
Cache
Video
Metadata
#ML7SAIS
25.Batch Processing - Zoomed in
Impression
Windowed
Impression
Dedupe
Windowed
Play
Join
Play
Video Experimentation
Metadata Data
#ML7SAIS
26.Common Infrastructure
#ML7SAIS
27. Netflix Spark Stack
● Jenkins
● Spinnaker
● ASG
● Runners: Marathon, Meson
● Resource Manager: Mesos
● Storage: HDFS, S3, EFS
● Multi-Region
#ML7SAIS
28. Multi Region Challenges
● Geo routing
● Impression in one region;
play in another
● Streaming - Multi Region
● Batch Reduce/Merge - One
region
#ML7SAIS
29.Can it withstand Chaos?
• Chaos is a design principle
• Instance Failovers => Region Failovers
• Transparent to the consumers
• Over provisioned
• Long term - Autoscale
#ML7SAIS
